Autonomous robotic intracardiac catheter navigation using haptic vision

Catheter design

We designed the catheter using concentric tube robot technology in which robots are composed of multiple needle-sized concentrically combined precurved superelastic tubes. A motorized drive system located at the base of the tubes rotated and telescopically extended the tubes with respect to each other to control the shape of the catheter and its tip position (movie S1) (28, 29). The drive system was mounted on the operating room table using a passively adjustable frame that allowed the catheter tip to be positioned and oriented for entry into the apex (Fig. 2B).

Tools and devices were delivered through the lumen of the innermost robot tube, which incorporated a valve and flush system at its proximal end. This system enabled the catheter lumen to be flushed with saline to prevent air entry into the heart and to prevent pressurized blood from the heart from entering the lumen of the catheter. We used a design optimization algorithm to solve for the tube parameters based on the anatomical constraints and clinical task (aortic paravalvular leak closure) (30). The anatomical and task constraints were defined using a 3D model of an adult human left ventricle (Fig. 4). Because the relative dimensions of the human and porcine hearts are similar, the resulting design was appropriate for our in vivo experiments. The design algorithm solved for the tube parameters enabling the catheter tip to reach from the apex of the heart to a set of 25 uniformly sampled points around the aortic valve annulus without the catheter contacting the ventricular wall along its length. The orientation of the catheter tip was further constrained at the 25 points to be within 10° of orthogonal to the valve plane. The resulting design was composed of three tubes forming two telescoping sections with motion as shown in movie S1. The tube parameters of the robotic catheter are given in Table 1.

Haptic vision sensor design

We fabricated the sensor (Fig. 1A) using a 1-mm³ complementary metal-oxide semiconductor camera (NanEye Camera System, AWAIBA) and a 1.6 mm by 1.6 mm light-emitting diode (LED) (XQ-B LED, Cree Inc.) encased in an 8-mm-diameter silicone optical window (QSil 218, Quantum Silicones LLC.) molded onto a stainless-steel body (15). The optical window diameter was selected to provide a field of view facilitating both autonomous and operator-guided navigation. The sensor also incorporated a 2.5-mm-diameter working channel for device delivery. The system clamped to the tip of the catheter such that the lumen of the innermost catheter tube aligned with the endoscope working channel. Although we have also designed catheters in which the sensor wiring was run through the catheter lumen, in these experiments, the wiring for the camera and LED were run outside the catheter so that the sensor could be replaced without disassembling the robotic catheter.

Computer software design

The software was executed on two PCs. One was used for catheter motion control [Intel Core Quad CPU Q9450@2.66 GHz with 4-GB random-access memory (RAM)], whereas the second was used to acquire and process images from the haptic vision sensor (Intel Core i7-6700HQ CPU@2.6 GHz with 16-GB RAM). The two computers exchanged information at runtime via transmission control protocol/internet protocol. The motion control computer received real-time heart rate data by serial port (Advisor, SurgiVet) and was also connected through universal serial bus to a six-DOF joystick (Touch, 3D Systems) that was used during teleoperated control of catheter motion. The motion control computer could execute either the autonomous navigation algorithms or the joystick motion commands. In either case, catheter tip motion commands were converted to signals sent to the motor amplifiers of the catheter drive system.

Robotic catheter control

The catheter control code converting desired catheter tip displacements to the equivalent rotations and translations of the individual tubes was written in C++. The code was based on modeling the kinematics using a functional approximation (truncated Fourier series) that was calibrated offline using tip location data collected over the workspace (28). The calibrated functional approximation model had been previously demonstrated to predict catheter tip position more accurately (i.e., smaller average and maximum prediction error) over the workspace compared with the calibrated mechanics-based model (31). Catheter contact with tissue along its length produced unmodeled and unmeasured deformations that must be compensated for via tip imaging. A hierarchical control approach was used (32) to ensure that the desired tip position was given a higher priority than the desired orientation if both criteria could not be satisfied simultaneously.

Haptic vision–based contact classification

To perform wall following, we designed a machine learning–based image classifier that can distinguish between blood (no contact) or ventricular wall tissue and the bioprosthetic aortic valve. The algorithm used the bag-of-words approach (33) to separate images into groups (classes) based on the number of occurrences of specific features of interest. During training, the algorithm determined which features were of interest and the relationship between their number and the image class. For training, we used OpenCV to detect features in a set of manually labeled training images. Next, the detected features were encoded mathematically using LUCID descriptors for efficient online computation (34). To reduce the number of features, we identified the optimal feature representatives using clustering (k-means). The resulting cluster centers were the representative features used for the rest of the training, as well as for runtime image classification. Having identified the set of representative features, we made a second pass through the training data to build a feature histogram for each image by counting how many times each representative feature appeared in the image. The final step was to train a support vector machine (SVM) (35) classifier that learned the relationship between the feature histogram and the corresponding class.

Using the trained algorithm, image classification proceeded by first detecting features and computing the corresponding LUCID descriptors. The features were then matched to the closest representative features, and the resulting feature histogram was constructed. On the basis of the histogram, the SVM classifier predicted the tissue-based contact state. We achieved good results using a small set of training images (~2000 images) with training taking ~4 min. Because image classification took 1 ms, our haptic vision system estimated contact state at the frame rate of the camera (45 frames/s). The contact classification algorithm was accurate 97% of the time (tested on 7000 images not used for training) with type I error (false positive) of 3.7% and type II (false negative) of 2.3%.

Continuous contact navigation algorithm

When the catheter was positioned laterally against cardiac tissue, its flexibility could enable continuous contact to be maintained without applying excessive force to the tissue. We used haptic vision to control the amount of tissue contact by controlling catheter motion in the direction orthogonal to the tissue surface. Catheter motion in the plane of the tissue surface was independently controlled so as to produce wall following at the desired velocity and in the desired direction. The controller was initialized with an estimate of wall location so that if it was not initially in tissue contact, it moved toward the wall to generate contact. This occurred in our in vivo experiments during navigation from the apex to the aortic valve. The catheter started in the center of the apex with the haptic vision sensor detecting only blood. It would then move in the direction of the desired wall (Fig. 1D, a ➔ b), specified using valve clock coordinates (Fig. 2C), to establish contact and then follow that wall until it reached the valve.

When the haptic vision sensor was pressed laterally against the tissue, the tissue deformed around the sensor tip so that it covered a portion of the field of view (Fig. 1B). The navigation algorithm adjusted the catheter position orthogonal to the tissue surface to maintain the centroid of the tissue contact area within a desired range on the periphery of the image, typically 30 to 40%. We implemented tissue segmentation by first applying a Gaussian filter to reduce the level of noise in the image. Next, we segmented the tissue using color thresholding on the hue saturation value (HSV) and the CIE L*a*b* representations of color images. The result was a binary image, where white pixels indicated tissue. Last, we performed a morphological opening operation to remove noise from the binary image. After segmentation, the tissue centroid was computed and sent to the motion control computer. Image processing speed was sufficient to provide updates at the camera frame rate (45 frames/s).

Intermittent contact navigation algorithm

When a catheter is stiff along its longitudinal axis and positioned orthogonal to a tissue surface that moved significantly in the direction of this axis over the cardiac cycle, the contact forces can become sufficiently high so as to result in tissue damage or puncture. To maintain contact forces at safe levels, one approach is to design the catheter so that it can perform high-velocity trajectories that move the robotic catheter tip in synchrony with the tissue (36). We used an alternative technique requiring only slow catheter motion so as to position the tip such that it was in contact with the tissue for a specified fraction of the cardiac cycle, the contact duty cycle, D (Fig. 1C). As described in the section below titled “In-vivo calibration of contact duty cycle versus maximum tissue force,” the contact duty cycle was linearly related to the maximum contact force. The intermittent contact mode was used during navigation around the aortic valve annulus.

We implemented intermittent contact navigation using haptic vision to detect tissue contact and, combined with heart rate data, to compute the contact duty cycle at the frame rate of the camera (45 frames/s). We implemented a controller that adjusted catheter position along its longitudinal axis to drive the contact duty cycle to the desired value. Catheter motion in the plane of the tissue surface was performed either autonomously or by the operator (shared control mode). In the autonomous mode, catheter motion in the tissue plane was performed only during the fraction of the cardiac cycle when the haptic vision sensor indicated that the catheter was not touching tissue. This reduced the occurrence of the catheter tip sticking to the tissue surface during wall following.

Autonomous navigation on the valve annulus

Intermittent contact control was used to control catheter motion orthogonal to the plane of the annulus. The desired value of contact duty cycle was typically set to be ~40%. Thus, 40% of the cardiac cycle was available for image processing (during contact), whereas the motion in the plane of the annulus was performed during the 60% noncontact portion of the cardiac cycle. During contact, the robot detected the blue tangent sutures on the valve (Fig. 2C) using color thresholding in the HSV color space and computed the centroid of the detected sutures. Next, a Hough transform on the thresholded image was used to estimate the tangent of the aortic annulus. During the noncontact portion of the cardiac cycle, the algorithm generated independent motion commands in the radial and tangential directions. In the radial direction, the catheter adjusted its position such that the centroid of the detected sutures was centered in the imaging frame. Motion in the tangential direction was performed at a specified velocity. While navigating around the valve, the robot incrementally built a map of the location of the annulus in 3D space based on the centroids of the detected sutures and the catheter tip coordinates as computed using the robot kinematic model. The model was initialized with the known valve diameter and the specified direction of approach. By comparing the current tangent with the model, the robot estimated its clock position on the annulus. Although not implemented, this model may also be used to estimate the valve tangent and radial position in situations where the sutures are not well detected.

Registration of valve rotation with respect to the robot

We assumed that paravalvular leaks have been identified in preoperative imaging, which also indicated their location relative to the features of the bioprosthetic valve, e.g., leaflet commissures. In the ventricular view of the valve annulus provided by haptic vision, such features are hidden. Although the model built during annular navigation defines the coordinates of the annulus circle in 3D space, there was no means to refine the initial estimate of where 12 o’clock fell on the circle, i.e., to establish the orientation of the valve about its axis. To enable the robot to refine its orientation estimate, we introduced registration features into the annulus composed of green sutures located at 4, 8, and 12 o’clock. During annular navigation, whenever the robot detected one of these features, it compared its actual location with the current prediction of the model and updated its estimate of valve rotation accordingly.

Endothelialization of blue and green sutures on the valve annulus

In clinical use, the sutures would remain visible for several months before endothelialization. Thus, they could be used for autonomous repair of paravalvular leaks that occur at the time of valve implantation or soon after, as is the case for transcatheter valves (12).

Autonomous navigation for paravalvular leak closure

The algorithm inputs are consisted of the clock-face leak location and the desired ventricular approach direction, also specified as a clock-face position. Starting from just inside the apex of the left ventricle, the catheter moved in the desired approach direction until it detected tissue contact. It then switched to continuous contact mode and performed wall following in the direction of the valve. When the classifier detected the bioprosthetic valve in the haptic vision image, the controller switched to intermittent contact mode and computed the minimum distance direction around the annulus to the leak location based on its initial map of the annulus. As the catheter moved around the annulus in this direction, its map was refined on the basis of the detection of tangent and registration sutures. Once the leak location was reached, the robot controller acted to maintain its position at this location and sent an alert to the operator. Using joystick control, the operator could then reposition the working channel over the leak as needed, and then, the occluder could be deployed.

Software development cycle

To develop and test our autonomous navigation algorithms, we implemented a development cycle composed of three steps: in silico simulation, ex vivo experiments, and in vivo experiments (Fig. 5). We created a simulation engine that can replay time-stamped data, comprising haptic vision images and robot trajectories, recorded during in vivo cases. We used the simulation engine to implement new software functionality and to troubleshoot unexpected results from in vivo experiments. After simulation, we tested new functionality on an ex vivo model comprising an explanted porcine heart, pressurized using a peristaltic pump (Masterflex Pump, 115 VAC). We immobilized the pressurized heart using sutures to attach it to a fixture. On the basis of the outcome of the ex vivo tests, we either performed additional simulations to refine the software implementation or proceeded to in vivo testing. This process was repeated iteratively for each algorithm as it was developed.

Surgical procedure

We created a porcine paravalvular leak model by implanting a custom bioprosthetic device (Fig. 2C) into the aortic valve position in 84.3 ± 4.7 kg of Yorkshire swine. The device was designed with three sewing ring gaps evenly distributed around its circumference (120° apart) to produce the areas of paravalvular leakage. The bioprosthetic valve consists of a titanium frame covered by a nonwoven polyester fabric. A polypropylene felt sewing ring is sutured to the frame around the annulus. Suture is passed through this ring when the valve is sewn in place inside the heart. Last, glutaraldehyde-fixed porcine pericardium leaflets are sutured to the frame (37).

Animal care followed procedures prescribed by the Institutional Animal Care and Use Committee. To implant the bioprosthetic valve, we premedicated the swine with atropine (0.04 mg/kg intramuscularly), followed by Telazol (4.4 mg/kg) and xylazine (2.2 mg/kg intravenously), and we accessed the thoracic cavity through a median sternotomy incision. We acquired epicardial echocardiographic images to determine the size of the valve to be implanted. Next, we initiated cardiopulmonary bypass by placing purse-string sutures for cannulation, cross-clamping the aorta, and infusing cardioplegia solution to induce asystole. We incised the aorta to expose the valve leaflets, which were then removed, and the artificial valve was implanted using nine 2-0 ETHIBOND valve sutures supra-annularly. At this point, we closed by suture the aortomy incision, started rewarming, and released the aortic cross-clamp. We maintained cardiopulmonary bypass to provide 35 to 50% of normal cardiac output to ensure hemodynamic and cardiac rhythm stability. The function of the implanted valve, as well as the leak locations and sizes, were determined by transepicardial short- and long-axis 2D and color Doppler echocardiography. Apical ventriculotomy was then performed, with previous placement of purse-string sutures to stabilize the cardiac apex for the introduction of the robotic catheter. The catheter was introduced through the apex and positioned such that its tip was not in contact with the ventricular walls. All experiments in a group were performed using the same apical catheter position. Throughout the procedure, we continuously monitored arterial blood pressure, central venous pressure, heart rate, blood oxygenation, temperature, and urine output. At the end of the experiment, a euthanasia solution was injected, and we harvested the heart for postmortem evaluation.

Autonomous navigation from apex to valve

We performed experiments on five animals. For each animal, navigation was performed using three valve approach directions corresponding to 6 o’clock (posterior ventricular wall), 9 o’clock (ventricular septal wall), and 12 o’clock (anterior ventricular wall) (Fig. 2C). Of the 90 total trials, the number performed in the 6, 9, and 12 o’clock directions were 31, 32, and 27, respectively.

Autonomous circumnavigation of aortic valve annulus

Experiments were performed on three animals. In the first experiment, a range of contact duty cycles was tested, whereas in the latter two experiments, the contact duty cycle was maintained between 0.3 and 0.4. In all experiments, the tangential velocity was specified as 2 mm/s during those periods when the tip was not in contact with the valve and 0 mm/s when in contact.

Autonomous navigation from apex to paravalvular leaks

We performed experiments on five animals. As an initial step for all experiments, we built a 3D spatial model of the valve by exploring the valve with the catheter under operator control. We used this model, which is separate from the model built by the autonomous controller, to monitor autonomous navigation. For three animals, we also used this model to estimate valve rotation with respect to the robot and provided this estimate as an input to the autonomous navigation algorithm. In two animals, valve rotation was estimated autonomously on the basis of the valve model built by the navigation algorithm and its detection of registration sutures.

In each experiment, navigation trials were individually performed for each of the three leaks located at 2 o’clock (n = 28), 6 o’clock (n = 27), and 10 o’clock (n = 28) (Fig. 2C). For each leak location, we selected a clock direction to follow on the ventricular wall such that the catheter would arrive at the valve annulus close to the leak but far enough away that it would have to pass over registration sutures to reach the leak. In general, this corresponded to approaching the valve at 11, 9, and 1 o’clock to reach the leaks at 2 o’clock (clockwise), 6 o’clock (counter clockwise), and 10 o’clock (counter clockwise), respectively. If we observed that along these paths the annulus was covered by valve tissue or a suturing pledget, then we instructed the navigation algorithm to approach the leak from the opposite direction. Note that the mitral valve is located from 2 to 5 o’clock; the ventricular wall cannot be followed in these directions to reach the aortic valve. Thus, in one experiment involving operator-specified valve registration, the clockwise-approach path was covered by tissue, and we chose to approach the leak directly from the 2 o’clock direction rather than start farther away at 6 o’clock.

We designed the registration sutures to be 120° apart under the assumption that valve rotation with respect to the robot would be less than ±60° from the nominal orientation. In one animal in which valve rotation was estimated autonomously, however, the rotation angle was equal to 60°. In this situation, it is impossible for either man or machine to determine whether the error is +60° or −60°. For these experiments, we shifted the approach direction for the leak at 6 o’clock from 9 to 8 o’clock so that the catheter would only see one set of registration sutures along the path to the leak. This ensured that it would navigate to the correct leak.

Occluder deployment

After navigation to the desired leak location, the operator took control of the catheter and, as needed, centered the working channel over the leak. A three-lobed vascular occluder (AMPLATZER Vascular Plug II, AGA Medical Corporation), attached to a wire and preloaded inside a delivery cannula, was advanced ~3 mm into the leak channel (Fig. 6). The cannula was then withdrawn, allowing the occluder to expand inside the leak channel. We then retracted the wire and robotic catheter until the proximal lobe of the occluder was positioned flush with the valve annulus and surrounding tissue. If positioning was satisfactory, then the device was released by unscrewing it from the wire. If not, then the device was retracted back into the delivery cannula and the procedure was repeated as necessary.

In vivo calibration of contact duty cycle versus maximumtissue force

To investigate the relationship between maximum contact force and contact duty cycle, we designed a handheld instrument integrating haptic vision and force sensing (Fig. 7). Force sensor integration was inspired by (38). The haptic vision sensor is mounted on a stiff tube that is supported by two polymer sliding bearings mounted inside the proximal handle. The proximal end of the shaft is connected to a force sensor. An outer cannula encloses the sensing shaft and extends from the handle to about 6 cm from the sensing tip. When the instrument is inserted into the apex of the heart, the outer cannula is in contact with the apical tissue, but the sensing tube is not. The gap between the outer cannula and sensing tube is filled with silicone grease to prevent blood flow from the heart into the instrument while generating minimal friction on the sensing tube. Calibration experiments indicated that the friction due to the bearings and grease was less than ±0.2 N.

We performed in vivo experiments in which we positioned the haptic vision sensor on the bioprosthetic valve annulus in locations where we could be sure that the sensor was experiencing contact only on its tip. At these locations, we collected force, heart rate, and haptic vision data (Fig. 7B). By manually adjusting instrument position along its axis, we were able to obtain data for a range of duty cycle values. The image and force data were collected at 46 Hz, and contact duty cycle based on valve contact was computed at each sampling time using a data window of width equal to the current measured cardiac period (~36 images). To remove high-frequency components not present in the force data, we then filtered the computed duty cycle using a 121-sample moving average filter corresponding to ~3.4 heartbeats. The filtered input data and the output data (e.g., Fig. 7B) were then binned using duty cycle intervals of 0.05. Last, we computed the relationship between filtered contact duty cycle and maximum applied force by averaging the maximum forces for each binned duty cycle value (Fig. 7C). We computed the Pearson’s coefficient as a measure of linear relationship between the contact duty cycle and the maximum annular force. The Pearson’s coefficient was equal to 0.97, which indicates a strong linear relationship. The plot of Fig. 7C indicates that the contact duty cycle range of 0.35 to 0.45 that we used in most of our experiments corresponded to a maximum force of 1.25 to 2.3 N.