This paper presents FLAF, a focal line and feature-constrained active view planning method for autonomous orientation adjustment of a rotatable active camera during mobile robot navigation. FLAF is built on a visual teach-and-repeat (VT&R) system, which enables robots to cruise various paths that fulfill many daily autonomous navigation requirements. The VT&R system integrates Visual Simultaneous Localization and Mapping (VSLAM) with trajectory following. However, tracking failures in feature-based VSLAM, particularly in textureless regions common in human-made environments, poses a significant challenge to real-world VT&R deployment. To address this, the proposed view planner is integrated into a feature-based VSLAM system, creating an active camerabased VSLAM (AC-SLAM) solution that mitigates tracking failures. Our system features a Pan-Tilt Unit (PTU)-based active camera mounted on a mobile robot. FLAF actively directs the camera toward more map points during path learning and toward more feature-identifiable map points while following the learned trajectory. Using FLAF, the AC-SLAM system constructs a complete path map during teaching and maintains stable localization during repeating. Experimental results in real scenarios show that FLAF significantly outperforms existing methods by accounting for feature identifiability, particularly the view angle of the features. During effectively dealing with low-texture regions in active view planning, considering feature identifiability enables our active VT&R system to perform well in challenging environments.

With reinforcement learning, an agent can learn complex behaviors from high-level abstractions of the task. However, exploration and reward shaping remain challenging for existing methods, especially in scenarios where extrinsic feedback is sparse. Expert demonstrations have been investigated to solve these difficulties, but a tremendous number of high-quality demonstrations are usually required. In this work, an integrated policy gradient algorithm is proposed to boost exploration and facilitate intrinsic reward learning from only a limited number of demonstrations. We achieved this by reformulating the original reward function with two additional terms, where the first term measured the Jensen-Shannon divergence between current policy and the expert's demonstrations, and the second term estimated the agent's uncertainty about the environment. The presented algorithm was evaluated by a range of simulated tasks with sparse extrinsic reward signals, where only limited demonstrated trajectories were provided to each task. Superior exploration efficiency and high average return were demonstrated in all tasks. Furthermore, it was found that the agent could imitate the expert's behavior and meanwhile sustain high return.

Accurately detecting tumors and estimating the depth of tumors is essential in the surgical removal of tumors. In robotic-assisted surgery, autonomous robotic palpation has the potential to provide more precise detection, tumors' depth estimation, and less intrusion when normal tissues surround tumors. In this article, by mimicking the human finger touch, we propose a tactile sensing-based deep recurrent neural network (DRNN) with long short-term memory (LSTM) architecture to improve the accuracy of the detection and depth estimation of tumors embedded in soft tissue. In the experimental setup, the hard inclusions simulate the tumors, while the phantom tissue is fabricated by silicon to simulate the soft tissue. During the experiment, the data from the force sensor and displacement of the robot palpation probe are for detection and depth estimation purposes. The collected sequential data set of the force and the displacement of the probe during one completed palpation process will go through the proposed DRNN network with deep LSTM architecture, in which the temporal dependencies of the sequential data will be captured in the cell states in the deep LSTM layers. Subsequently, the softmax classifier is adopted to determine if there is any hard inclusion exists and offer the depth estimation of the hard inclusions. Experiments based on 396 real data sets demonstrate that the detection accuracy for the testing data set is 99.2% and the depth estimation accuracy for the testing data set is 95.8%. The accuracy of the proposed method is best when comparing with other widely used methods. Note to Practitioners-The palpation of tumors motivated this article in the robot-assisted surgical systems through tactile feedback. In order to mimic the human touch on the soft tissue, this article presents a deep-learning-based approach to estimate the depth of the hard inclusions in the phantom tissue through force information. The displacement of the palpation probe and the touch force during one palpation are recorded as data sequences to train the deep model, which aims to capture dynamics and long-term dependence of the palpation process. In this article, we made the first successful attempt to accurately estimate the depth of the hard inclusions buried at different locations of the phantom tissue using only force information. The proposed approach can work in different robot-assisted scenarios, such as master-slave robotic surgery. In the clinic applications, the force sensor will be integrated at the end-effector of the robotic manipulator. According to the specific requirements, the force sensor and the robotic manipulator might be different from those used in this article. For some applications, such as the laparoscopic interventions, the complete vertical contact tends to be difficult to obtain due to the laparoscopic port effects. The projection of the recorded force data and displacement can obtain the information in the normal direction. The future work is going to be extended to tissue environments with arbitrary surface and tumors with various shapes/depths for more complex and prospective clinical applications.

Surgical robotic systems with miniaturized flexible Tendon-driven Serpentine Manipulators (TSM) have enjoyed increasing popularities among surgeons and researchers for their advantages of working in constrained and torturous human lumen such as oral cavity and upper GI tract. However, they suffer from sufficient nonlinearities and model uncertainties due to friction, tension varying, tendon slacking, etc. Model based control is insufficient to overcome such uncertainties and automate challenging surgical related tasks. The objective of this work is to automate certain clinical tasks to alleviate surgeon fatigue and promote task efficiency in kinematics free and sensor free circumstances. We present a data-driven approach based on Learning from Demonstration (LfD), which utilizes statistical machine learning models to encode system underlying dynamics and generalize smooth motor trajectories by direct actuation space learning. Motion segmentation is enabled with soft margin Support Vector Machine (soft-SVM) in complicated tasks. We also make attempts to retrieve task-specific properties by Locally Weighted Regression (LWR). We evaluated the approach on two surgical related tasks: compliant insertion and simplified Endoscopic Submucosal Dissection (ESD). The flexible TSM successfully reproduced both tasks and demonstrated superior trajectory performance. A video is available at: https://youtu.be/rLQo6xKtyMI.

Flexible serpentine manipulators are widely used in surgical robots as it can be operated inside the patient's body cavity by backbone bending. However, during the bending the manipulator sweeps over a region, where sensitive organs may locate. This raises the safety concern. In this paper, a motion planning algorithm focusing on minimize the sweeping area for flexible serpentine manipulators is presented. Particularly, a three dimensional backward average neural dynamic model (BANDM) is proposed to build minimum sweeping area planning field in the configuration space of the serpentine manipulator. Given a target position, the motion sequence is generated automatically based on the established planning field. The simulations and experimental results validate the effectiveness and superiority of the proposed planning approach over conventional planning algorithms in terms of sweeping area with keeping target reach and obstacle avoidance.