Robot assisted endoluminal intervention is an emerging technique for both benign and malignant luminal lesions. With vision-based navigation, when combined with pre-operative imaging data as priors, it is possible to recover position and pose of the endoscope without the need of additional sensors. In practice, however, aligning pre-operative and intra-operative domains is complicated by significant texture differences. Although methods such as style transfer can be used to address this issue, they require large datasets from both source and target domains with prolonged training times. This paper proposes an efficient domain transfer method based on stylized Gaussian splatting, only requiring a few of real images (10 images) with very fast training time. Specifically, the transfer process includes two phases. In the first phase, the 3D models reconstructed from CT scans are represented as differential Gaussian point clouds. In the second phase, only color appearance related parameters are optimized to transfer the style and preserve the visual content. A novel structure consistency loss is applied to latent features and depth levels to enhance the stability of the transferred images. Detailed validation was performed to demonstrate the performance advantages of the proposed method compared to that of the current state-of-the-art, highlighting the potential for intra-operative surgical navigation.

Invasive flexible neural electrodes are becoming increasingly prevalent in monitoring and modulating brain neural activity, necessitating the precise and minimally invasive implantation of these electrodes to a depth of a few millimeters beneath the cerebral surface. Although Neuralink has pioneered robot-assisted neural electrode implantation guided by microscopy, it currently lacks the ability to detect non-cerebral surface microvessels that are invisible under the white-light microscope, leading to inaccurate implantation planning and a high risk of trauma. To address this limitation, we introduce a vascular-enhanced strategy that fuses intraoperative white-light microscopy and preoperative photoacoustic microscopy and applies the fusion results to our established microsurgical robotic system for brain electrode implantation. Specifically, a multi-modality data preprocessing pipeline is devised to extract representative features, and a 2.5D fusion network that incorporates a depth encoding mechanism is proposed to predict cross-modality correspondence. The enhanced fusion results are utilized for implantation planning and intraoperative guidance during in vivo surgical procedures. Both quantitative and qualitative results are presented to demonstrate the effectiveness of our proposed cross-modality fusion methods. Furthermore, in vivo surgical implementations on mice underscore the potential of the proposed approach for achieving more precise and minimally invasive brain electrode implantation.

Robot assisted endoluminal intervention is an emerging tool for treating luminal lesions. Vision-based endoluminal navigation, particularly through video-CT registration, is a tangible way of obtaining absolute camera position information. By using pre-operative CT data, accurate endoscope localization can be achieved, without the need of additional tracking hardware intraoperatively. However, aligning preoperative CT with intraoperative domain remains a challenge. Although approaches such as style transfer have been explored, patient-specific textures and intra-operative artifacts can significantly complicate the task. To overcome these challenges, we propose R2Nav, a robust, real-time test time adaptation method for endoluminal navigation. R2Nav constructs a confidence buffer during the testing phase, refining the model only for frames with high uncertainty. We introduce a registration-augmented model refinement strategy, which enhances both accuracy and efficiency of the system by selecting relevant training samples from the virtual gallery. Additionally, we propose a novel warm-up strategy for the registration encoder during the initial testing phase, enabling the extraction of more robust features when the model is suboptimal. Extensive validation demonstrates that R2Nav outperforms the current state-of-the-art methods, offering significant advantages for real-time, intra-operative endoluminal navigation. Code is at: https://github.com/EndoluminalSurgicalVision-IMR/R2Nav.

Continuum robots are widely used in the medical scenarios due to their dexterity and flexibility. However, precise end-to-end control of continuum robots remains challenging, limited by the kinematic or kinetostatic accuracy and no enough space for additional sensors configurations. This paper proposes a precise position control method for fiber-driven continuum robots using the reconstructed shape based on distributed force sensing from the same fibers, where the optical fibers serve as both robot actuation and force sensing simultaneously without requiring additional sensors. First, we use single-core optical fibers (SCFs) as the actuation cables of the continuum robot, and each fiber has multiple fiber Bragg grating (FBG) sensors inscribed on it to sense distributed force along the entire cables. Then, the forward kinetostatics model of the fiber-driven continuum robot is established using the known distributed forces as the inputs. Notably, the nonlinear friction between the cables and actuation channels does not require an additional estimation model. Benefiting from this, the shape can be accurately reconstructed after the stiffness calibration of the continuum robot. Finally, a position controller based on real-time feedback from shape is developed to achieve the tip position control of the continuum robot. Experimental results demonstrate that the proposed forward kinetostatics model can achieve the shape reconstruction with the errors of 0.45 mm and 0.57 mm in planar bending and spatial bending states, respectively. By comparison to the traditional constant curvature kinematics-based control method, the proposed methods can achieve the mean absolute error of 0.37 and 0.6 mm in two distinct path tracking tests. The proposed method using distributed forces sensing enables a real-time accurate position feedback control combined with kinetostatic model, instead of modelling the nonlinear friction or adding additional external sensors.

Autonomous suturing is a critical challenge in robot-assisted surgery, where accurate segmentation and pose estimation of suturing threads are essential prerequisites. However, suturing threads are easily occluded by moving instruments and embedded in deformable tissues which make the task much more challenging. To address this, we propose a coarse-to-fine network for detailed segmentation and pose estimation of suturing threads. The coarse stage aims to capture global thread structure, while the fine stage refines the detailed structure through error residual correction. A spatial context fusion module is incorporated to improve the perception of occluded regions, and weighted balanced cross entropy loss as well as hard sample mining strategy is implemented to enhance small target segmentation performance. To deal with severe occlusions, topological constraints are utilized to effectively identify and reconstruct invisible thread segments. Experiments have been conducted on three datasets collected from different surgical scenes including phantom, endoscopy, and microsurgery. Both quantitative and qualitative results have demonstrated that our proposed framework outperforms baseline methods on segmentation and pose estimation of suturing threads, particularly in detecting occluded threads. Our proposed framework generalizes well across different surgical scenarios, showing its potential for automatic suturing.

Deformation of flexible robots can be practically assessed using extension/compression, shear, curvature, and torsion. Sensing based on one or more of the above characteristics enables closed-loop control for delicate tasks that require precision and dexterity. Due to the increasing popularity of flexible robotics in recent years, significant research effort has been directed to this burgeoning field. Although numerous studies have addressed soft sensing technologies, their successful integration into flexible robotic systems remains limited. This article provides a comprehensive review of sensing methods, from multidimensional deformation to the underlying principles of deriving hard-to-measure deformation from surrogate parameters. It focuses on sensing modalities such as strain measurement via piezoelectric, capacitive, resistive, and optical techniques. The applications of deformation sensing in industrial and service robotics are described. Future challenges and potential research issues including resolution, conformability, multifunctionality, crosstalk, and miniaturization are discussed. The need for a synergistic approach across disciplines is highlighted, emphasizing the integration of new materials, microstructures, advanced manufacturing technologies, and state-of-the-art signal processing techniques.

Predicting accurate human pose from 3-D visual observation presents a formidable challenge in computer vision, with numerous applications across various industries. However, most existing studies tackled this issue by regressing the 3-D pose from depth maps via 2-D convolutional neural networks or parametric human models, with limited development in point cloud-based methods. To this end, we propose PoseSDF++, i.e., a point cloud-based encoder–decoder network utilizing implicit neural representation to perform 3-D human pose estimation (HPE) and nonparametric shape reconstruction simultaneously. Leveraging the representative capacity of the signed distance function (SDF), we conceptualize the 3-D HPE as a multiple-shape reconstruction task and propose a distance-aware regression method to accurately estimate the 3-D joint positions. In specific, our PoseSDF++ consists of three modules: first,a hierarchical encoderwith vector neuron layers extracts the multiscale rotation equivariant features from the point clouds captured from an arbitrary viewpoint, addressing the degradation issue caused by viewpoint variation of implicit representation; second,a shape decodermaps the extracted feature and the query to its corresponding shape SDF; third,a pose decodercomputes the distance between the query and the target keypoints, namely, the pose SDF. Extensive experiments on four publicly available datasets demonstrate that our PoseSDF++ achieves competitive performance against the state-of-the-art point cloud-based methods and covering the human hand (HANDS 2019), lower limbs (ICL-Gait), and full body (DFAUST, LiDARHuman2.6M) pose estimation.

For the development of robotic systems for micromanipulation, it is challenging to design appropriate control strategies due to either the lack of sufficient information for feedback or the difficulty in extracting subtle yet critical visual features. With the same system under the teleoperated mode, however, human operators seem to be able to complete the task more successfully with an inherent motion and control strategy. The extraction of implicit human attention during the task and integration of this with robot control could provide crucial guidance in the design of feature extraction and motion control algorithms. In this paper, a micro-assembly task of miniature thin membrane sensors is considered. For human demonstrations, we collected data from repeated tests performed by ten operators following three motion strategies. The human attention during the task is explored according to the coordinates of the eye gaze, and then a neural network with gaze-guided attention is trained to segment the visual Region of Interest (ROI). After quantitative evaluation of operator results in terms of success rate, efficiency, reset time, and the Index of Pupillary Activity (IPA), an optimized motion strategy based on the "palpation" framework was derived. Consequently, we apply this strategy to automated tasks and achieve superior results than human operators, showing an average task completion time of 34.8±5.9s and a success rate of over 90%.

Photoacoustic Microscopy (PAM) is a relatively new imaging modality in biomedicine. However, point-by-point raster scanning in PAM suffers from low imaging speed. Sparse sampling has been studied in recent years and with the development of deep learning algorithms, extensive efforts have been devoted to sparse image reconstruction while little attention has been paid to sparse sampling trajectory design required for actual implementation. The use of real-time adaptive robotically controlled sampling with micro-scale accuracy with due consideration of physical constraints can pave the way for using PAM for robot-assisted microsurgery. This work proposes a fast PAM scheme with robot-controlled microtrajectory optimization. The proposed method is adaptive to imaging details of different regions of interest (ROI) and detailed experiments have been conducted on both simulation and in-vivo settings. Results show that our proposed method can achieve faster scanning speed than traditional raster scanning and improved image quality in ROI than the standard spiral trajectory, which demonstrates the effectiveness of our proposed method and its potential to be deployed in other point-by-point scanning systems.

Motor Imagery (MI) Electroencephalography (EEG) is one of the most common Brain-Computer Interface (BCI) paradigms that has been widely used in neural rehabilitation and gaming. Although considerable research efforts have been dedicated to developing MI EEG classification algorithms, they are mostly limited in handling scenarios where the training and testing data are not from the same subject or session. Such poor generalization capability significantly limits the realization of BCI in real-world applications. In this paper, we proposed a novel framework to disentangle the representation of raw EEG data into three components, subject/session-specific, MI-task-specific, and random noises, so that the subject/session-specific feature extends the generalization capability of the system. This is realized by a joint discriminative and generative framework, supported by a series of fundamental training losses and training strategies. We evaluated our framework on three public MI EEG datasets, and detailed experimental results show that our method can achieve superior performance by a large margin compared to current state-of-the-art benchmark algorithms.

Cable-driven continuum robots are widely used for endoluminal intervention because of their dexterity and shape conforming steerability. However, body contact between the continuum robot and its surrounding anatomy is unavoidable, which imposes a potential safety risk, including vessel wall damage or even perforation. This paper presents an approach for body contact estimation of continuum robots with tension-profile sensing of actuation fibers. First, tension-sensing optical fibers with multiple inscribed fiber Bragg grating (FBG) sensors are used for both actuation and in-situ sensing of the continuum robot. Second, a beam theory-based mechanical model considering segmental differences, multiple fiber interactions and external force interactions is established, followed by robust estimation of contact positions and forces. Finally, detailed simulations are conducted to validate the accuracy and effectiveness of the proposed method. Experiments on a notched continuum robot are carried out, and the results show that the proposed approach can effectively recover in-situ segmental actuation forces without the need of explicit modeling of the friction between the fibers and guiding channels. The method enables the estimation of the number of contact points, as well as contact positions and contact forces along the body of the continuum robot.

Human movement intention recognition is important for human-robot interaction. Existing work based on motor imagery electroencephalogram (EEG) provides a non-invasive and portable solution for intention detection. However, the data-driven methods may suffer from the limited scale and diversity of the training datasets, which result in poor generalization performance on new test subjects. It is practically difficult to directly aggregate data from multiple datasets for training, since they often employ different channels and collected data suffers from significant domain shifts caused by different devices, experiment setup, etc. On the other hand, the inter-subject heterogeneity is also substantial due to individual differences in EEG representations. In this work, we developed two networks to learn from both the shared and the complete channels across datasets, handling inter-subject and inter-dataset heterogeneity respectively. Based on both networks, we further developed an online knowledge co-distillation framework to collaboratively learn from both networks, achieving coherent performance boosts. Experimental results have shown that our proposed method can effectively aggregate knowledge from multiple datasets, demonstrating better generalization in the context of cross-subject validation.

Egocentric vision has gained increasing popularity in social robotics, demonstrating great potentials for personal assistance and human-centric behavior analysis. Holistic per-ception of human body itself is a prerequisite for downstream applications, including action recognition and anticipation. Extensive research has been performed for human mesh recovery from the exocentric images captured from a third-person view, but limited studies are conducted for heavily distorted yet occluded egocentric images. In this paper, we propose Egocentric Human Mesh Recovery (EgoHMR), a novel hierarchical network based on latent diffusion models. Our method takes a single egocentric frame as the input and it can be trained in an end-to-end manner without supervision of 2D pose. The network is built upon the latent diffusion model by incorporating both global and local features in a hierarchical structure. To train the proposed network, we generate weak labels from synchronized exocentric images. The proposed method can perform human mesh recovery directly from egocentric images and detailed quantitative and qualitative experiments have been conducted to demonstrate the effectiveness of the proposed EgoHMR method.

Endobronchial intervention is increasingly used as a minimally invasive means for the treatment of pulmonary diseases. In order to reduce the difficulty of manipulation in complex airway networks, robust lumen detection is essential for intraoperative guidance. However, these methods are sensitive to visual artifacts which are inevitable during the surgery. In this work, a cross domain feature interaction (CDFI) network is proposed to extract the structural features of lumens, as well as to provide artifact cues to characterize the visual features. To effectively extract the structural and artifact features, the Quadruple Feature Constraints (QFC) module is designed to constrain the intrinsic connections of samples with various imaging-quality. Furthermore, we design a Guided Feature Fusion (GFF) module to supervise the model for adaptive feature fusion based on different types of artifacts. Results show that the features extracted by the proposed method can preserve the structural information of lumen in the presence of large visual variations, bringing much-improved lumen detection accuracy.

Eye gaze can convey rich information of human intentions, which enables the social robots to comprehend the cognition and behavior of human targets. However, the existing 3D gaze estimation methods generally have high requirements either on the dedicated hardware or the quantity and quality of training databases, which largely limits their practical application values. This paper proposes EasyGaze3D, an effective 3D gaze estimation framework using a single RGB camera. First, the framework detects the 2D facial landmarks and recovers the 3D facial shape from the input image, and derives the required camera parameters with these features. Then, without loss of generality, the gaze direction can be regarded as the vector pointing from the eyeball center to the pupil center, which are derived respectively from the detected facial landmarks and the spherical fitting performed on the recovered 3D facial shape. Besides, we propose a flexible yet efficient calibration module, namely Easy-Cali, for deriving the subject-specific 3D facial shape and eyeball centers. The features calibrated by Easy-Cali can further boost the performance of EasyGaze3D. Experimental results show that our proposed method, being plug-and-play and without the need of training on large-scale dataset, can achieve superior performance against the existing methods based on deep models.

Implantable devices represent the future of remote medical monitoring and administration of both chemical and physical therapies to the patients. Although some of these devices are already in the market, the security mechanisms deployed inside them to withstand deliberate external influence are still decades away from the robust digital data security schemes employed in modern distributed networks these days. Medical data theft, spoofing, and disclosure pose serious threats that can ultimately lead to individual and social stigmas or even death. In this article, we present a small-form and batteryless implantable device with acquisition channels for biopotential (30-dB gain and 16-Hz bandwidth), arterial pulse oximetry, and temperature (0.12°C accuracy) recordings, suitable for cardiovascular, neuronal, and endocrine parameters assessment. The proposed device is powered by the near-field communication (NFC) interface with an external mobile phone, with a power consumption of 0.9 mW and achieving the full operation for distances close to 1 cm under the skin. In situ encryption of the acquired physiological signals is performed by a lightweight and short-term symmetric-key distribution scheme with data stream hopping, in order to ensure secure data transference over the air between the patient and trusted entities only, complemented by data storage, processing, and recovery through a medical blockchain type of network that involves the main stakeholders inside a medical community.

Endomicroscopy is an emerging imaging modality for real-time optical biopsy. One limitation of existing endomicroscopy based on coherent fibre bundles is that the image resolution is intrinsically limited by the number of fibres that can be practically integrated within the small imaging probe. To improve the image resolution, Super-Resolution (SR) techniques combined with image priors can enhance the clinical utility of endomicroscopy whereas existing SR algorithms suffer from the lack of explicit guidance from ground truth high-resolution (HR) images. In this article, we propose an unsupervised SR pipeline to allow stable offline and kernel-generic learning. Our method takes advantage of both internal statistics and external cross-modality priors. To improve the joint learning process, we present a Sharpness-aware Contrastive Generative Adversarial Network (SCGAN) with two dedicated modules, a sharpness-aware generator and a contrastive-learning discriminator. In the generator, an auxiliary task of sharpness discrimination is formulated to facilitate internal learning by considering the rankings of training instances in various sharpness levels. In the discriminator, we design a contrastive-learning module to mitigate the ill-posed nature of SR tasks via constraints from both positive and negative images. Experiments on multiple datasets demonstrate that SCGAN reduces the performance gap between previous unsupervised approaches and the upper bounds defined in supervised settings by more than 50%, delivering a new state-of-the-art performance score for endomicroscopy super-resolution. Further application on a realistic Voronoi-based pCLE downsampling kernel proves that SCGAN attains PSNR of 35.851 dB, improving 5.23 dB compared with the traditional Delaunay interpolation.

Magnetic Resonance Elastography (MRE) can characterize biomechanical properties of soft tissue for disease diagnosis and treatment planning. However, complicated wavefields acquired from MRE coupled with noise pose challenges for accurate displacement extraction and modulus estimation. Using optimization-based displacement extraction and Traveling Wave Expansion-based Neural Network (TWENN) modulus estimation, we propose a new pipeline for processing MRE images. An objective function with Dual Data Consistency (Dual-DC) has been used to ensure accurate phase unwrapping and displacement extraction. For the estimation of complex wavenumbers, a complex-valued neural network with displacement covariance as an input has been developed. A model of traveling wave expansion is used to generate training datasets for the network with varying levels of noise. The complex shear modulus map is obtained through fusion of multifrequency and multidirectional data. Validation using brain and liver simulation images demonstrates the practical value of the proposed pipeline, which can estimate the biomechanical properties with minimal root-mean-square errors when compared to state-of-the-art methods. Applications of the proposed method for processing MRE images of phantom, brain, and liver reveal clear anatomical features, robustness to noise, and good generalizability of the pipeline.

Detailed anatomical labeling of bronchial trees extracted from CT images can be used as fine-grained maps for intra-operative navigation. To cater to the sparse distribution of airway voxels and large class imbalance in 3D image space, a graph-neural-network-based method is proposed to map branches to nodes in a graph space and assign anatomical labels down to subsegmental level. To address the inherent problem of overlapping distribution of positional and morphological features, especially for subsegmental categories, the proposed method focuses on the relative position between sibling subsegments which is fixed in most cases. The hierarchical nomenclature is represented by multi-level labeling and each category is associated with one or two subtrees in the graph. Hyperedges are used to extract the representation of subtrees while a hypergraph neural network is developed to encode their intrinsic relationship through hyperedge interaction. A filter module is further designed to guide feature aggregation between nodes and hyperedges. With the proposed method, the final accuracies for segmental and subsegmental node classification can achieve 93.6% and 82.0% respectively. The corresponding code is publicly available at https://github.com/haozheng-sjtu/airway-labeling.

Egocentric vision has gained increasing popularity recently, opening new avenues for human-centric applications. However, the use of the egocentric fisheye cameras allows wide angle coverage but image distortion is introduced along with strong human body self-occlusion imposing significant challenges in data processing and model reconstruction. Unlike previous work only leveraging synthetic data for model training, this paper presents a new real-world EgoCentric Human Pose (ECHP) dataset. To tackle the difficulty of collecting 3D ground truth using motion capture systems, we simultaneously collect images from a head-mounted egocentric fisheye camera as well as from two third-person-view cameras, circumventing the environmental restrictions. By using self-supervised learning under multi-view constraints, we propose a simple yet effective framework, namely EgoFish3D, for egocentric 3D pose estimation from a single image in different real-world scenarios. The proposed EgoFish3D incorporates three main modules. 1)The third-person-view moduletakes two exocentric images as input and estimates the 3D pose represented in the third-person camera frame; 2)the egocentric modulepredicts the 3D pose in the egocentric camera frame; and 3)the interactive moduleestimates the rotation matrix between the third-person and the egocentric views. Experimental results on our ECHP dataset and existing benchmark datasets demonstrate the effectiveness of the proposed EgoFish3D, which can achieve superior performance to existing methods.

Tip force sensing for cable-driven continuum robots are vital to provide the force information for safe and reliable human-robot interaction. However, traditional triaxial force sensors usually have a complicated structure occupying its inner lumen, without enough space for additional instrumental tools. To solve this, this paper proposes a fixed and sliding fiber Bragg grating (FBG) sensors-based triaxial force sensing method for cable-driven continuum robots. The fixed FBG sensors are attached to the circumferential surface of continuum robot at the tip and base, and the sliding optical fibers with FBG sensors are located in the actuation channels as the sensing integrated pulling cables. This configuration guarantees a compact structure and large inner lumen. Two five-degreed-of-freedom (5-DOF) electromagnetic (EM) and a 6-DOF EM sensors are assembled to the tip and the base of the robot respectively, which can obtain the pose of the tip with respect to the base. The tip force in three directions can be decoupled using the information of the Bragg wavelength changes and EM sensors. Results show that the mean errors of force sensing along x-direction, y-direction, and z-direction are 4.1%, 4.7%, and 9.8%, respectively. The proposed sensing method does not rely on the elasticity of continuum robot, enabling its wide applicability for other cable-driven pseudo-continuum robots.

Vision-based 3D human pose estimation and shape reconstruction play important roles in robot-assisted healthcare monitoring and personal assistance. However, 3D data captured from a single viewpoint always encounter occlusions and exhibit substantial heterogeneity across different views, resulting in significant challenges for both tasks. Extensive approaches have been proposed to perform each task separately, but few of them present a unified solution. In this paper, we propose a novel network based on signed distance functions, namely PoseSDF, to simultaneously reconstruct 3D lower limb shape and estimate gait pose by two dedicated branches. To promote multi-task learning, several strategies are developed to ensure that these two branches leverage the same latent shape code while exchanging information between them. More importantly, an auxiliary RotNet is incorporated into the inference phase, overcoming the inherent limitations of implicit neural functions under cross-view scenarios. Experimental results demonstrate that our proposed PoseSDF can achieve both high-quality shape reconstruction and precise pose estimation, generalizing well on the data from novel views, gait patterns, as well as real-world.

Human-robot shared control, which integrates the advantages of both humans and robots, is an effective approach to facilitate efficient surgical operation. Learning from demonstration (LfD) techniques can be used to automate some of the surgical sub tasks for the construction of the shared control mechanism. However, a sufficient amount of data is required for the robot to learn the manoeuvres. Using a surgical simulator to collect data is a less resource-demanding approach. With sim-to-real adaptation, the manoeuvres learned from a simulator can be transferred to a physical robot. To this end, we propose a sim-to-real adaptation method to construct a human-robot shared control framework for robotic surgery. In this paper, a desired trajectory is generated from a simulator using LfD method, while dynamic motion primitives (DMP) is used to transfer the desired trajectory from the simulator to the physical robotic platform. Moreover, a role adaptation mechanism is developed such that the robot can adjust its role according to the surgical operation contexts predicted by a neural network model. The effectiveness of the proposed framework is validated on the da Vinci Research Kit (dVRK). Results of the user studies indicated that with the adaptive human-robot shared control framework, the path length of the remote controller, the total clutching number and the task completion time can be reduced significantly. The proposed method outperformed the traditional manual control via teleoperation.

In endoluminal surgery, the miniature instruments shall be of high accuracy and flexibility for minimal invasive diagnosis and surgical intervention. To this end, continuum robots with flexible joints have been proposed as the mechanism of endoscopic instruments. The compliance and deformability of the continuum joints enable access into the curved lumen. However, the manufacturing tolerances are normally not considered in the design procedure, and led to inaccuracy in the robotic control. To improve the control accuracy and flexibility of endoluminal surgical robots, we propose a novel design of a metal printed continuum joint in this paper, which incorporates a variable pitch design into the spring-like structure. The design can reduce the position errors accumulated on the distal tip of the joint, especially at large bending angles. The specification of variable pitch is investigated and determined with a friction model. In addition, to eliminate the distortion of the joint induced during the metal printing process, an extensive experiment was conducted to access the effect of the variables in the design (pitch, thickness, width and number of coils), with the aim of determining optimal parameters for reducing discrepancy caused by manufacturing variations. The final results indicated that the bending error of a single joint can be reduced from 18.10% to 4.63%, and a multi-segment prototype was developed to verify its effectiveness for potential surgical applications.

Percutaneous coronary intervention (PCI) involves the delivery of a flexible submillimeter guidewire and existing x- ray based approaches impose significant ironing radiation. The use of magnetic resonance imaging (MRI) for intraoperative guidance has the advantages of not only being safe but also having high positioning accuracy and excellent tissue contrast. This paper develops a pneumatically driven MR-conditional delivery mechanism for the ease of manipulation of the guidewire in vivo. It incorporates newly developed rotary pneumatic step motors and a pneumatic slip ring for actuation and decoupling of translational and rotational motions. An effective clamping mechanism for the locking and releasing of the guidewire is also incorporated. The proposed pneumatic slip ring mechanism decouples six gas lines, where four are used to supply a pneumatic step motor for translational motion and two for the clamping mechanism. High friction sil sleeve is used to hold the guidewire firmly. The rotary pneumatic motor has excellent sealing and stability, providing an output torque of 15.75 Nm/MPa. Experiments show that the average error of translational motion is 0.37 mm. Real-time MRI-guided endovascular intervention is performed in a vascular phantom with pulsatile flows to validate its potential clinical use. The imaging artifact test under MRI shows no noticeable distortion and the loss of Signal-to-Noise Ratio (SNR) is less than 2%.

Egocentric vision is an emerging topic, which has demonstrated great potential in assistive healthcare scenarios, ranging from human-centric behavior analysis to personal social assistance. Within this field, due to the heterogeneity of visual perception from first-person views, egocentric pose estimation is one of the most significant prerequisites for enabling various downstream applications. However, existing methods for egocentric pose estimation mainly focus on predicting the pose represented in the camera coordinates from a single image, which ignores the latent cues in the temporal domain and results in less accuracy. In this paper, we propose Ego+X, an egocentric vision based system for 3D canonical pose estimation and human-centric social interaction characterization. Our system is composed of two head-mounted egocentric cameras, where one is faced downwards and the other looks outwards. By leveraging the global context provided by visual SLAM, we first propose Ego-Glo for spatial-accurate and temporal-consistent egocentric 3D pose estimation in the canonical coordinate system. With the help of an egocentric camera looking outwards, we then propose Ego-Soc by extending Ego-Glo to various social interaction tasks, e.g., object detection and human-human interaction. Quantitative and qualitative experiments have been conducted to demonstrate the effectiveness of our proposed Ego+X.

Ever since their first introduction in the late 1980s[1],[2], surgical robots have played an increasingly prominent role in medical practice[3],[4]. For example, a recent study[5]found that over 15% of all general surgery procedures in 2020 were performed robotically, compared to only 1.8% in 2012. The current worldwide robotic surgery market is estimated to be$\$ $5.3 billion and is expected to reach$\$ $19 billion by 2027, with a compound annual growth rate over 21%[6].

Most robotic systems designed for mass manufacturing are optimized for a specific type of product. They generally lack the ability to adapt to low-volume customized products. In this article, we present a system based on a modular design for manufacturing personalized medical stent graft implants. The concept is based on learning-by-demonstration by integrating real-time 3-D vision, multirobot collaboration, and personalization to guide the robots to learn and execute tasks continuously with adaptation to different implant geometry. The system is optimized to generate customized and collision-free paths for efficient object manipulation and task completion. We show that the system is generalizable to different stent graft designs and the proposed multirobot system can seamlessly work together with high efficiency without collisions. The results have also suggested its usability for other manipulation tasks, especially for flexible production of customized products where bimanual or multirobot cooperation is required.Note to Practitioners—The motivation of this article is the problem of automatic sewing of personalized stent grafts (a tailor-made artificial vessel). Existing personalized stent grafts are mostly hand sewn, which is time consuming and often undersupplied. Automating such process can significantly improve the production and this requires a sewing system that can handle different designs. This article suggests a new schema to design a robotic system to handle personalized designs. The first methodology is modularized design to separate the task into a repetitive part and a personalized part, each handled by a module. The second methodology is to find the best relative pose between the modules such that the robots can complete their task within their working space and with minimum motion. This ensures that a stent graft can be sewn feasibly and with the lowest cost. Computational results show this approach can find optimal solution for different personalized stent grafts and preliminary on robot experiment verifies that this approach is feasible. Please note that this approach is not limited to sewing personalized stent graft. The schema can be applied to solve similar problem of customized product motion planning and system design.

Monitoring the mental workload of operators is of paramount importance in space telerobotic training and other teleoperation tasks. Instead of the estimation of task-specific workload, this article aims at investigating the impact of two significant confounding factors (time-pressure and latency) on space teleoperation and explored the use of eye-tracking technology for factor-induced mental workload estimation and performance evaluation. Ten subjects teleoperated a Canadarm2 robot to complete a complex on-orbit assembly task in our photo-realistic training simulator while wearing a head-mounted eye-tracker. To understand how time-pressure and latency influence eye-tracking features works, we first performed the statistical analysis on various features with respect to a single factor and across multiple groups. Next, eye-tracking features extracted from segment data and trial data is used to identify the mental workload induced by confounding factors, which can be used for developing personalized training programs and guaranteeing safe teleoperation. Furthermore, to improve the recognition performance using segment data, we propose the activity ratio and time ratio to characterize the informative segments. Finally, the relationship between simulator-defined performance measures and eye-tracking features is examined. Results show that fixation duration, saccade frequency and duration, pupil diameter, and index of pupillary activity are significant features that can be used in both factor-induced mental workload estimation and task performance evaluation.

Accurate lower-limb pose estimation is aprerequisite of skeleton based pathological gait analysis. To achieve this goal in free-living environments for long-term monitoring, single depth sensor has been proposed in research. However, the depth map acquired from a single viewpoint encodes only partial geometric information of the lower limbs and exhibits large variations across different viewpoints. Existing off-the-shelf 3D pose tracking algorithms and public datasets for depth based human pose estimation are mainly targeted at activity recognition applications. They are relatively insensitive to skeleton estimation accuracy, especially at the foot segments. Furthermore, acquiring ground truth skeleton data for detailed biomechanics analysis also requires considerable efforts. To address these issues, we propose a novel cross-domain self-supervised complete geometric representation learning framework, with knowledge transfer from the unlabelled synthetic point clouds of full lower-limb surfaces. The proposed method can significantly reduce the number of ground truth skeletons (with only 1%) in the training phase, meanwhile ensuring accurate and precise pose estimation and capturing discriminative features across different pathological gait patterns compared to other methods.

Endobronchial intervention is increasingly used as a minimally invasive means for the treatment of pulmonary diseases. In order to acquire the position of bronchoscopy, vision-based localization approaches are clinically preferable but are sensitive to visual variations. The static nature of pre-operative planning makes mapping of intraoperative anatomical features challenging for learning-based methods using visual features alone. In this work, we propose a robust navigation framework based on Vision Kinematic Interaction (VKI) for monocular bronchoscopic videos. To address visual-imbalance between the virtual and real views of bronchoscopy images, a Visual Similarity Network (VSN) is proposed to extract domain-invariant features to represent the lumen structure from endoscopic views, as well as domain-specific features to characterize the surface texture and visual artefacts. To improve the robustness of online estimation of camera pose, we also introduce a Kinematic Refinement Network (KRN) that allows progressive refinement of camera pose estimation based on network prediction and robot control signals. The accuracy of camera localization is validated on phantom and porcine lung datasets from a robotically controlled endobronchial intervention system, with both quantitative and qualitative results demonstrating the performance of the techniques. Results show that the features extracted by the proposed method can preserve the structural information of small airways in the presence of large visual variations along with the much-improved camera localization accuracy. The absolute trajectory errors (ATE) on phantom data and porcine data are 8.01 mm and 8.62 mm respectively.

Representation learning is the critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the dataset and the lack of labels. In this paper, we propose a stochastic routing normalization and neighborhood embedding framework with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy. In order to align the low-level and mid-level of pCLE and histology domain, we firstly build the domain-specific normalization module with stochastic activation strategy considering both depth-wise and feature-wise criterion. For high-level features, the latent centers are learned from the histology domain as the template for feature matching. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image classification with limited training samples demonstrates that the proposed method can outperform previous works on domain alignment.

Though machine learning algorithms are able to achieve pattern recognition from the correlation between data and labels, the presence of spurious features in the data decreases the robustness of these learned relationships with respect to varied testing environments. This is known as out-of-distribution (OoD) generalization problem. Recently, invariant risk minimization (IRM) attempts to tackle this issue by penalizing predictions based on the unstable spurious features in the data collected from different environments. However, similar to domain adaptation or domain generalization, a prevalent non-trivial limitation in these works is that the environment information is assigned by human specialists, i.e. a priori, or determined heuristically. However, an inappropriate group partitioning can dramatically deteriorate the OoD generalization and this process is expensive and time-consuming. To deal with this issue, we propose a novel theoretically principled min-max framework to iteratively construct a worst-case splitting, i.e. creating the most challenging environment splittings for the backbone learning paradigm (e.g. IRM) to learn the robust feature representation. We also design a differentiable training strategy to facilitate the feasible gradient- based computation. Numerical experiments show that our algorithmic framework has achieved superior and stable performance in various datasets, such as Colored MNIST and Punctuated Stanford sentiment treebank (SST). Furthermore, we also find our algorithm to be robust even to a strong data poisoning attack. To the best of our knowledge, this is one of the first to adopt differentiable environment splitting method to enable stable predictions across environments without environment index information, which achieves the state-of-the-art performance on datasets with strong spurious correlation, such as Colored MNIST.

MRI-guided robotic systems are emerging platforms for minimally invasive intervention because of high positioning accuracy and excellent tissue contrast. MR safe encoders are critical components for closed-loop robotic control. This paper develops an MR safe absolute rotary encoder based on eccentric sheave and FBG sensors. The eccentric sheave transforms the rotational motion of the shaft to the bending deflection of the beam on which FBG sensors are integrated. A model is built by establishing the relationship of the kinematics of the sheave, the mechanical properties of the beam with unknown length, and the strain model of two Fiber Bragg Grating (FBG) sensors. A Pseudo-Rigid Body (PRB) 3R model is used to solve a set of constrained equations for accurate rotary encoding. A prototype is built to calibrate the parameters and validate the accuracy of the encoder and its MR compatibility. Results show that the maximum angular error is 1.6°, and the RMS error is 0.46°. MRI shows that no noticeable artifacts are observed, and the Signal to Noise Ratio (SNR) is not affected. The results demonstrate the potential of the proposed method for it to be integrated with MR safe robots with easy fabrication, compact structures, and continuous measurement.

Electrospinning has exhibited excellent benefits to treat the trauma for tissue engineering due to its produced micro/nano fibrous structure. It can effectively adhere to the tissue surface for long-term continuous therapy. This paper develops a robotic electrospinning platform for endoluminal therapy. The platform consists of a continuum manipulator, the electrospinning device, and the actuation unit. The continuum manipulator has two bending sections to facilitate the steering of the tip needle for a controllable spinning direction. Non-circular joint profile is carefully designed to enable a constant length of the centreline of a continuum manipulator for stable fluid transmission inside it. Experiments are performed on a bronchus phantom, and the steering ability and bending limitation in each direction are also investigated. The endoluminal electrospinning is also fulfilled by a trajectory following and points targeting experiments. The effective adhesive area of the produced fibre is also illustrated. The proposed robotic electrospinning shows its feasibility to precisely spread more therapeutic drug to construct fibrous structure for potential endoluminal treatments.

For gait analysis, especially for the detection of subtle gait abnormalities, the collected datasets involve high variability across subjects due to inherent biometric traits and movement behaviors, leading to limited detection accuracy and poor generalizability. To address this, we propose a novel deep multi-source Unsupervised Domain Adaptation (UDA) approach, namely Maximum Cross-Domain Classifier Discrepancy (MCDCD), which aims to improve the classification performance on the test subject (target domain) by leveraging the information from multiple labelled training subjects (source domains). Specifically, the proposed model consists of a feature extractor and a domain-specific category classifier per source domain. The former feature extractor learns to generate discriminative gait features. For the latter classifiers, we minimize the cross-entropy loss to accurately classify source samples, and simultaneously maximize a novel cross-domain discrepancy loss between any two category classifiers to minimize domain shift between multiple sources and the target domain. To validate the proposed MCDCD for detecting gait abnormalities on novel subjects, we collected both high-quality Motion capture (Mocap) and noisy Electromyography (EMG) data from eighteen subjects with both normal and imitated abnormal gaits. Experiment results using both data modalities demonstrate that the proposed approach can achieve superior performance in abnormal gait classification compared to baseline deep models and state-of-the-art UDA methods.

Training convolutional neural networks (CNNs) for segmentation of pulmonary airway, artery, and vein is challenging due to sparse supervisory signals caused by the severe class imbalance between tubular targets and background. We present a CNNs-based method for accurate airway and artery-vein segmentation in non-contrast computed tomography. It enjoys superior sensitivity to tenuous peripheral bronchioles, arterioles, and venules. The method first uses a feature recalibration module to make the best use of features learned from the neural networks. Spatial information of features is properly integrated to retain relative priority of activated regions, which benefits the subsequent channel-wise recalibration. Then, attention distillation module is introduced to reinforce representation learning of tubular objects. Fine-grained details in high-resolution attention maps are passing down from one layer to its previous layer recursively to enrich context. Anatomy prior of lung context map and distance transform map is designed and incorporated for better artery-vein differentiation capacity. Extensive experiments demonstrated considerable performance gains brought by these components. Compared with state-of-the-art methods, our method extracted much more branches while maintaining competitive overall segmentation performance. Codes and models are available at http://www.pami.sjtu.edu.cn/News/56.

Automated airway segmentation is a prerequisite for pre-operative diagnosis and intra-operative navigation for pulmonary intervention. Due to the small size and scattered spatial distribution of peripheral bronchi, this is hampered by a severe class imbalance between foreground and background regions, which makes it challenging for CNN-based methods to parse distal small airways. In this paper, we demonstrate that this problem is arisen by gradient erosion and dilation of the neighborhood voxels. During back-propagation, if the ratio of the foreground gradient to background gradient is small while the class imbalance is local, the foreground gradients can be eroded by their neighborhoods. This process cumulatively increases the noise information included in the gradient flow from top layers to the bottom ones, limiting the learning of small structures in CNNs. To alleviate this problem, we use group supervision and the corresponding WingsNet to provide complementary gradient flows to enhance the training of shallow layers. To further address the intra-class imbalance between large and small airways, we design a General Union loss function that obviates the impact of airway size by distance-based weights and adaptively tunes the gradient ratio based on the learning process. Extensive experiments on public datasets demonstrate that the proposed method can predict the airway structures with higher accuracy and better morphological completeness than the baselines.

For abnormal gait recognition, pattern-specific features indicating abnormalities are interleaved with the subject-specific differences representing biometric traits. Deep representations are, therefore, prone to overfitting, and the models derived cannot generalize well to new subjects. Furthermore, there is limited availability of abnormal gait data obtained from precise Motion Capture (Mocap) systems because of regulatory issues and slow adaptation of new technologies in health care. On the other hand, data captured from markerless vision sensors or wearable sensors can be obtained in home environments, but noises from such devices may prevent the effective extraction of relevant features. To address these challenges, we propose a cascade of deep architectures that can encode cross-modal and cross-subject transfer for abnormal gait recognition. Cross-modal transfer maps noisy data obtained from RGBD and wearable sensors to accurate 4-D representations of the lower limb and joints obtained from the Mocap system. Subsequently, cross-subject transfer allows disentangling subject-specific from abnormal pattern-specific gait features based on a multiencoder autoencoder architecture. To validate the proposed methodology, we obtained multimodal gait data based on a multicamera motion capture system along with synchronized recordings of electromyography (EMG) data and 4-D skeleton data extracted from a single RGBD camera. Classification accuracy was improved significantly in both Mocap and noisy modalities.

Representation learning is a critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the data set and the lack of labels. In this article, we propose a deep graph-based multimodal feature embedding (DGMFE) framework for medical image retrieval with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy (pCLE). We first build a multimodality graph model based on the visual similarity between pCLE data and reference histology images. The latent similar pCLE-histology pairs are extracted by walking with the cyclic path on the graph while the dissimilar pairs are extracted based on the geodesic distance. Given the similar and dissimilar pairs, the latent feature space is discovered by reconstructing the similarity between pCLE and histology images via deep Siamese neural networks. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image retrieval demonstrates that DGMFE can outperform previous works on feature learning. Especially, the top-1 accuracy in an eight-class retrieval task is 0.739, thus demonstrating a 10% improvement compared to the state-of-the-art method.

Master-slave systems for endovascular catheterization have brought major clinical benefits including reduced radiation doses to the operators, improved precision and stability of the instruments, as well as reduced procedural duration. Emerging deep reinforcement learning (RL) technologies could potentially automate more complex endovascular tasks with enhanced success rates, more consistent motion and reduced fatigue and cognitive workload of the operators. However, the complexity of the pulsatile flows within the vasculature and non-linear behavior of the instruments hinder the use of model-based approaches for RL. This paper describes model-free generative adversarial imitation learning to automate a standard arterial catherization task. The automation policies have been trained in a pre-clinical setting. Detailed validation results show high success rates after skill transfer to a different vascular anatomical model. The quality of the catheter motions also shows less mean and maximum contact forces compared to manual-based approaches.

Snake-like robots for endoscopic surgery make it possible to reach deep-seated lesions. With the use of small flexible tendon-driven instruments, it is possible to perform bimanual micro-surgical tasks that are challenging for standard endoscopic surgeries. Existing devices, however, lack articulated wrists and rolling motion of the end-effector. This paper presents a new instrument design with a distal-roll gripper for snake-like robots. The developed 5 DoFs miniaturized instruments with a diameter of 3 mm enable the deployment into narrow endoluminal channels. Issues related to actuation coupling, tendon slack, and backlash are addressed. Experimental results show that the distal-roll gripper can rotate 106°, and the actuated joints can achieve good repeatability and accuracy with the proposed compensation control scheme.

Accurate real-time catheter segmentation is an important pre-requisite for robot-assisted endovascular intervention. Most of the existing learning-based methods for catheter segmentation and tracking are only trained on smallscale datasets or synthetic data due to the difficulties of ground-truth annotation. Furthermore, the temporal continuity in intraoperative imaging sequences is not fully utilised. In this paper, we present FW-Net, an end-to-end and real-time deep learning framework for endovascular intervention. The proposed FW-Net has three modules: a segmentation network with encoder-decoder architecture, a flow network to extract optical flow information, and a novel flow-guided warping function to learn the frame-to-frame temporal continuity. We show that by effectively learning temporal continuity, the network can successfully segment and track the catheters in real-time sequences using only raw ground-truth for training. Detailed validation results confirm that our FW-Net outperforms stateof-the-art techniques while achieving real-time performance.

Accurate force sensing is important for endoluminal intervention in terms of both safety and lesion targeting. This paper develops an FBG-based force sensor for robotic bronchoscopy by configuring three FBG sensors at the lateral side of a conical substrate. It allows a large and eccentric inner lumen for the interventional instrument, enabling a flexible imaging probe inside to perform optical biopsy. The force sensor is embodied with a laser-profiled continuum robot and thermo drift is fully compensated by three temperature sensors integrated on the circumference surface of the sensor substrate. Different decoupling approaches are investigated, and nonlinear decoupling is adopted based on the cross-validation SVM and a Gaussian kernel function, achieving an accuracy of 10.58 mN, 14.57 mN and 26.32 mN along X, Y and Z axis, respectively. The tissue test is also investigated to further demonstrate the feasibility of the developed triaxial force sensor.

Robotic bronchoscopic intervention requires detailed 3D airway maps for both localisation and enhanced visualisation, especially at peripheral airways. Patient-specific airway maps can be generated from preoperative chest CT scans. Due to pathological abnormalities and anatomical variations, automatically delineating the airway tree with distal branches is a challenging task. In the paper, we propose a cascaded 2D+3D model that has been tailored for airway segmentation from pathological CT scans. A novel 2D neural network is developed to generate the initial predictions where the peripheral airways are refined by a 3D adversarial training model. A sampling strategy based on a sequence of morphological operations is employed for the concatenation of the 2D and 3D models. The method has been validated on 20 pathological CT scans with results demonstrating improved segmentation accuracy and consistency, especially in peripheral airways.

Deep Convolutional Neural Networks (DCNNs) are used extensively in medical image segmentation and hence 3D navigation for robot-assisted Minimally Invasive Surgeries (MISs). However, current DCNNs usually use down sampling layers for increasing the receptive field and gaining abstract semantic information. These down sampling layers decrease the spatial dimension of feature maps, which can be detrimental to image segmentation. Atrous convolution is an alternative for the down sampling layer. It increases the receptive field whilst maintains the spatial dimension of feature maps. In this paper, a method for effective atrous rate setting is proposed to achieve the largest and fully-covered receptive field with a minimum number of atrous convolutional layers. Furthermore, a new and full resolution DCNN - Atrous Convolutional Neural Network (ACNN), which incorporates cascaded atrous II-blocks, residual learning and Instance Normalization (IN) is proposed. Application results of the proposed ACNN to Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) image segmentation demonstrate that the proposed ACNN can achieve higher segmentation Intersection over Unions (IoUs) than U-Net and Deeplabv3+, but with reduced trainable parameters.

The recent development of Robot-Assisted Minimally Invasive Surgery (RAMIS) has brought much benefit to ease the performance of complex Minimally Invasive Surgery (MIS) tasks and lead to more clinical outcomes. Compared to direct master-slave manipulation, semi-autonomous control for the surgical robot can enhance the efficiency of the operation, particularly for repetitive tasks. However, operating in a highly dynamic in-vivo environment is complex. Supervisory control functions should be included to ensure flexibility and safety during the autonomous control phase. This paper presents a haptic rendering interface to enable supervised semi-autonomous control for a surgical robot. Bayesian optimization is used to tune user-specific parameters during the surgical training process. User studies were conducted on a customized simulator for validation. Detailed comparisons are made between with and without the supervised semi-autonomous control mode in terms of the number of clutching events, task completion time, master robot end-effector trajectory and average control speed of the slave robot. The effectiveness of the Bayesian optimization is also evaluated, demonstrating that the optimized parameters can significantly improve users' performance. Results indicate that the proposed control method can reduce the operator's workload and enhance operation efficiency.

Gastrointestinal (GI) endoscopy is a conventional and prevalent procedure used to diagnose and treat diseases in the digestive tract. This procedure requires inserting an endoscope equipped with a camera and instruments inside a patient to the target of interest. To manoeuvre the endoscope, an endoscopist would rotate the knob at the handle to change the direction of the distal tip and apply the feeding force to advance the endoscope. However, due to the nature of the design, this often causes a looping problem during insertion making it difficult to be further advanced to the deeper section of the tract such as the transverse and ascending colon. To this end, in this paper, we propose a novel robotic endoscope which is covered by a rotating screw-like sheath and uses a spiral insertion technique to generate 'pull' forces at the distal tip of the endoscope to facilitate insertion. The whole shaft of the endoscope can be actively rotated, providing the crawling ability from the attached spiral sheath. With the redundant control on a spring-like continuum joint, the bending tip is capable of maintaining its orientation to assist endoscope navigation. To test its functions and feasibility to address the looping problem, three experiments were carried out. The first two experiments were to analyse the kinematic of the device and test the ability of the device to hold its distal tip at different orientation angles during spiral insertion. In the third experiment, we inserted the device in the bent colon phantom to evaluate the effectiveness of the proposed design against looping when advancing through a curved section of a colon. Results show the moving ability using spiral technique and verify its potential of clinical application.

Accurate volume segmentation from the Computed Tomography (CT) scan is a common prerequisite for pre-operative planning, intra-operative guidance and quantitative assessment of therapeutic outcomes in robot-assisted Minimally Invasive Surgery (MIS). 3D Deep Convolutional Neural Network (DCNN) is a viable solution for this task, but is memory intensive. Small isotropic patches are cropped from the original and large CT volume to mitigate this issue in practice, but it may cause discontinuities between the adjacent patches and severe class-imbalances within individual sub-volumes. This paper presents a new 3D DCNN framework, namely Z-Net, to tackle the discontinuity and class-imbalance issue by preserving a full field-of-view of the objects in the XY planes using anisotropic spatial separable convolutions. The proposed Z-Net can be seamlessly integrated into existing 3D DCNNs with isotropic convolutions such as 3D U-Net and V-Net, with improved volume segmentation Intersection over Union (IoU) - up to 12.6%. Detailed validation of Z-Net is provided for CT aortic, liver and lung segmentation, demonstrating the effectiveness and practical value of Z-Net for intra-operative 3D navigation in robot-assisted MIS.

Advances in depth sensing technologies have allowed simultaneous acquisition of both color and depth data under different environments. However, most depth sensors have lower resolution than that of the associated color channels and such a mismatch can affect applications that require accurate depth recovery. Existing depth enhancement methods use simplistic noise models and cannot generalize well under real-world conditions. In this paper, a coupled real-synthetic domain adaptation method is proposed, which enables domain transfer between high-quality depth simulators and real depth camera information for super-resolution depth recovery. The method first enables the realistic degradation from synthetic images, and then enhances degraded depth data to high quality with a color-guided sub-network. The key advantage of the work is that it generalizes well to real-world datasets without further training or fine-tuning. Detailed quantitative and qualitative results are presented, and it is demonstrated that the proposed method achieves improved performance compared to previous methods fine-tuned on the specific datasets.

Excessive stress is one of the main causes of mental illness. Long-term exposure of stress could affect one's physiological wellbeing (such as hypertension) and psychological condition (such as depression). Multisensory information such as heart rate variability (HRV) and pH can provide suitable information about mental and physical stress. This paper proposes a novel approach for stress condition monitoring using disposable flexible sensors. By integrating flexible amplifiers with a commercially available flexible polyvinylidene difluoride (PVDF) mechanical deformation sensor and a pH-type chemical sensor, the proposed system can detect arterial pulses from the neck and pH levels from sweat located in the back of the body. The system uses organic thin film transistor (OTFT)-based signal amplification front-end circuits with modifications to accommodate the dynamic signal ranges obtained from the sensors. The OTFTs were manufactured on a low-cost flexible polyethylene naphthalate (PEN) substrate using a coater capable of Roll-to-Roll (R2R) deposition. The proposed system can capture physiological indicators with data interrogated by Near Field Communication (NFC). The device has been successfully tested with healthy subjects, demonstrating its feasibility for real-time stress monitoring.

In this study we adopt predictive modelling to identify simultaneously commonalities and differences in multimodal brain networks acquired within subjects. Typically, predictive modelling of functional connectomes from structural connectomes explores commonalities across multimodal imaging data. However, direct application of multivariate approaches such as sparse Canonical Correlation Analysis (sCCA) applies on the vectorised elements of functional connectivity across subjects and it does not guarantee that the predicted models of functional connectivity are Symmetric Positive Matrices (SPD). We suggest an elegant solution based on the transportation of the connectivity matrices on a Riemannian manifold, which notably improves the prediction performance of the model. Randomised lasso is used to alleviate the dependency of the sCCA on the lasso parameters and control the false positive rate. Subsequently, the binomial distribution is exploited to set a threshold statistic that reflects whether a connection is selected or rejected by chance. Finally, we estimate the sCCA loadings based on a de-noising approach that improves the estimation of the coefficients. We validate our approach based on resting-state fMRI and diffusion weighted MRI data. Quantitative validation of the prediction performance shows superior performance, whereas qualitative results of the identification process are promising.

Body temperature is one of the vital parameters and the most common measurand for physiological monitoring. Heat has been used from ancient times as a medicinal and healing modality. It can be used generally for pain relief and can be particularly useful to arthritis patients, while it can be beneficial for battling dermatological and surgical site infections. Thermoregulation is essential for homeostasis and for sustaining body temperature. In this paper we present a wrist-worn flexible device integrating temperature sensing and heating capabilities for localized temperature measurements, heating for thermotherapy and closed-loop thermoregulation in the hands. The latter is useful for sustain hand temperature within healthy limits in extreme environments. The temperature sensing and heating capabilities of the device are characterized and presented in detail. We finally present a precise study of the effect of bending to the proposed device.

Electrical bioimpedance is a promising in vivo tissue characterization method. To develop optimized electronic instrumentation, knowledge of the electrical characteristics of the bioimpedance sensor and the targeted tissue are essential. This paper presents novel results from the characterization of a tetrapolar bioimpedance sensor for intestinal intraluminal mucosal ischemia assessment fabricated using flexible printed circuit (FPC) technology. The electrode impedance is measured individually and in pairs in saline solutions and equivalent circuits are proposed. The sensor is subsequently assessed in tetrapolar impedance measurements in saline solutions to extract experimentally the geometrical cell constant of the device. Finally, in vitro tetrapolar measurements from porcine intraluminal intestinal tissue are presented. The electrode impedance was found to be 145 ± 42 kΩ, while the tissue between 1.77 and 2.06 kΩ at 20 Hz. This work allows the design of next generation optimized CMOS instrumentation for implantable bioimpedance measurements for the particular application and sensor.

Wearable inertial sensors have demonstrated their potential to screen for various neuropathies and neurological disorders. Most such research has been based on classification algorithms that differentiate the control group from the pathological group, using biomarkers extracted from wearable data as predictors. However, such methods often lack quantitative evaluation of how much information provided by the wearable biomarkers contributes to the overall prediction. Despite promising results from internal cross validation, their utility in clinical practice remains unclear. In this paper, we highlight in a case study - early screening for diabetic peripheral neuropathy (DPN) - evaluation methods for quantifying the contribution of wearable inertial sensors. Using a quick-to-deploy wearable sensor system, we collected 106 in-hospital diabetic patients' gait data and developed logistic regression models to predict the risk of a diabetic patient having DPN. Adopting various metrics, we evaluated the discriminative information added by gait biomarkers and how much it improved screening. The results show that the proposed wearable system added useful information significantly to the existing clinical standards, and boosted the C-index significantly from 0.75 to 0.84, surpassing the current survey-based screening methods used in clinics.

Traditional methods of training a Brain-Computer Interface (BCI) on motor imagery (MI) data generally involve multiple intensive sessions. The initial sessions produce simple prompts to users, while later sessions additionally provide realtime feedback to users, allowing for human adaptation to take place. However, this protocol only permits the BCI to update between sessions, with little real-time evaluation of how the classifier has improved. To solve this problem, we propose an adaptive BCI training framework which will update the classifier in real time to provide more accurate feedback to the user on 4-class motor imagery data. This framework will require only one session to fully train a BCI to a given subject. Three variations of an adaptive Riemannian BCI were implemented and compared on data from both our own recorded datasets and the commonly used BCI Competition IV Dataset 2a. Results indicate that the fastest and least computationally expensive adaptive BCI was able to correctly classify motor imagery data at a rate 5.8% higher than when using a standard protocol with limited data. In addition it was confirmed that the adaptive BCI automatically improved its performance as more data became available.

Food intake is a major healthcare issue in developed countries that has become an economic and social burden across all sectors of society. Bad food intake habits lead to increased risk for development of obesity in children, young people and adults, with the latter more prone to suffer from health diseases such as diabetes, shortening the life expectancy. Environmental, cultural and behavioural factors have been appointed to be responsible for altering the balance between energy intake and expenditure, resulting in excess body weight. Methods to counteract the food intake problem are vast and include self-reported food questionnaires, body-worn sensors that record the sound, pressure or movements in the mouth and GI tract or image-based approaches that recognize the different types of food being ingested. In this paper we present an ear-worn device to track food intake habits by recording the acoustic signal produced by the chewing movements as well as the glucose level amperiometrically. Combined with a small camera on a future version of the device, we hope to deliver a complete system to control dietary habits with caloric intake estimation during satiation and deficit during satiety periods, which can be adapted to the physiology of each user.

The compliant nature of exosuits makes them ideal for providing assistance to complex joints like the shoulder. Exosuits require soft, compact and accurate sensing units for reliable feedback control. In this work, we introduce an OpenSim simulation-based prototype of a proprioception-inspired sensing framework for a multi-DoF shoulder exosuit. The prototype is used to study the feasibility of the sensing system concept to accurately track multiple degrees of freedom (DoFs) of the shoulder simultaneously. The sensing system fuses data from 4 custom string poten-tiometers (SPs), that work together to sense the joint angles at the shoulder. The tendon-routing of the SP modules in the exosuit is proprioception-inspired and based on the organization of the muscles influencing shoulder movement. The sensor fusion/mapping of the simulation data from multi-sensor space to joint space is a multivariate multiple regression problem and was solved using Multi-Layer Perceptron (MLP) & Long Short-Term Memory (LSTM) neural networks. A simulation of the framework in OpenSim on 200,000 random shoulder movements achieved a root mean square error (RMSE) of ≈ 0.2owhen trained on 70,000 random movements and tested on 130,000 random movements in both DoFs simultaneously.

Recent advances in teleoperation-based robotic-assisted Minimally Invasive Surgery (MIS) have made significant inroads in clinical adoption. However, such master-slave surgical systems create a physical separation between the surgeon and the patient. The concept of Active Constraints (ACs) provides guidance and sensory information to surgical robot operators in a form of haptic, visual or audible cues. This work proposes a novel ACs approach to avoid surgical tool-clashing and collision of the tool-shaft with delicate anatomy using elasto-plastic frictional force control. The presented framework is designed to reduce the occurence of direct coupling during electrocautery and to protect high-risk regions in Minimally Invasive Partial Nephrectomy (MIPN). Moreover, we combine aforementioned ACs methods and propose a solution when simultaneous penetration of both constraints occurs. The proposed methodology is implemented on the teleoperated da Vinci Surgical System using the da Vinci Research Kit (dVRK) and its performance is demonstrated through three types of user experiments. The experimental results show that the developed algorithms are of significant benefit in performing the tasks with ACs assistance.

Snake-like robots are commonly used in Minimally Invasive Surgery as they are able to reach areas deep inside the human body. These robots have instruments that are deployed out of the robot's head and controlled via tendons, which connect the instrument to motors at the proximal end. In most currently available systems the instruments are lacking a rolling motion of the end-effector.In this paper, we present a new instrument prototype for a snake-like robot that can perform a stable in-place rolling motion. The prototype has a diameter of 4mm, uses 13 tendons and has 6 degrees of freedom. The robot can bend and roll to high angles, and strongly improves the dexterity compared to an instrument without rolling capabilities. In the evaluation we show that the rolling-tip gripper can rotate about 165° and is capable of applying forces up to 6.5N.

In this paper, we present a novel approach for surgical task segmentation. A segmentation policy learns the correlations between features and segmentation points from manually labeled data. The most correlated features and rules for segmenting them are identified and learned. These form a complete set of segmentation policy. The proposed approach is developed to segment new but similar tasks through transfer learning. It is verified through applying the segmentation rule learned from the labeled data to segment other tasks. The performance of the proposed algorithm was evaluated by comparing the results against the ground truths. Experimental results demonstrate that our approach can achieve high segmentation rates with an accuracy of between 68.8% - 81.8%.

The current standard of intra-operative navigation during Fenestrated Endovascular Aortic Repair (FEVAR) calls for the need of 3D alignments between inserted devices and aortic branches. The navigation commonly via 2D fluoroscopic images, lacks anatomical information, resulting in longer operation hours and radiation exposure. In this paper, a skeleton instantiation framework of Abdominal Aortic Aneurysm (AAA) from a single 2D fluoroscopic image is introduced for real-time 3D robotic path planning. A graph matching method is proposed to establish the correspondences between the 3D preoperative and 2D intra-operative AAA skeletons, and then the two skeletons are registered by skeleton deformation and regularization in respect to skeleton length and smoothness. Furthermore, deep learning was used to segment 3D preoperative AAA from Computed Tomography (CT) scans to facilitate the framework automation. Simulation, phantom and patient AAA data sets have been used to validate the proposed framework. 3D distance error of 2mm was achieved in the phantom setup. Performance advantages were also achieved in terms of accuracy, robustness and time-efficiency.

Cardiovascular diseases remain as the most common cause of death worldwide. Remotely manipulated robotic systems are utilized to perform minimally invasive endovascular interventions. The main benefits of this methodology include reduced recovery time, improvement of clinical skills and procedural facilitation. Currently, robotic assistance, precision, and stability of instrument manipulation are compensated by the lack of haptic feedback and an excessive amount of radiation to the patient. This paper proposes a novel master-slave robotic platform that aims to bring the haptic feedback benefit on the master side, providing an intuitive user interface, and clinical familiar workflow. The slave robot is capable of manipulating conventional catheters and guidewires in multi-modal imaging environments. The system has been initially tested in a phantom cannulation study under fluoroscopic guidance, evaluating its reliability and procedural protocol. As the slave robot has been entirely produced by additive manufacturing and using pneumatic actuation, MR compatibility is enabled and was evaluated in a preliminary study. Results of both studies strongly support the applicability of the robot in different imaging environments and prospective clinical translation.

In the past few decades machine learning and data analysis have been having a huge growth and they have been applied in many different problems in the field of robotics. Data are usually the result of sensor measurements and, as such, they might be subjected to noise and outliers. The presence of outliers has a huge impact on modelling the acquired data, resulting in inappropriate models. In this work a novel approach for outlier detection and rejection for input/output mapping in regression problems is presented. The robustness of the method is shown both through simulated data for linear and nonlinear regression, and real sensory data. Despite being validated by using artificial neural networks, the method can be generalized to any other regression method.

In this paper, a novel semi-autonomous control framework is presented for enabling probe-based confocal laser endomicroscopy (pCLE) scan of the retinal tissue. With pCLE, retinal layers such as nerve fiber layer (NFL) and retinal ganglion cell (RGC) can be scanned and characterized in real-time for an improved diagnosis and surgical outcome prediction. However, the limited field of view of the pCLE system and the micron-scale optimal focus distance of the probe, which are in the order of physiological hand tremor, act as barriers to successful manual scan of retinal tissue. Therefore, a novel sensorless framework is proposed for real-time semi-autonomous endomicroscopy scanning during retinal surgery. The framework consists of the Steady-Hand Eye Robot (SHER) integrated with a pCLE system, where the motion of the probe is controlled semi-autonomously. Through a hybrid motion control strategy, the system autonomously controls the confocal probe to optimize the sharpness and quality of the pCLE images, while providing the surgeon with the ability to scan the tissue in a tremor-free manner. Effectiveness of the proposed architecture is validated through experimental evaluations as well as a user study involving 9 participants. It is shown through statistical analyses that the proposed framework can reduce the work load experienced by the users in a statistically-significant manner, while also enhancing their performance in retaining pCLE images with optimized quality.

Vascular diseases are the most common precursors to ischemic heart disease and stroke, which are two of the leading causes of death worldwide. Endovascular intervention is a minimally invasive surgical approach to treat such diseases. Compared to open surgery, it has the advantages of faster recovery, reduced need for general anesthesia, reduced blood loss and significantly lower mortality. Endovascular procedures require high surgical skills to minimize contacts between the manipulated instruments (catheters and guidewires) and the vessel wall, which represent one of the major risks for the patient. Robotic assistance can potentially improve the precision and stability of instruments manipulation. One key limitation of current commercial robotic platforms is the lack of haptic feedback, preventing their acceptance and limiting the clinical usability. This paper proposes to bring the benefit of haptic feedback to robot-assisted endovascular intervention. Here we hypothesize that the introduction of 3D haptic guidance during robot-assisted endovascular procedure can further improve the surgical performance and safety while overcoming the limitations of currently available technology. The proposed 3D haptic guidance allows the surgeon to sense the vasculature while controlling a catheter through a robotic haptic manipulator. Validation of the system is performed through end-user experiments with vascular surgeons on a bespoke surgical simulator. The obtained results demonstrate that 3D haptic guidance has the potential of improving effectiveness, precision, and safety of endovascular intervention. Furthermore, vascular surgeons found the proposed technology safe and overall easy to use, indicating its potential on real surgical procedures.

Over the last few years, there has been a significant rise in designing small, agile and flexible medical systems that can navigate through natural orifices. In the case of endoscopic surgery, existing systems vary significantly from each other which raises the question of the existence of a general design that can do it all. In this context, this paper proposes to use a genetic algorithm combined with recorded suturing and anatomical data to automatically design a pair of robotic instruments for the i2Snake under strict mechanical constraints. The resulting automatically generated instrument designs include a 6 degrees of freedom instrument that can follow a predefined trajectory accurately and a more simple 4 degrees of freedom instrument that can accomplish most of the task. The results also showed the importance of having a prismatic joint to gain the precision required for endoscopic surgery.

In surgical workflow analysis and training in robot-assisted surgery, automatic task segmentation could significantly reduce the manual labeling time and enhance robot learning efficiency. This paper presents an unsupervised segmentation approach to automatically segment a given surgical task without manual intervention. A new segmentation method is presented, which relies only on bimanual kinematic trajectories without the need for prior information about the data. Specifically, surgical tasks are segmented by fusing trajectories' spatiotemporal and variance properties. To demonstrate the effectiveness of the proposed method, detailed experiments were first conducted on our dataset. We segmented trajectories of three different surgical stitches and observed an average F1score of 77.9% against the ground truths. The same trajectories were then added with different levels of noises and the segmentation comparison was made with four other methods. The proposed algorithm had demonstrated its robustness against the noises. Finally, to assess its generalization ability, the method was evaluated on publicly available JIGSAWS dataset and an average F1score of 75.5% was achieved.

Assistive Robotic Manipulators (ARMs) play an important role for people with upper-limb disabilities and the elderly by helping them complete Activities of Daily Living (ADLs). However, as the objects to handle in ADLs differ in size, shape and manipulation constraints, many two or three fingered end-effectors of ARMs have difficulty robustly interacting with these objects. In this paper, we propose vision-based control of a 5-fingered manipulator (Schunk SVH), automatically changing its approach based on object classification using computer vision combined with deep learning. The control method is tested in a simulated environment and achieves a more robust grasp with the properly shaped five-fingered hand than with a comparable three-fingered gripper (Barrett Hand) using the same control sequence. In addition, the final optimal grasp pose (x, y, and θ) is learned through a deep regressor in the penultimate stage of the grasp. This method correctly identifies the optimal grasp pose in 78.35% of cases when considering all three parameters for an object included in the training set, but in a different setting than that of the training set.

Accurate master-slave control is important for Robot-Assisted Microsurgery (RAMS). This paper presents a handheld master controller for the operation and training of RAMS. A 9-axis Inertial Measure Unit (IMU) and a micro camera are utilized to form the sensing system for the handheld controller. A new hybrid marker pattern is designed to achieve reliable visual tracking, which integrated QR codes, Aruco markers, and chessboard vertices. Real-time multi-sensor fusion is implemented to further improve the tracking accuracy. The proposed handheld controller has been verified on an in-house microsurgical robot to assess its usability and robustness. User studies were conducted based on a trajectory following task, which indicated that the proposed handheld controller had comparable performance with the Phantom Omni, demonstrating its potential applications in microsurgical robot control and training.

Master manipulators represent a key component of Robot-Assisted Minimally Invasive Surgery (RAMIS). In this paper, an Analytic Hierarchy Process (AHP) method is used to construct an effective workspace analysis framework, which can assist the configuration selection and design evaluation of a portable master manipulator for surgical robot control and training. The proposed framework is designed based on three criteria: 1) compactness, 2) workspace quality, and 3) mapping efficiency. A hardware prototype, called the Hamlyn Compact Robotic Master (Hamlyn CRM), is constructed following the proposed framework. Experimental verification of the platform is conducted on the da Vinci Research Kit (dVRK) with which a da Vinci robot is controlled as a slave. The proposed Hamlyn CRM is compared with Phantom Omni, a commercial portable master device, with results demonstrating the relative merits of the new platform in terms of task completion time, average control speed and number of clutching.

Mood disorders affect more than 300 million people worldwide and can cause devastating consequences. Elderly people and patients with neurological conditions are particularly susceptible to depression. Gait and body movements can be affected by mood disorders, and thus they can be used as a surrogate sign, as well as an objective index for pervasive monitoring of emotion and mood disorders in daily life. Here we review evidence that demonstrates the relationship between gait, emotions and mood disorders, highlighting the potential of a multimodal approach that couples gait data with physiological signals and home-based monitoring for early detection and management of mood disorders. This could enhance self-awareness, enable the development of objective biomarkers that identify high risk subjects and promote subject-specific treatment.

Mental illnesses are vast and cause a lot of individual and social discomfort, with significant healthcare costs associated in terms of diagnosis and treatment. They can be triggered by a number of factors including stress, fatigue or anxiety. The associated physiological, cardiac and autonomic changes can be assessed, centrally, through brain imaging or, peripherally, by other signal recording modalities. With recent advances in wearable devices, these parameters can now be assessed in natural living conditions as associated mood disorders such as obsessive/compulsive behavior or depression are difficult to be examined in controlled settings. In this paper, we present a low-powered and flexible device with electrocardiogram (ECG), galvanic skin response (GSR), temperature and bio-motion detection channels, with signal accuracies of 62 μV for ECG, 6.6 kΩ for GSR, 0.13 °C for temperature and 0.04 g for acceleration. Potential applications include mental health assessment of patients during daily activities at home and/or work through non-continuous and multimodal sensing as demonstrated in this paper during exercise, rest and mental activities performed by healthy individuals only, achieving an overall accuracy of 89% in the classification of the different tasks executed by volunteers.

Probe-based confocal laser endomicroscopy (pCLE) is an emerging tool for epithelial cancer diagnosis, which enables in-vivo microscopic imaging during endoscopic procedures and facilitates the development of automatic recognition algorithms to identify the status of tissues. In this paper, we propose a transfer recurrent feature learning framework for classification tasks on pCLE videos. At the first stage, the discriminative feature of single pCLE frame is learned via generative adversarial networks based on both pCLE and histology modalities. At the second stage, we use recurrent neural networks to handle the varying length and irregular shape of pCLE mosaics taking the frame-based features as input. The experiments on real pCLE data sets demonstrate that our approach outperforms, with statistical significance, state-of-the-art approaches. A binary classification accuracy of 84.1% has been achieved.

Chromosome classification is critical for karyotyping in abnormality diagnosis. To expedite the diagnosis, we present a novel method named Varifocal-Net for simultaneous classification of chromosome's type and polarity using deep convolutional networks. The approach consists of one global-scale network (G-Net) and one local-scale network (L-Net). It follows three stages. The first stage is to learn both global and local features. We extract global features and detect finer local regions via the G-Net. By proposing a varifocal mechanism, we zoom into local parts and extract local features via the L-Net. Residual learning and multi-task learning strategies are utilized to promote high-level feature extraction. The detection of discriminative local parts is fulfilled by a localization subnet of the G-Net, whose training process involves both supervised and weakly supervised learning. The second stage is to build two multi-layer perceptron classifiers that exploit features of both two scales to boost classification performance. The third stage is to introduce a dispatch strategy of assigning each chromosome to a type within each patient case, by utilizing the domain knowledge of karyotyping. The evaluation results from 1909 karyotyping cases showed that the proposed Varifocal-Net achieved the highest accuracy per patient case (%) of 99.2 for both type and polarity tasks. It outperformed state-of-the-art methods, demonstrating the effectiveness of our varifocal mechanism, multi-scale feature ensemble, and dispatch strategy. The proposed method has been applied to assist practical karyotype diagnosis.

Continuous buried soft tissue free flap postoperative monitoring is crucial to detect flap failure and enable early intervention. In this case, clinical assessment is challenging as the flap is buried and only implantable or hand held devices can be used for regular monitoring. These devices have limitations in their price, usability and specificity. Near-infrared spectroscopy (NIRS) has shown promising results for superficial free flap postoperative monitoring, but it has not been considered for buried free flap, mainly due to the limited penetration depth of conventional approaches. A wearable wireless tomographic probe has been developed for continuous monitoring of tissue perfusion at different depths. Using the NIRS method, blood flow can be continuously measured at different tissue depths. This device has been designed following conclusions of extensive computerised simulations and it has been validated using a vascular phantom.

Gait analysis is an important tool for monitoring and preventing injuries as well as to quantify functional decline in neurological diseases and elderly people. In most cases, it is more meaningful to monitor patients in natural living environments with low-end equipment such as cameras and wearable sensors. However, inertial sensors cannot provide enough details on angular dynamics. This paper presents a method that uses a single RGB camera to track the 2D joint coordinates with state-of-the-art vision algorithms. Reconstruction of the 3D trajectories uses sparse representation of an active shape model. Subsequently, we extract gait features and validate our results in comparison with a state-of-the-art commercial multi-camera tracking system. Our results are comparable to those from the current literature based on depth cameras and optical markers to extract gait characteristics.

This paper proposes a novel artificial neural network based method for real-time gait analysis with minimal number of Inertial Measurement Units (IMUs). Accurate lower limb attitude estimation has great potential for clinical gait diagnosis for orthopaedic patients and patients with neurological diseases. However, the use of multiple wearable sensors hinder the ubiquitous use of inertial sensors for detailed gait analysis. This paper proposes the use of two IMUs mounted on the shoes to estimate the IMU signals at the shin, thigh and waist for accurate attitude estimation of the lower limbs. By using the artificial neural network framework, the gait parameters, such as angle, velocity and displacements of the IMUs can be estimated. The experimental results have shown that the proposed method can accurately estimate the IMUs signals on the lower limbs based only on the IMU signals on the shoes, which demonstrates its potential for lower limb motion tracking and real-time gait analysis.

Skin cancer is a medical condition that is becoming more common in many countries as a result of excessive exposure of individuals to sunlight. The ultraviolet range of the electromagnetic radiation is responsible for 90% of the cases involving the development of melanomas. Additional factors like the skin tone and texture can increase the risk of radiation exposure when the water content retained by the skin starts to drop dramatically. In this paper we present a small, batteryless wearable device that combines the computation of sunlight exposure with the measurement of the impedance of the skin and temperature, at any time of the day and independently of the location of the person wearing the sensor. Results have shown a good performance in tracking the ultraviolet index and the variation of impedance for different levels of skin hydration.

A key component to the success of master-slave surgical systems is the quality of the master interface used to relay the surgeon's instructions to the slave robot. In previous work the authors developed a gaze-assisted intention recognition scheme, allowing the system to dynamically adapt the motion scaling based on where the user is trying to reach. This allowed users to perform tasks significantly more quickly and with less need for clutching. However, the system possessed a number of core parameters that were manually optimized, potentially providing a non-optimal solution depending on the user. This paper presents a Bayesian approach to the problem of optimizing a human-robot interface in a user-specific manner. Two Bayesian optimization methods are studied: one in which users are asked to grade robot behavior for a given set of parameters, and one where only preference relative to other parameter sets is expressed. The performance of these optimizations is evaluated in a blind comparison user study, demonstrating that the optimized parameters are preferred to the manually optimized ones in over 90 % of cases after only 12 test samples. These parameters are further shown to perform at least as well as the manually optimized ones in all cases, and showing statistically significant improvement in the case of the graded optimization.

The technique of robust suture detection is vital in many applications including trainee suturing skill evaluation, suture augmentation in robotic-assisted surgery and suture recognition for automatic suturing. Due to the complicated environment of surgery, the detection of a suture is challenged by high deformation and frequent occlusion. In this paper, we propose a deep multi-stage framework for suture detection. The fully convolutional neural networks are firstly used to predict a gradient map which not only serves as a segmentation mask, but also provides useful structure information for the following thread centerline reconstruction. An overlapping map is also predicted to improve the quality of the gradient map in self-intersection area. Based on the gradient map, multiple segments of the thread are extracted and linked to form the whole thread using a curvilinear structure detector. Experiments on two types of threads demonstrate that the proposed method is able to detect the thread with human level performance when the thread is no occlusion or under finite self-intersection.

Bronchoscopic intervention, as a minimally invasive method for the diagnosis and treatment of lung diseases, has attracted more and more attention in recent years. However, existing endobronchial instruments lack the steerability accessing the peripheral airways with difficult bifurcations. This paper presents a novel wire-driven dexterous manipulator for the guidance of such instruments. Precision laser profiling is used to cut a stainless steel tube into multiple interlocked segments with revolute joints. The outer diameter of the manipulator is 2.20 mm which is small enough to be inserted into the working channels of most commercial bronchoscopes and distal airways, while keeping a large inner lumen with a diameter of 1.44 mm for passing various bronchoscopic instruments. The small bending radius provides enough flexibility to navigate inside the complex bronchial tree. Two kinematic models are proposed to predict the manipulator configuration from the translation of actuation wires. The former model is geometrically derived with the assumption of constant curvature bending and the latter one is statistically driven by capturing the motion trajectories of manipulator joints. A prototype of our low-cost add-on instrument guidance robot for bronchoscopic intervention is presented which can be easily integrated into current clinical routine.

Recent advances in probe-based Confocal Laser Endomicroscopy (pCLE) enable real-time, in situ and in vivo tissue assessment at the micro scale. The limited field-of-view offered by pCLE necessitates the use of mosaicking to allow for accurate tissue characterization from the incoming image stream. However, mosaicking requires a series of contiguous good-quality images, which is particularly challenging because probe-tissue distance must be maintained within a very narrow working range at all times and probe-tissue contact force must be kept to a minimum so that tissue deformation is avoided. Robotic manipulation of the endomicroscopy probe has provided partial solution to these challenges, but sensorless approaches have not been thoroughly investigated up to date. This paper proposes a novel sensorless framework that uses a single non-reference image-quality metric to learn an approximation of tissue motion and subsequently track it. Moreover, a pCLE robotic tool for autonomous endomicroscopy scanning is designed and used for testing and validation purposes. Experiments on lens paper and ex vivo porcine tissue validate the philosophy of the framework.

The use of snake-like robots for surgery is a popular choice for intra-luminal procedures. In practice, the requirements for strength, flexibility and accuracy are difficult to be satisfied simultaneously. This paper presents a computational approach for optimizing the design of a snake-like robot using serial rolling-joints and tendons as the base architecture. The method optimizes the design in terms of joint angle range and tendon placement to prevent the tendons and joints from colliding during bending motion. The resulting optimized joints were manufactured using 3D printing. The robot was characterized in terms of workspace, dexterity, precision and manipulation forces. The results show a repeatability as low as 0.9mm and manipulation forces of up to 5.6N.

Emerging robot-assisted endovascular intervention has the potential to reduce X-ray radiations to the operator while enhancing the stability and dexterity of catheter manipulation. Supervised and shared autonomy of endovascular procedures could add further improvements in reduced fatigue and cognitive workloads of the operator, higher success rates of cannulation and improved surgical outcomes. However, robotic path planning for endovascular procedure is challenging due to complex and non-linear flow dynamics inside the vasculature. This paper presents a learning-based robotic catheterization platform addressing those challenges, this approach incorporates path integral reinforcement learning (RL) framework based on dynamic movement primitives (DMP) to enhance catheterization tasks by a customized robotic manipulator. The robotic trajectories were optimized through RL in order to avoid unwanted contacts between the catheter tip and the vessel wall. The proposed methods can adapt to different flow simulations, vascular models, and catheterization tasks. The quality of the catheterization was evaluated with performance metrics. The results show significant refinement of catheter paths by the proposed approach, resulting in shorter overall lengths and fewer contact forces, which can potentially reduce risks in endothelial wall damages, embolization, and stroke. The results support the development of robotic path planning for endovascular procedures as well as designing intelligent, hands-on robotic navigation platforms.

Robotic and computer assistance can bring significant benefits to endovascular procedures in terms of precision and stability, reduced radiation doses, improved comfort and access to difficult and tortuous anatomy. However, the design of current commercially available platforms tends to alter the natural bedside manipulation skills of the operator, so that the manually acquired experience and dexterity are not well utilized. Furthermore, most of these systems lack of haptic feedback, preventing their acceptance and limiting the clinical usability. In this paper a new robotic platform for endovascular catheterization, the CathBot, is presented. It is an ergonomic master-slave system with navigation system and integrated vision-based haptic feedback, designed to maintain the natural bedside skills of the vascular surgeon. Unlike previous work reported in literature, dynamic motion tracking of both the vessel walls the catheter tip is incorporated to create dynamic active constraints. The system was evaluated through a combined quantitative and qualitative user study simulating catheterization tasks on a phantom. Forces exerted on the phantom were measured. The results showed a 70% decrease in mean force and 61% decrease in maximum force when force feedback is provided. This research provides the first integration of vision-based dynamic active constraints within an ergonomic robotic catheter manipulator. The technological advances presented here, demonstrates that vision-based haptic feedback can improve the effectiveness, precision, and safety of robot-assisted endovascular procedures.

In Minimally Invasive Robotic Surgery (MIRS) a robot is interposed between the surgeon and the surgical site to increase the precision, dexterity, and to reduce surgeon's effort and cognitive load with respect to the standard laparoscopic interventions. However, the modern robotic systems for MIRS are still based on the traditional telemanipulation paradigm, e.g. the robot behaviour is fully under surgeon's control, and no autonomy or assistance is implemented. In this work, supervised and shared controllers have been developed in a vision-free, human-in-the-Ioop, control framework to help surgeon during a surgical suturing procedure. Experiments conducted on the da Vinci Research Kit robot proves the effectiveness of the method indicating also the guidelines for improving results.

Estimating the three-dimensional (3D) position of microrobots is necessary in order to develop closed-loop control techniques and to improve the user's 3D perception in the micro-scale. This paper describes a depth estimation method based on supervised learning for optically transparent microrobots of known geometry. The proposed methodology uses Convolutional Neural Networks (CNNs) combined with a Recurrent Network, in particular a Long Short-Term Memory (LSTM) cell for depth regression. The proposed depth regression model is independent of the 3D orientation of the microrobot and is robust to varying illumination levels while it uses learned data-specific features. The model is trained and validated using microscope images and ground truth data generated from 3D-printed microrobots imaged in an Optical Tweezers (OT) setup. The validation results demonstrate that the proposed trained model can reconstruct the depth of the microrobot independently of its 3D orientation with submicron accuracy for the test set.

Task autonomy is an important consideration for the development of future surgical robots. For robot-assisted anastomosis, suture thread detection is a prerequisite for subsequent robot manipulation. Previous works on automatic thread detection are focused on the learning of the models with specific surgical settings that are poorly generalisable to generic settings. In this paper, we propose a joint feature learning framework that caters for the foreground and background adaptation for surgical suture thread detection. The proposed method is developed in the context of semi-supervised and unsupervised domain adaptation, leveraging the labelled training data from the source domain to learn the detection model for unlabelled or partially labelled target domain, which can also be from different types of threads or organs. Based on adversarial learning, we further preserve the semantic identity and introduce curriculum adaptation to generate synthetic data. Experiments on four domain adaptation tasks for suture thread detection demonstrate the strength of the proposed method being able to generate good quality synthetic data and transfer between specific domains with limited or even no labelled data of the target domain.

This paper presents a versatile robotic system for sewing a 3D structured object. Leveraging on using a customized robotic sewing device and closed-loop visual servoing control, an all-in-one solution for sewing personalized stent graft is demonstrated. Stitch size planning and automated knot tying are proposed as two key functions of the system. By using effective stitch size planning, sub-millimetre sewing accuracy is achieved for stitch sizes ranging from 2mm to 5mm. In addition, a thread manipulator for thread management and tension control is also proposed to perform successive knot tying to secure each stitch. Detailed laboratory experiments have been performed to evaluate the proposed instruments and allied algorithms. The proposed framework can be generalised to a wide range of applications including 3D industrial sewing, as well as transferred to other clinical areas such as surgical suturing.

Robot-assisted Fenestrated Endovascular Aortic Repair (FEVAR) is currently navigated by 2D fluoroscopy which is insufficiently informative. Previously, a semi-automatic 3D shape instantiation method was developed to instantiate the 3D shape of a main, deployed, and fenestrated stent graft from a single fluoroscopy projection in real-time, which could help 3D FEVAR navigation and robotic path planning. This proposed semi-automatic method was based on the Robust Perspective-S-Point (RP5P) method, graft gap interpolation and semiautomatic multiple-class marker center determination. In this paper, an automatic 3D shape instantiation could be achieved by automatic multiple-class marker segmentation and hence automatic multiple-class marker center determination. Firstly, the markers were designed into five different shapes. Then, Equally-weighted Focal U-Net was proposed to segment the fluoroscopy projections of customized markers into five classes and hence to determine the marker centers. The proposed Equally-weighted Focal U-Net utilized U-Net as the network architecture, equally-weighted loss function for initial marker segmentation, and then equally-weighted focal loss function for improving the initial marker segmentation. This proposed network outperformed traditional Weighted U-Net on the class-imbalance segmentation in this paper with reducing one hyperparameter - the weight. An overall mean Intersection over Union (mIoU) of 0.6943 was achieved on 78 testing images, where 81.01 % markers were segmented with a center position error <; 1.6mm. Comparable accuracy of 3D shape instantiation was also achieved and stated. The data, trained models and TensorFlow codes are available on-line.

While minimally invasive surgery offers great benefits in terms of reduced patient trauma, bleeding, as well as faster recovery time, it still presents surgeons with major ergonomic challenges. Laparoscopic surgery requires the surgeon to bimanually control surgical instruments during the operation. A dedicated assistant is thus required to manoeuvre the camera, which is often difficult to synchronise with the surgeon's movements. This article introduces a robotic system in which a rigid endoscope held by a robotic arm is controlled via the surgeon's eye movement, thus forgoing the need for a camera assistant. Gaze gestures detected via a series of eye movements are used to convey the surgeon's intention to initiate gaze contingent camera control. Hidden Markov Models (HMMs) are used for real-time gaze gesture recognition, allowing the robotic camera to pan, tilt, and zoom, whilst immune to aberrant or unintentional eye movements. A novel online calibration method for the gaze tracker is proposed, which overcomes calibration drift and simplifies its clinical application. This robotic system has been validated by comprehensive user trials and a detailed analysis performed on usability metrics to assess the performance of the system. The results demonstrate that the surgeons can perform their tasks quicker and more efficiently when compared to the use of a camera assistant or foot switches.

This paper presents a self-adaptive autonomous online learning through a general type-2 fuzzy system (GT2 FS) for the motor imagery (MI) decoding of a brain-machine interface (BMI) and navigation of a bipedal humanoid robot in a real experiment, using electroencephalography (EEG) brain recordings only. GT2 FSs are applied to BMI for the first time in this study. We also account for several constraints commonly associated with BMI in real practice: 1) the maximum number of EEG channels is limited and fixed; 2) no possibility of performing repeated user training sessions; and 3) desirable use of unsupervised and low-complexity feature extraction methods. The novel online learning method presented in this paper consists of a self-adaptive GT2 FS that can autonomously self-adapt both its parameters and structure via creation, fusion, and scaling of the fuzzy system rules in an online BMI experiment with a real robot. The structure identification is based on an online GT2 Gath-Geva algorithm where every MI decoding class can be represented by multiple fuzzy rules (models), which are learnt in a continous (trial-by-trial) non-iterative basis. The effectiveness of the proposed method is demonstrated in a detailed BMI experiment, in which 15 untrained users were able to accurately interface with a humanoid robot, in a single session, using signals from six EEG electrodes only.

This paper presents a multirobot system for manufacturing personalized medical stent grafts. The proposed system adopts a modular design, which includes a (personalized) mandrel module, a bimanual sewing module, and a vision module. The mandrel module incorporates the personalized geometry of patients, while the bimanual sewing module adopts a learning-by-demonstration approach to transfer human hand-sewing skills to the robots. The human demonstrations were first observed by the vision module and then encoded using a statistical model to generate the reference motion trajectories. During autonomous robot sewing, the vision module plays the role of coordinating multirobot collaboration. Experimental results show that the robots can adapt to generalized stent designs. The proposed system can also be used for other manipulation tasks, especially for flexible production of customized products and where bimanual or multirobot cooperation is required.

In fasciocutaneous free flap surgery, close postoperative monitoring is crucial for detecting flap failure, as around 10% of cases require additional surgery due to compromised anastomosis. Different biochemical and biophysical techniques have been developed for continuous flap monitoring, however, they all have shortcoming in terms of reliability, elevated cost, potential risks to the patient, and inability to adapt to the patient's phenotype. A wearable wireless device based on near infrared spectroscopy has been developed for continuous blood flow and perfusion monitoring by quantifying tissue oxygen saturation (). This miniaturized and low-cost device is designed for postoperative monitoring of flap viability. With self-calibration, the device can adapt itself to the characteristics of the patients' skin such as tone and thickness. An extensive study was conducted with 32 volunteers. The experimental results show that the device can obtain reliable measurements across different phenotypes (age, sex, skin tone, and thickness). To assess its ability to detect flap failure, the sensor was tested in a pilot animal study. Free groin flaps were performed on 16 Sprague Dawley rats. Results demonstrate the accuracy of the sensor in assessing flap viability and identifying the origin of failure (venous or arterial thrombosis).

Blood flow, posture and phenotype (such as age, sex, smoking habit or physical activity) are closely related to vascular health. Episodic monitoring of the vascular system in clinical setting can lead to late diagnose. Inexpensive wearable devices for continuous monitoring of vascular parameters have been widely used, however, they often have limitations in data interpretation: changes in the environment setting can significantly affect the meaning of the results. This paper proposes a low cost networked body worn sensors for real-time analysis of hemodynamics and reports preliminary results on the relation between blood flow (measured through pulse arrival time (PAT)), the effect of postures and age ranges based on experiments with 13 volunteers of different age ranges (50 years old). Standing, supine and sitting postures were investigated while photoplethysmograph (PPG) sensors were placed at different locations (ear, wrist and ankle). Results show the PAT changes according to the investigated locations and postures for both age group. Also, the average PAT values of the older group are generally higher than those of the younger group. In the older group, the average PAT value is higher for the supine posture than that of the sitting posture which is itself higher than that of the standing posture. In the younger group, the average PAT is higher in supine than that of the sitting and standing postures which have similar average PAT values. This indicates that hemodynamics vary with posture and age.

To functionally aid patients suffering from neurological disorder, a 3 degrees-of-freedom (DoF) upper limb wearable robot is presented (Fig. 1). In order to provide seamless user assistance, the intention of the wearer must be determined. As a sensing mechanism, electromyographic (EMG) signals have commonly been used to estimate human movement. In this study, the effectiveness of movement recognition using a generalized 8-port EMG sensor (Myo Armband) around the forearm was evaluated. Four fundamental movements of the arm (wrist flexion/extension and forearm pronation/supination) were classified using a neural network (NN) with a single hidden layer. The classification method was optimized through analysis of pre-processing algorithms and window size (0.25 to 1 second) to reduce computational expense and maintain classification accuracy. Through these accomplishments, significant groundwork has been provided for the development of a robust and non-invasive solution to tremor of the upper limb.

We're very excited about this year's meeting, which is being held on the High Tech Campus in Eindhoven, the Netherlands, a sparkling location originally owned by Philips Research. Due to the open innovation strategy, it has been turned into a rich biotope of R&D companies working in close cooperation with neighboring institutions, universities (including TU Eindhoven and RWTH Aachen University) and academic hospitals.

Wearable technology has become ubiquitous in recent years due to the miniaturization of circuit electronics and advances in smart materials that can conform to the requirements posed by the human body, behaviour and experience. Sensors of this type are found attached almost to every body segment, capable of delivering signals even in harsh activity scenarios. The reliability and relevance of the physiological data retrieved by wearables have yet to surpass the conventional technologies in the healthcare system today. In this paper we present a small device incorporated inside an headphone set that continuously monitors the ECG, impedance and acceleration of the head. As opposed to most biometric sensors, ECG measurement relies on non-optical methods by capturing the electrical potential around the ear in both sides of the head, whereas impedance monitoring involves AC stimulation instead of DC, the latter commonly involved in skin galvanic response estimation. Signal processing of impedance parameters is performed in situ using a fast variant of the Discrete Fourier Transform in order to save computational resources and power expenditure from a microcontroller equipped with Bluetooth Low Energy. Applications that can benefit from this device include cardiovascular and stress level assessment of individuals for whom an hearable is a requirement for work or leisure.

With increasing popularity of wearable and Body Sensor Networks technologies, there is a growing concern on the security and data protection of such low-power pervasive devices. With very limited computational power, BSN sensors often cannot provide the necessary data protection to collect and process sensitive personal information. Since conventional network security schemes are too computationally demanding for miniaturized BSN sensors, new methods of securing BSNs have proposed, in which Biometric Cryptosystem (BCS) appears to be an effective solution. With regards to BCS security solutions, physiological traits, such as an individual's face, iris, fingerprint, electrocardiogram (ECG), and photoplethysmogram (PPG) have been widely exploited. However, behavioural traits such as gait are rarely studied. In this paper, a novel lightweight symmetric key generation scheme based on the timing information of gait is proposed. By extracting similar timing information from gait acceleration signals simultaneously from body worn sensors, symmetric keys can be generated on all the sensor nodes at the same time. Based on the characteristics of generated keys and BSNs, a fuzzy commitment based key distribution scheme is also developed to distribute the keys amongst the sensor nodes.

Recent studies have shown that air pollution has a negative impact on people's health, especially for patients with respiratory and cardiac diseases (e.g. COPD, asthma, ischemic heart disease). Although there are already many air quality monitoring stations in major cities, such as London, these stations are sparsely located, and the periodic collection of information is insufficient to provide the granularity needed to assess the environmental risk for an individual (e.g. to avoid exacerbation). Wearable devices, on the other hand, are more suitable in this context, providing a better estimation of the air quality in the proximity of the person. Therefore, relevant warnings and information on health risks can be provided in real-time. As a proof of concept, we have developed a wearable sensor for continuous monitoring of air quality around the user, and a preliminary study was conducted to validate the sensor and assess the air quality in London underground stations. Based on the PM2.5 (particulate matter with a diameter of 2.5 μm), temperature and location information, a model is generated for predicting the air quality of each station at different times. Our preliminary results have shown that there are significant differences in air quality among stations and metro lines. It also demonstrates that wearable sensors can provide necessary information for users to make travel arrangements that minimize their exposure to polluted air.

Increasing demands in endovascular intervention have motivated technical skill training and competency-based measures of performance. However, there are no well-established online metrics for technical skill assessment; few studies have explored operator behavioral patterns from catheter motion and operator hand motions. This paper proposes a platform for active online training and objective assessment of endovascular skills, through learning optimum catheter motions from multiple demonstrations. An ungrounded hand-held haptic device for providing intuitive haptic guidance to novice users based on this learnt information is also proposed. Statistical models are implemented to extract the underlying catheter motion patterns, and utilize them for performance evaluation and haptic guidance. The results show significant improvements in endovascular navigation for inexperienced operators. Finer catheter motions were achieved with the provided haptic guidance. The results suggest that the proposed platform can be integrated into current clinical training setups, and motivate the improvement of endovascular training platforms with better realism.

Optical Tweezers (OT) have the advantage of non-contact interaction with target objects such as cells, overcoming the pitfall of obstructive adhesion forces which are present in contact micromanipulation. It is also feasible to manipulate a number of small microparts simultaneously or 3D structures by using multiple laser traps. These capabilities give rise to the potential to develop a human-robot interface to facilitate microassembly tasks. This paper presents a gaze contingent control framework and a method for 3D orientation estimation for optical micromanipulation. The proposed strategy aims to use OT as an interactive microassembly platform. The framework comprises I) a strategy to recognize the operator's intentions in order to interactively place and reconfigure the optical traps using the operator's eye fixation point, II) haptic constraints generated from the user's eye gaze to assist positioning of the assembled microparts and III) a method for 3D orientation estimation. The performance of the proposed framework is assessed through a set of experiments comparing it to the standard OT user interface. Three-dimensional manipulation and orientation estimation of a non-spherical microstructure are also performed.

Traditional robotic surgical systems rely entirely on robotic arms to triangulate articulated instruments inside the human anatomy. This configuration can be ill-suited for working in tight spaces or during single access approaches, where little to no triangulation between the instrument shafts is possible. The control of these instruments is further obstructed by ergonomic issues: The presence of motion scaling imposes the use of clutching mechanics to avoid the workspace limitations of master devices, and forces the user to choose between slow, precise movements, or fast, less accurate ones. This paper presents a bi-manual system using novel self-triangulating 6-degrees-of-freedom (DoF) tools through a flexible elbow, which are mounted on robotic arms. The control scheme for the resulting 9-DoF system is detailed, with particular emphasis placed on retaining maximum dexterity close to joint limits. Furthermore, this paper introduces the concept of gaze-assisted adaptive motion scaling. By combining eye tracking with hand motion and instrument information, the system is capable of inferring the user's destination and modifying the motion scaling accordingly. This safe, novel approach allows the user to quickly reach distant locations while retaining full precision for delicate manoeuvres. The performance and usability of this adaptive motion scaling is evaluated in a user study, showing a clear improvement in task completion speed and in the reduction of the need for clutching.

This work is towards targeted drug delivery using microrobots functionalized to navigate towards naturally occurring pH gradients caused by cancer cells, and to release a payload in response to a light stimulus. Stimuli-responsive microrobots for the localization of specific cell types and targeted drug delivery could provide a new and promising therapy to prevent and treat the spread of cancer. In this work, we present two novel biocompatible photoresists for the fabrication of hybrid microrobots using two-photon polymerization (TPP) for medical applications. One biomarker for cancerous cells is that they exhibit lower pH compared to surrounding healthy tissue. In this work, a pH-responsive resist was developed and demonstrated to automatically seek a low-pH solid in a microfluidic channel, simulating metastatic cells within a vessel. The second resist, a hydrogel-based photoresist, was created to contract in response to light. The two resists were combined together in a two-step printing process to create a microswimmer with potential for tumor localization and drug release capabilities in the human circulatory system.

Advances in optical imaging, and probe-based Confocal Laser Endomicroscopy (pCLE) in particular, offer real-time cellular level information for in-vivo tissue characterization. However for large area coverage, the limited field-of-view necessitates the use of a technique known as mosaicking to generate usable information from the incoming image stream. Mosaicking also needs a continuous stream of good quality images, but this is challenging as the probe needs to be maintained within an optimal working range and the contact force controlled to minimize tissue deformation. Robotic manipulation presents a potential solution to these challenges, but the lack of haptic feedback in current surgical robot systems hinders the technology's clinical adoption. This paper proposes a sensorless alternative based on processing the incoming image stream and deriving a quantitative measure representative of the image quality. This measure is then used by a controller, designed using model-free reinforcement learning techniques, to maintain optimal contact autonomously. The developed controller has shown near real-time performance in overcoming typical loss-of-contact and excess-deformation scenarios experienced during endomicroscopy scanning procedures.

Effective in situ, in vivo tumour margin assessment is an important, yet unmet, clinical demand in surgical oncology. Recent advances in probe-based optical imaging tools such as confocal endomicroscopy is making inroads in clinical applications. In practice, maintaining consistent tissue contact whilst ensuring large area surveillance is crucial for its practical adoption and for this reason there is a great demand for robotic assistance so that high-speed endomicroscopes can be combined with autonomous scanning, thus simplifying its incorporation in routine surgical workflows. In this paper, a cooperatively controlled robotic manipulator is developed, which provides a stable mechatronically-enhanced platform for micro-scanning tools to perform local high resolution mosaics over 3D undulating moving surfaces. Detailed kinematic and overall system performance analyses are provided and the results demonstrate the adaptability in terms of both contact force and orientation control of the system, and thus its simplicity in practical deployment and value for clinical adoption.

Robot-assisted minimally invasive surgery can benefit from the automation of common, repetitive or well-defined but ergonomically difficult tasks. One such task is the scanning of a pick-up endomicroscopy probe over a complex, undulating tissue surface to enhance the effective field-of-view through video mosaicing. In this paper, the da Vinci®surgical robot, through the dVRK framework, is used for autonomous scanning and 2D mosaicing over a user-defined region of interest. To achieve the level of precision required for high quality mosaic generation, which relies on sufficient overlap between consecutive image frames, visual servoing is performed using a combination of a tracking marker attached to the probe and the endomicroscopy images themselves. The resulting sub-millimetre accuracy of the probe motion allows for the generation of large mosaics with minimal intervention from the surgeon. Images are streamed from the endomicroscope and overlaid live onto the surgeons view, while 2D mosaics are generated in real-time, and fused into a 3D stereo reconstruction of the surgical scene, thus providing intuitive visualisation and fusion of the multi-scale images. The system therefore offers significant potential to enhance surgical procedures, by providing the operator with cellular-scale information over a larger area than could typically be achieved by manual scanning.

Confocal endomicroscopy can be used for identification of early mucosal dysplasia in various gastrointestinal conditions, and has particular potential in the monitoring of Barrett's oesophagus and the early stages of oesophageal cancer. However, it can be difficult to systematically scan a significant area of the oesophagus because of the small field-of-view and limited flexibility of the probe. Tissue deformation and inconsistent probe-tissue contact also make it difficult to form large mosaics. A mechanical scanning device is therefore desirable for controlled, large area surface scanning and mosaicing of the oesophagus. This paper proposes a robotic catheter encapsulated in an inflatable balloon, providing stable scanning over the oesophageal surface. It has an outer diameter of 3 mm, making it suitable for deployment through an endoscope working channel, and uses a custom endomicroscopy probe based on a leached flexible fibre bundle and an external confocal laser scanning system. Detailed mechanical performance and image quality evaluations were performed to assess the clinical potential of the device. In ex vivo studies using swine oesophagus, long helical scans were obtained, demonstrating that the device is able to scan the lumen stably and maintain good probe-tissue contact. The experimental results demonstrate the potential of the robotic catheter for systematic high-resolution imaging of the oesophageal mucosa, potentially reducing or even eliminating the need for physical biopsy.

Six degree-of-freedom (DoF) pose feedback is essential for the development of closed-loop control techniques for microrobotics. This paper presents two methods for depth estimation of transparent microrobots inside an Optical Tweezers (OT) setup using image sharpness measurements and model-based tracking. The x-y position and the 3D orientation of the object are estimated using online model-based template matching. The proposed depth estimation methodologies are validated experimentally by comparing the results with the ground truth.

This paper presents a vision-based learning-by-demonstration approach for multi-robot manipulation. With this method, a vision system is involved in both the task demonstration and reproduction stages, and the speed and accuracy of the task reproduction are adapted according to the context of the demonstration. An expert first demonstrates how to use tools to perform a task, while the tool motion is observed using a vision system. The demonstrations are then encoded using a statistical model to generate a reference motion trajectory. Equipped with the same tools and the learned model, the robot is guided by vision to reproduce the task. The task performance was evaluated in terms of both accuracy and speed. However, simply increasing the robot's speed could decrease the reproduction accuracy. To this end, a dual-rate Kalman filter is employed to compensate for latency between the robot and vision system. More importantly, the robot speed is adapted according to the learned motion model. We demonstrate the effectiveness of our approach by performing two tasks: a trajectory reproduction task and a bimanual sewing task. We show that using our vision-based approach, the robots can conduct effective learning by demonstrations and perform accurate and fast task reproduction. The proposed approach is generalisable to other manipulation tasks, where bimanual or multi-robot cooperation is required.

The use of concentric tube robots has recognized advantages for accessing target lesions while conforming to certain anatomical constraints. However, their complex kinematics makes their safe telemanipulation in convoluted anatomy a challenging task. Collaborative control schemes, which guide the operator through haptic and visual feedback, can simplify this task and reduce the cognitive burden of the operator. Guaranteeing stable, collision-free robot configurations during manipulation, however, is computationally demanding and, until now, either required long periods of pre-computation time or distributed computing clusters. Furthermore, the operator is often presented with guidance paths which have to be followed approximately. This paper presents a heterogeneous (CPU/GPU) computing approach to enable rapid workspace analysis on a single computer. The method is used in a new navigation scheme that guides the robot operator towards locations of high dexterity or manipulability of the robot. Under this guidance scheme, the user can make informed decisions and maintain full control of the path planning and manipulation processes, with intuitive visual feedback on when the robot's limitations are being reached.

Snake like continuum robots are increasingly used for minimally invasive surgery. Most robotic devices of this sort that have been reported to date are controlled in an open loop manner. Using shape sensing to provide closed loop feedback would allow for more accurate control of the robot's position and, hence, more precise surgery. Fiber Bragg Gratings, magnetic sensors and optical reflectance sensors have all been reported for this purpose but are often limited by their cost, size, stiffness or complexity of fabrication. To address this issue, we designed, manufactured and tested a prototype two-link robot with a built-in fiber-optic shape sensor that can deliver and control the position of a CO2-laser fiber for soft tissue ablation. The shape sensing is based on optical reflectance, and the device (which has a 4 mm outer diameter) is fabricated using 3D printing. Here we present proof-of-concept results demonstrating successful shape sensing - i.e. measurement of the angular displacement of the upper link of the robot relative to the lower link - in real time with a mean measurement error of only 0.7°.

Suturing is an essential requirement for surgical robots. Current designs of needle drivers require significant forces to be applied so that the needle will not flip or slip between the jaws when inserted into the tissue. The required force implies that most designs are based on rigid instruments with a straight shaft that induces less dexterity. Flexible robotics provides great dexterity to achieve the task, although often it lacks the capability to apply large forces and therefore hold the surgical needle firmly. Solutions can be found in how the needle driver's jaws are designed. This paper presents improved designs for needle grasping tools for flexible robotic instruments, to be produced directly with selective laser melting. Features on the gripping surface were embedded to increase the gripping effectiveness even in the presence of limited grasping force. A detailed characterisation has been performed for each design showing improvements with respect to the state of the art.

The performance of a master-slave robotic system depends significantly on the ergonomics and the capability of its master device to correctly interface the user with the slave robot. Master manipulators generating commands in task space represent a commonly adopted solution for controlling a range of slave robots while retaining an ergonomic design. However, these devices present several drawbacks, such as requiring the use of clutching mechanics to compensate for the mismatch between slave and master workspaces, and the lack of capability to intuitively transmit important information such as specific joint limits to the user. In this paper, a novel joint-space master manipulator is presented. This manipulator emulates the kinematic structure of highly flexible surgical instruments which it is designed to control. This system uses 6 active degrees of freedom to compensate for its own weight, as well as to provide force feedback corresponding to the slave robot's joint limits. A force/torque sensor integrated at the end effector is used to relay user-generated forces and torques directly to specific joints. This is performed to counteract the friction stemming from structural constraints imposed by the kinematic design of the instruments. Finally, a usability study is carried out to test the validity of the system, proving that the instruments can be intuitively controlled even at the extremities of the workspace.

A body sensor network that consists of wearable and/or implantable biosensors has been an important front-end for collecting personal health records. It is expected that the full integration of outside-hospital personal health information and hospital electronic health records will further promote preventative health services as well as global health. However, the integration and sharing of health information is bound to bring with it security and privacy issues. With extensive development of healthcare applications, security and privacy issues are becoming increasingly important. This paper addresses the potential security risks of healthcare data in Internet-based applications and proposes a method of signal scrambling as an add-on security mechanism in the application layer for a variety of healthcare information, where a piece of tiny data is used to scramble healthcare records. The former is kept locally and the latter, along with security protection, is sent for cloud storage. The tiny data can be derived from a random number generator or even a piece of healthcare data, which makes the method more flexible. The computational complexity and security performance in terms of theoretical and experimental analysis has been investigated to demonstrate the efficiency and effectiveness of the proposed method. The proposed method is applicable to all kinds of data that require extra security protection within complex networks.

With a massive influx of multimodality data, the role of data analytics in health informatics has grown rapidly in the last decade. This has also prompted increasing interests in the generation of analytical, data driven models based on machine learning in health informatics. Deep learning, a technique with its foundation in artificial neural networks, is emerging in recent years as a powerful tool for machine learning, promising to reshape the future of artificial intelligence. Rapid improvements in computational power, fast data storage, and parallelization have also contributed to the rapid uptake of the technology in addition to its predictive power and ability to generate automatically optimized high-level features and semantic interpretation from the input data. This article presents a comprehensive up-to-date review of research employing deep learning in health informatics, providing a critical analysis of the relative merit, and potential pitfalls of the technique as well as its future outlook. The paper mainly focuses on key applications of deep learning in the fields of translational bioinformatics, medical imaging, pervasive sensing, medical informatics, and public health.

The increasing popularity of wearable devices in recent years means that a diverse range of physiological and functional data can now be captured continuously for applications in sports, wellbeing, and healthcare. This wealth of information requires efficient methods of classification and analysis where deep learning is a promising technique for large-scale data analytics. While deep learning has been successful in implementations that utilize high-performance computing platforms, its use on low-power wearable devices is limited by resource constraints. In this paper, we propose a deep learning methodology, which combines features learned from inertial sensor data together with complementary information from a set of shallow features to enable accurate and real-time activity classification. The design of this combined method aims to overcome some of the limitations present in a typical deep learning framework where on-node computation is required. To optimize the proposed method for real-time on-node computation, spectral domain preprocessing is used before the data are passed onto the deep learning framework. The classification accuracy of our proposed deep learning approach is evaluated against state-of-the-art methods using both laboratory and real world activity datasets. Our results show the validity of the approach on different human activity datasets, outperforming other methods, including the two methods used within our combined pipeline. We also demonstrate that the computation times for the proposed method are consistent with the constraints of real-time on-node processing on smartphones and a wearable sensor platform.

Recent advances in hyperspectral imaging have made it a promising solution for intra-operative tissue characterization, with the advantages of being non-contact, non-ionizing, and non-invasive. Working with hyperspectral images in vivo, however, is not straightforward as the high dimensionality of the data makes real-time processing challenging. In this paper, a novel dimensionality reduction scheme and a new processing pipeline are introduced to obtain a detailed tumor classification map for intra-operative margin definition during brain surgery. However, existing approaches to dimensionality reduction based on manifold embedding can be time consuming and may not guarantee a consistent result, thus hindering final tissue classification. The proposed framework aims to overcome these problems through a process divided into two steps: dimensionality reduction based on an extension of the T-distributed stochastic neighbor approach is first performed and then a semantic segmentation technique is applied to the embedded results by using a Semantic Texton Forest for tissue classification. Detailed in vivo validation of the proposed method has been performed to demonstrate the potential clinical value of the system.

Tracking and shape estimation of flexible robots that navigate through the human anatomy are prerequisites to safe intracorporeal control. Despite extensive research in kinematic and dynamic modeling, inaccuracies and shape deformation of the robot due to unknown loads and collisions with the anatomy make shape sensing important for intraoperative navigation. To address this issue, vision-based solutions have been explored. The task of 2-D tracking and 3-D shape reconstruction of flexible robots as they reach deep-seated anatomical locations is challenging, since the image acquisition techniques usually suffer from low signal-to-noise ratio or slow temporal responses. Moreover, tracking and shape estimation are thus far treated independently despite their coupled relationship. This paper aims to address tracking and shape estimation in a unified framework based on Markov random fields. By using concentric tube robots as an example, the proposed algorithm fuses information extracted from standard monoplane X-ray fluoroscopy with the kinematics model to achieve joint 2-D tracking and 3-D shape estimation in realistic clinical scenarios. Detailed performance analyses of the results demonstrate the accuracy of the method for both tracking and shape reconstruction.

Blood flow and perfusion monitoring are critical appraisal to ensure survival of tissue flap after reconstructive surgery. Many techniques have been developed over the years: from optical to chemical, invasive or not, they all have limitations in their price, risks and adaptiveness to the patient. A wireless wearable self-calibrated device, based on near infrared spectroscopy (NIRS) was developed for blood flow and perfusion monitoring contingent on tissue oxygen saturation (StO2). The use of such device is particularly relevant in the case of free flap myocutaneous reconstructive surgery; postoperative monitoring of the flap is crucial for a prompt intervention in case of thrombosis. Although failure rate is low, the rate of additional surgery following anastomosis problem is about 50%. NIRS has shown promising results for the monitoring of free flap, however lack of adaptation to its environment (ambient light) and users (body mass index (BMI), skin tone, alcohol and smoking habits or physical activity level) hinders the practical use of this technique. To overcome those limitations, a self-calibrated approach is introduced. Tested with is chaemia and cold water experiments on healthy subjects of different skin tones, its ability to personalize its calibration is demonstrated. Furthermore, using a vascular phantom, it is also able to detect pulses, differentiate venous and arterial coloured-like fluids with distinct clusters and detect significant changes in simulated partial venous occlusion. Placed in the trained classifier, partial occlusion data showed similar results between predicted and true classification. Further analysis from partial occlusion data showed that distinct clusters for 75% and 100% occlusion emerged.

Human Activity Recognition provides valuable contextual information for wellbeing, healthcare, and sport applications. Over the past decades, many machine learning approaches have been proposed to identify activities from inertial sensor data for specific applications. Most methods, however, are designed for offline processing rather than processing on the sensor node. In this paper, a human activity recognition technique based on a deep learning methodology is designed to enable accurate and real-time classification for low-power wearable devices. To obtain invariance against changes in sensor orientation, sensor placement, and in sensor acquisition rates, we design a feature generation process that is applied to the spectral domain of the inertial data. Specifically, the proposed method uses sums of temporal convolutions of the transformed input. Accuracy of the proposed approach is evaluated against the current state-of-the-art methods using both laboratory and real world activity datasets. A systematic analysis of the feature generation parameters and a comparison of activity recognition computation times on mobile devices and sensor nodes are also presented.

Ultrasound imaging is a proven diagnostic tool to assess a myriad of physiological and pathological conditions in patients. Throughout the years, ultrasounds have been used as a passive recording modality where the backscattered echo arising from the interaction of the sound waves with the acoustic properties of the biological tissues helps to identify them. Apart from a wide range of therapeutic applications, the acoustic beam has not yet been explored to actuate within the biological environment in an active way. In this paper we present an implantable electronic device to be actuated remotely by ultrasounds with capabilities for measuring several physiological parameters of tissues: pH, temperature, electrolyte concentration and biopotentials. The small factory form device (with no attached batteries) harvests energy from the incoming ultrasound waves and uses it to power the embedded electronics. It operates from voltage levels as low as 0.8 V and consuming a total current of 60 μA (or an average power consumption of 84 μW) in the active mode when deployed at a distance of 3 cm from the active source of ultrasounds in vitro, excited by a sinusoid at 400 kHz with power density of 20 mWcm-2. The sensor can be actuated by a specifically-designed readout device (as detailed in this paper) or using the traditional medical probes for ultrasound imaging. The actual device can present an alternative to surpass the limitations of inductive and RF-powered sensors implanted in soft tissues.

The recent evolution of surgical robots has resolved a number of ergonomic issues associated with conventional minimally invasive surgery (MIS) in terms of aligned visiomotor axes, motion scaling and ergonomics. One of the latest advances is the introduction of human-robot cooperative control combining features such as active constraints, machine learning and automated movements. This paper aims to integrate these techniques into a framework which can be generalized to a wide range of surgical tasks. This paper proposes a system entitled Hubot; a Human-Robot collaborative framework which combines the strengths of the surgeon, the advantages of robotics and learning from demonstration into a single system. Hubot was successfully implemented on a Raven II surgical robot and a user study was conducted to evaluate its performance. Both a training and a simulated clinical case were investigated and showed promising results in comparison to fully manual task execution, including reduced completion time, fewer movements for the operator and improved efficiency.

Bronchoscopic interventions are widely performed for the diagnosis and treatment of lung diseases. However, for most endobronchial devices, the lack of a bendable tip restricts their access ability to get into distal bronchi with complex bifurcations. This paper presents the design of a new wire-driven continuum manipulator to help guide these devices. The proposed manipulator is built by assembling miniaturized blocks that are featured with interlocking circular joints. It has the capability of maintaining its integrity when the lengths of actuation wires change due to the shaft flex. It allows the existence of a relatively large central cavity to pass through other instruments and enables two rotational degrees of freedom. All these features make it suitable for procedures where tubular anatomies are involved and the flexible shafts have to be considerably bent in usage, just like bronchoscopic interventions. A kinematic model is built to estimate the relationship between the translations of actuation wires and the manipulator tip position. A scale-up model is produced for evaluation experiments and the results validate the performance of the proposed mechanism.

In this paper, we propose manipulation ability extension of the optical tweezers by developing microhands, which are to use as end-effectors of the laser beam. First, three different 3D micro-scale handles are designed, then manufactured by the two-photon polymerization method with nano-scale resolution of 100 nm. Second, printed microhands are manipulated by multi-spot laser beam which traps and manipulates numerous objects simultaneously. Third, where direct trapping of the target object is not possible due to target objects' features such as size, shape, material, index of refraction, etc., indirect manipulation of the target microobjects is achieved by using the microhands as an extension of optical tweezers. Finally, three different microhand designs are compared in terms of speed and success rate. Furthermore, suitability of different shapes of microhandles against common usage of spherical shape is discussed.

Motor channelling is a concept to provide navigation and sensory feedback to operators in master-slave surgical setups. It is beneficial since the introduction of robotic surgery creates a physical separation between the surgeon and patient anatomy. Active Constraints/Virtual Fixtures are proposed which integrate Guidance and Forbidden Region Constraints into a unified control framework. The developed approach provides guidance and safe manipulation to improve precision and reduce the risk of inadvertent tissue damage. Online three-degree-of-freedom motion prediction and compensation of the target anatomy is performed to complement the master constraints. The presented Active Constraints concept is applied to two clinical scenarios; surface scanning for in situ medical imaging and vessel manipulation in cardiac surgery. The proposed motor channelling control strategy is implemented on the da Vinci Surgical System using the da Vinci Research Kit (dVRK) and its effectiveness is demonstrated through a detailed user study.

Eye tracking technology has shown promising results for allowing hands-free control of robotically-mounted cameras and tools. However existing systems present only limited capabilities in allowing the full range of camera motions in a safe, intuitive manner. This paper introduces a framework for the recognition of surgeon intention, allowing activation and control of the camera through natural gaze behaviour. The system is resistant to noise such as blinking, while allowing the surgeon to look away safely at any time. Furthermore, this paper presents a novel approach to control the translation of the camera along its optical axis using a combination of eye tracking and stereo reconstruction. Combining eye tracking and stereo reconstruction allows the system to determine which point in 3D space the user is fixating, enabling a translation of the camera to achieve the optimal viewing distance. In addition, the eye tracking information is used to perform automatic laser targeting for laser ablation. The desired target point of the laser, mounted on a separate robotic arm, is determined with the eye tracking thus removing the need to manually adjust the laser's target point before starting each new ablation. The calibration methodology used to obtain millimetre precision for the laser targeting without the aid of visual servoing is described. Finally, a user study validating the system is presented, showing clear improvement with median task times under half of those of a manually controlled robotic system.

This paper presents an intelligent sewing system for personalized stent graft manufacturing, a challenging sewing task that is currently performed manually. Inspired by medical suturing robots, we have adopted a single-sided sewing technique using a curved needle to perform the task of sewing stents onto fabric. A motorized surgical needle driver was attached to a 7 d.o.f robot arm to manipulate the needle with a second robot controlling the position of the mandrel. A learning-from-demonstration approach was used to program the robot to sew stents onto fabric. The demonstrated sewing skill was segmented to several phases, each of which was encoded with a Gaussian Mixture Model. Generalized sewing movements were then generated from these models and were used for task execution. During execution, a stereo vision system was adopted to guide the robots and adjust the learnt movements according to the needle pose. Two experiments are presented here with this system and the results show that our system can robustly perform the sewing task as well as adapt to various needle poses. The accuracy of the sewing system was within 2mm.

Safe and effective telemanipulation of concentric tube robots is hindered by their complex, non-intuitive kinematics. Guidance schemes in the form of attractive and repulsive constraints can simplify task execution and facilitate natural operation of the robot by clinicians. The real-time seamless calculation and application of guidance, however, requires computationally efficient algorithms that solve the non-linear inverse kinematics of the robot and guarantee that the commanded robot configuration is stable and sufficiently away from the anatomy. This paper presents a multi-processor framework that allows on-the-fly calculation of optimal safe paths based on rapid workspace and roadmap pre-computation The real-time nature of the developed software enables complex guidance constraints to be implemented with minimal computational overhead. A user study on a simulated challenging clinical problem demonstrated that the incorporated guiding constraints are highly beneficial for fast and accurate navigation with concentric tube robots.

This paper introduces the design and characterization of a robotic surgical instrument produced mainly with rapid prototyping techniques. Surgical robots have generally complex structures and have therefore an elevated cost. The proposed instrument was designed to incorporate minimal number of components to simplify the assembly process by leveraging the unique strength of rapid prototyping for producing complex, assemble-free components. The modularity, cost-effectiveness and fast manufacturing and assembly features offer the possibility of producing patient or task specific instruments. The proposed robot incorporates an integrated force measurement system, thus allowing the determination of the force exchanged between the instrument and the environment. Detailed experiments were performed to validate the functionality and force sensing capability of the instrument.

The use of conventional surgical tool holders requires an assistant during positioning and adjustment due to the lack of weight compensation. In this paper, we introduce a robotic arm system with hands-on control approach. The robot incorporates a force sensor at the end effector which realises tool weight compensation as well as hands-on manipulation. On the operating table, the required workspace can be tight due to a number of instruments required. There are situations where the surgical tool is at the desired location but the holder arm pose is not ideal due to space constraints or obstacles. Although the arm is a non-redundant robot because of the limited degrees of freedom, the pseudo-null-space inverse kinematics can be used to constrain a particular joint of the robot to a specific angle while the other joints compensate in order to minimise the tool movement. This allows operator to adjust the arm configuration conveniently together with the weight compensation. Experimental results demonstrated that our robotic arm can maintain the tool position during reconfiguration significantly more stably than a conventional one.

Presents the introductory editorial for this issue of the publication.

After decades of evolution, measuring instruments for quantitative gait analysis have become an important clinical tool for assessing pathologies manifested by gait abnormalities. However, such instruments tend to be expensive and require expert operation and maintenance besides their high cost, thus limiting them to only a small number of specialized centers. Consequently, gait analysis in most clinics today still relies on observation-based assessment. Recent advances in wearable sensors, especially inertial body sensors, have opened up a promising future for gait analysis. Not only can these sensors be more easily adopted in clinical diagnosis and treatment procedures than their current counterparts, but they can also monitor gait continuously outside clinics - hence providing seamless patient analysis from clinics to free-living environments. The purpose of this paper is to provide a systematic review of current techniques for quantitative gait analysis and to propose key metrics for evaluating both existing and emerging methods for qualifying the gait features extracted from wearable sensors. It aims to highlight key advances in this rapidly evolving research field and outline potential future directions for both research and clinical applications.

The year 2016 marks the 200th birth anniversary of Carl Friedrich Wilhelm Ludwig (1816-1895). As one of the most remarkable scientists, Ludwig invented the kymograph, which for the first time enabled the recording of continuous blood pressure (BP), opening the door to the modern study of physiology. Almost a century later, intraarterial BP monitoring through an arterial line has been used clinically. Subsequently, arterial tonometry and volume clamp method were developed and applied in continuous BP measurement in a noninvasive way. In the last two decades, additional efforts have been made to transform the method of unobtrusive continuous BP monitoring without the use of a cuff. This review summarizes the key milestones in continuous BP measurement; that is, kymograph, intraarterial BP monitoring, arterial tonometry, volume clamp method, and cuffless BP technologies. Our emphasis is on recent studies of unobtrusive BP measurements as well as on challenges and future directions.

Reconstruction of the anterior cruciate ligament (ACL) through arthroscopy is one of the most common procedures in orthopaedics. It requires accurate alignment and drilling of the tibial and femoral tunnels through which the ligament graft is attached. Although commercial computer-assisted navigation systems exist to guide the placement of these tunnels, most of them are limited to a fixed pose without due consideration of dynamic factors involved in different knee flexion angles. This paper presents a new model for intraoperative guidance of arthroscopic ACL reconstruction with reduced error particularly in the ligament attachment area. The method uses 3D preoperative data at different flexion angles to build a subject-specific statistical model of knee pose. To circumvent the problem of limited training samples and ensure physically meaningful pose instantiation, homogeneous transformations between different poses and local-deformation finite element modelling are used to enlarge the training set. Subsequently, an anatomical geodesic flexion analysis is performed to extract the subject-specific flexion characteristics. The advantages of the method were also tested by detailed comparison to standard Principal Component Analysis (PCA), nonlinear PCA without training set enlargement, and other state-of-the-art articulated joint modelling methods. The method yielded sub-millimetre accuracy, demonstrating its potential clinical value.

This paper proposes a novel concept of using a multiple PPG and ECG based sensing platform aimed for monitoring the progress of diabetic peripheral neuropathy (DPN). It explores the use of PPG sensor to capture pulse arrival time (PAT). Based on the same principal of using Brachial-ankle pulse wave velocity (baPWV) to assess DPN, this paper proposes a platform which integrated two PPG sensors and one 2-lead ECG sensor to detect the difference in PAT (pulse arrive time on the finger compare to the time when the pulse reaches the ankle) as a surrogate measure for evaluating the progression of DPN. Preliminary results show that PAT increases when a pressure was applied onto upper leg using a blood pressure cuff simulating arterial stiffness/DPN. It shows that PDN can potentially be quantified by measuring PAT by using the proposed platform.

Similar to fingerprint and iris pattern, everyone's gait is unique, and gait has been proposed as a biometric feature for security applications. This paper presents a lightweight accelerometer-based technique for user authentication on smart wearable devices. Designed as an unsupervised classification approach, the proposed authentication technique can learn the user's gait pattern automatically when the user first starts wearing the device. Anomaly detection is then used to verify the device owner. The technique has been evaluated both in controlled and uncontrolled environments, with 20 and 6 healthy volunteers respectively. The Equal Error Rate (EER) in the controlled environments ranged from 5.7% (waist-mounted sensor) to 8.0% (trouser pocket). In the uncontrolled experiment, the device was put in the subject's trouser pocket, and the results were similar to the respective supervised experiment (EER=9.7%).

Recent trends in wearable applications demand flexible architectures being able to monitor people while they move in free-living environments. Current solutions use either store-download-offline processing or simple communication schemes with real-time streaming of sensor data. This limits the applicability of wearable applications to controlled environments (e.g, clinics, homes, or laboratories), because they need to maintain connectivity with the base station throughout the monitoring process. In this paper, we present the design and implementation of an opportunistic communication framework that simplifies the general use of wearable devices in free-living environments. It relies on a low-power data collection protocol that allows the end user to opportunistically, yet seamlessly manage the transmission of sensor data. We validate the feasibility of the framework by demonstrating its use for swimming, where the normal wireless communication is constantly interfered by the environment.

This paper analyses gait patterns of patients with Parkinson;s Disease (PD) based on the acceleration data given by an e-AR sensor. Ten PD patients wearing the e-AR sensor walked along a 7m walkway and each session contained 16 repeated trials. An iterative algorithm has been proposed to produce robust estimations in the case of measurement noise and short-duration of gait signals. Step-frequency as a gait parameter derived from the estimated heel-contacts is calculated and validated using the CODA motion-capture system. Intersession variability of step-frequency for each patient and the overall variability across patients demonstrate a good agreement between estimations from the e-AR and CODA systems.

Surgical Site Infection (SSI) imposes a significant burden clinically and compromises patient recovery. Anastomosis in the gastrointestinal (GI) tract is a particularly challenging case where failure of the anastomosis can lead to leakage, resulting in an increase in mortality rates. However early diagnosis and intervention are hampered by a lack of continuous sensing and long diagnostic intervals of current clinical practices. Tissue ischemia in the vicinity of the anastomosis has been found to be an early surrogate marker for anastomotic leakage. Electrical bio-impedance is a promising non-invasive technique for identifying and monitoring tissue ischemia. In this paper the modelling, design and validation of a bio-impedance system including compact instrumentation and a novel bio-impedance sensor optimized for mucosal tissue measurements in the GI tract are presented. The preliminary system, including the impedance probe, is validated experimentally for GI implant applications to provide early detection of tissue ischemia following GI surgery.

Assessment of food intake has a wide range of applications in public health and life-style related chronic disease management. In this paper, we propose a real-time food recognition platform combined with daily activity and energy expenditure estimation. In the proposed method, food recognition is based on hierarchical classification using multiple visual cues, supported by efficient software implementation suitable for realtime mobile device execution. A Fischer Vector representation together with a set of linear classifiers are used to categorize food intake. Daily energy expenditure estimation is achieved by using the built-in inertial motion sensors of the mobile device. The performance of the vision-based food recognition algorithm is compared to the current state-of-the-art, showing improved accuracy and high computational efficiency suitable for realtime feedback. Detailed user studies have also been performed to demonstrate the practical value of the software environment.

Sensor-to-segment calibration is a critical step for motion reconstruction from inertial and magnetic measurement units (IMMUs). In this paper, a novel sensor-to-segment calibration protocol is proposed. The protocol consists of three stages that allow for in situ calibration. After the sensor units are attached to the body, predefined postures and movements are used for sensor calibration. Acceleration and angular velocity measurements are used to estimate axes of functional frame (FF) by Principal Component Analysis (PCA). Finally, Levenberg-Marquardt optimization is used to identify rotation matrices between the expected FF and their estimations with respect to the sensor frame. Validation of the method demonstrates its practical value and how the proposed protocol reduces the extent of cross-talk for evaluating joint kinematics.

Quality of sleep is an important index of wellbeing and health. Irregular sleep patterns are often associated with stress and disorders such as cardiovascular disease, diabetes, depression, sleep apnea and obesity. In addition to key physiological indices, body movements and posture during sleep are also important for assessing causal relationship of irregular sleep patterns and underlying health issues. In this paper, we explore the feasibility of using a single accelerometer strapped onto the chest to detect posture and cardio-respiratory parameters during sleep. An efficient movement detector suitable for on-node implementation is developed to distinguish static postures from dynamics movements. When in static postures, a linear discriminant analysis (IDA) classifier is used to further divide the static postures into four common sleeping positions. Simultaneously, both heart rate and respiratory rate are extracted from the acceleration signal. A small cohort of 7 healthy subjects were recruited for lab-controlled experiments to evaluate the performance of our proposed methods. ECG signal and K4b2 system's V02 measurements were also collected to extract heart rate and respiratory rate as the ground truth for comparison. An overall classification accuracy of 99% is achieved for recognising the correct sleeping positions. Good matches to ground truths were also obtained for the derived cardiac and respiratory rates.

With increasing use of cloud storage for healthcare applications, potential security risks and the need for enhanced security solutions are becoming a pressing issue for clinical adoption. In this paper, a data partitioning and scrambling method at the application layer is proposed for healthcare data, where a tiny part of the original data is used to scramble the remaining data without any cryptographic key, and the former is kept locally while the latter under extra protection is sent to cloud platforms. Theoretical and experimental analyses have been carried out to demonstrate the security performance of the proposed method, which can be easily deployed in any existing communication systems as an add-on for security.

Transanal Endoscopic Microsurgery (TEM) is a minimally invasive oncological resection procedure that utilises a natural orifice approach rather than the traditional abdominal or open approach. However, TEM has a significant recurrence rate due to incomplete excisions, which can possibly be attributed to the absence of intraoperative image guidance. The use of real-time histological data could allow the surgeons to assess the surgical margins intraoperatively and adjust the procedure accordingly. This paper presents the integration of endomicroscopy and ultrasound imaging through a robotically actuated instrument. Endomicroscopy can provide high resolution images at a surface level while ultrasound provides depth resolved information at a macroscopic level. Endomicroscopy scanning is achieved with a novel scanning approach featuring a passive force adaptive mechanism. The instrument is manipulated across the surgical workspace through an articulated flexible shaft. This results in the ability to perform large area mosaics coupled with ultrasound scanning. In addition, the use of endoscopic tracking is demonstrated, allowing three-dimensional reconstruction of the ultrasound data displayed onto the endoscopic view. An ex vivo study on porcine colon tissue has been performed, demonstrating the clinical applicability of the instrument.

Whilst current minimally invasive surgical robots offer many advantages to the surgeon, most of them are still controlled using the traditional master-slave approach, without fully exploiting the complementary strengths of both the human user and the robot. This paper proposes a framework that provides a cooperative control approach to human-robot interaction. Typical teleoperation is enhanced by incorporating haptic guidance-based feedback for surgical tasks, which are demonstrated to and learned by the robot. Safety in the surgical scene is maintained during reproduction of the learned tasks by including the surgeon in the guided execution of the learned task at all times. Continuous Hidden Markov Models are used for task learning, real-time learned task recognition and generating setpoint trajectories for haptic guidance. Two different surgical training tasks were demonstrated and encoded by the system, and the framework was evaluated using the Raven II surgical robot research platform. The results indicate an improvement in user task performance with the haptic guidance in comparison to unguided teleoperation.

Despite the increased use of robotic catheter navigation systems, and the growing interest in surgical skill evaluation in the field of endovascular intervention, there is a lack of objective and quantitative metrics for performance evaluation. So far very little research has studied operator behavioral patterns using catheter kinematics, operator forces and motions, and catheter-tissue interactions. This paper proposes a framework for automated and objective assessment of performance by measuring catheter-tissue contact forces and operator motion patterns across different skill levels, and using language models to learn the underlying force and motion patterns that are characteristic of skill. Discrete HMMs are utilized to model operator behavior for varying skill levels performing different catheterization tasks, resulting in cross-validation classification accuracies of 94% (expert) and 98% (novice) using the force-based skill models, as well as 83% (expert) and 94% (novice) using the motion-based models. The results motivate the design of improved metrics for endovascular skill assessment with further applications towards performance evaluation of robot-assisted endovascular catheterization.

A fundamental problem with dual-arm robotic control is to find the coordinated motion resolution under high kinematic redundancy and intrinsic constraints of each robot. To solve this problem, this paper presents a multi-tasking, co-operative control framework, in which potential task conflicts and robot joint constraints are properly handled. Based on the relative Jacobian formulation, singularity-robust inverse kinematics and the scheme of null space distributing exceeded joint velocity, this work contributes by introducing a framework to handle multi-tasking conflicts both in task and joint space for dual-arm robots. Detailed validation of the proposed framework is first conducted by using a simulated dual-arm robot, followed by a demonstration on two 7-dof Kuka lightweight manipulators in a bimanual stent graft manufacturing task.

Concentric tube robots are catheter-sized robots that are ideally suited for navigating along natural anatomical pathways and treating deep-seated pathologies. Their telemanipulation in dynamic environments requires on-line computation of inverse kinematics with simultaneous avoidance of anatomical obstacles. Moreover, unstable configurations, which arise for elongated curved robots that navigate extremely tortuous paths, must be avoided. To achieve on-line computations, existing work has investigated Jacobian approximations and configuration-space precomputation. This paper leverages the state-of-the-art multi-core computer architectures to deliver real-time local inverse kinematics solutions using the established concentric tube robot mechanics models while avoiding both instabilities and anatomical collisions. Furthermore, it considers frictional active constraints for concentric tube robots, i.e. viscoelastic force fields that guide the operator away from obstacles and towards safe configurations. The value of the proposed framework is demonstrated on realistic clinical scenarios.

The surgical robotics community have developed many different flexible robot designs to address the access problems of minimally invasive surgery. In this paper, we present a hand-held mechatronic tool with a miniaturized distal flexible manipulator incorporating a microscopy probe, camera and a light source for diagnostic arthroscopy. Extensive characterization of the flexible manipulator is provided, including an optimization of the manipulator workspace, hysteresis characteristics and repeatability of the instrument. The lateral stiffness of the flexible manipulator for different bending conditions is assessed along with the overall robustness of the platform. A cadaveric study was performed to demonstrate the potential clinical value of the device.

Micromanipulation tasks are usually carried out using simplistic tools such as rigid probes and needles. More sophisticated tools such as grippers are fragile, expensive and non-dexterous. This paper addresses some of the main challenges of manipulation at the micrometer scale, including robots with limited degrees of freedom, small range of available tools and open-loop control due to a lack of position sensors. This work presents a preliminary investigation into the viability of using the Direct Laser Writing and Two-Photon Polymerization microfabrication techniques for creating flexible micrometer-scale end-effectors. A novel compliant end-effector design is presented for closed-chain cooperative manipulation involving multiple micromanipulator robots. A visual servoing framework was also developed to allow the user to control the robots in a closed loop manner, with haptic feedback to help match the user's input to the speed of the robots. Characterization of the new compliant manipulator was conducted and multi-robot configurations were tested to demonstrate the flexibility, robustness and increased workspace of the new design for both 2D and 3D object manipulation tasks.

Additive manufacturing is a technology in constant evolution. It is able to produce objects otherwise either impractical or unfavorable for traditional manufacturing technologies, especially in relatively limited quantities. The advancement of Selective Laser Melting (SLM) has made it possible to manufacture functional, production-quality components directly from rapid prototyping for long-term use. This work studies the SLM process and identifies the optimal process parameters for the rapid manufacturing of a miniaturized robotic surgical instrument. The proposed robotic instrument has been designed to exploit the advantages of rapid manufacturing and rapid assembly, following a shift from large scale manufacturing of generalized surgical instruments to the production of small batches of patient or procedure-specific.

Concentric tube robots have shown promise for minimally invasive surgical (MIS) tasks that require navigation via tortuous anatomical paths. Despite extensive research on their kinematic and dynamic modelling, however, inaccuracies and deformations of their shape due to unknown loads and collisions with the anatomy make intraoperative shape sensing a requirement. This paper presents a vision-based shape-sensing algorithm for concentric tube robots. The proposed algorithm fuses information extracted from a standard imaging modality, monoplane X-ray fluoroscopy, with the kinematics model of the concentric tube robot, to achieve automatic, real-time, accurate and continuous robot-shape estimations despite kinematics' noise and unmodelled forces. Fusion is performed by a fast 2D/3D non-rigid registration, which combines kinematics and intraoperative tracking of the robot. Extensive simulations with a range of noise models and virtual loads acting on the robot, and experimental evaluation in air and in a skull phantom, demonstrate the clinical value of the proposed technique1.

As mobile robots are becoming increasingly more popular in workplaces, public places and homes, the understanding of surface dynamics is one of the new challenges to improve their mobility and reduce overall disturbance in the human environment. This work proposes a comparison between different classifiers of identifying surfaces by using a 3-axis accelerometer sensor on a holonomic robot. The robot is equipped with four omni-directional wheels and the sensor is mounted on its frame to measure the whole body vibration. Four typical indoor floors were tested over four robot motions. Final results show that a mixture of statistical and spectrum density based features are sufficient to identify surfaces with more than 85% overall accuracy.

Presents an introductory editorial on the special issue for this issue of the publication which examines the fields of biomedical and health informatics.

This paper provides an overview of recent developments in big data in the context of biomedical and health informatics. It outlines the key characteristics of big data and how medical and health informatics, translational bioinformatics, sensor informatics, and imaging informatics will benefit from an integrated approach of piecing together different aspects of personalized information from a diverse range of data sources, both structured and unstructured, covering genomics, proteomics, metabolomics, as well as imaging, clinical diagnosis, and long-term continuous physiological sensing of an individual. It is expected that recent advances in big data will expand our knowledge for testing new hypotheses about disease management from diagnosis to prevention to personalized treatment. The rise of big data, however, also raises challenges in terms of privacy, security, data ownership, data stewardship, and governance. This paper discusses some of the existing activities and future opportunities related to big data for health, outlining some of the key underlying issues that need to be tackled.

Humanoid robots have been used in a wide range of applications including entertainment, healthcare, and assistive living. In these applications, the robots are expected to perform a range of natural body motions, which can be either preprogrammed or learnt from human demonstration. This paper proposes a strategy for imitating dynamic walking gait for a humanoid robot by formulating the problem as an optimization process. The human motion data are recorded with an inertial sensor-based motion tracking system (Biomotion+). Joint angle trajectories are obtained from the transformation of the estimated posture. Key locomotion frames corresponding to gait events are chosen from the trajectories. Due to differences in joint structures of the human and robot, the joint angles at these frames need to be optimized to satisfy the physical constraints of the robot while preserving robot stability. Interpolation among the optimized angles is needed to generate continuous angle trajectories. The method is validated using a NAO humanoid robot, with results demonstrating the effectiveness of the proposed strategy for dynamic walking.

Photoplethysmography (PPG) is a well established method of measuring Heart Rate Variability (HRV) and blood oxygen saturation (SpO2) at the fingers, forehead or other areas of the body where pulsatile flow is present. However, obtaining reliable tissue oxygen saturation (StO2) from optical devices is more challenging due to a number of factors including motion and signal-to-noise ratio. Instrumentation of such devices as miniaturised wearable platforms would allow the device to be worn freely by patients in hospitals or at home. The purposes of this paper are to present: 1) a bespoke, low power StO2 sensor, 2) preliminary comparison to a commercially available photospectroscopy and laser Doppler machine (Oxygen 2 See, Medizintechnik, LEA, Germany) using a pressure cuff forearm ischaemia model, and 3) validation of fluid (blood) flow/relative haemoglobin measurements using a novel forearm phantom.

The use of e-AR (ear-worn activity recognition) sensorfor gait pattern estimation has shown promise for a range of health and wellbeing applications. To establish its more detailed quantitative accuracy, an in-shoe pressure measurement system (Parotec) has been used to validate the estimated gait events from the e-AR sensor. Ten healthy adults equipped with Parotec and e-AR systems walked in acorridor of about 15m. The sampling frequency of both systems was set at 100Hz and a manual synchronisation has been performed for subsequent error measurements. The gait events from the e-AR sensor are estimated by using a recently developed method based on singular spectrum analysis and longest common subsequence algorithms [1]. Thecorresponding gait events from the Parotec system are estimated using the ground reaction forces. The upper and lower limits of absolute errors using 95% confidence intervals for heel contact and toe off events obtained as 35.38±3.22ms and 73.05±7.24ms respectively. We furtherprovide a preliminary patient study to demonstrate how the estimated gait events and the gait analysis platform can be used for assessing patients recovering after orthopaedic surgery inside the clinic.

Sensor based motion capture system enables motion analysis with applications ranging from entertainment, healthcare and robotics. It can provide an intuitive interface for human to provide motion training and control a humanoid robot. In this paper, we propose a novel framework for the imitation of human motion for a humanoid robot. In the proposed framework, human motion data is directly captured from a wireless, wearable motion capture platform (Biomotion+). The reconstructed posture is then converted into joint angle trajectories. Due to the structural differences between the joints of the robot and those of the human, the trajectories are then optimized to satisfy the mechanical constraints of the robot and to maintain appropriate balance. To validate the proposed framework, different motion trajectories were verified. The results demonstrate the stability and effectiveness of the proposed framework to reproduce realistic human motion for a humanoid robot and the potential for a tele-rehabilitation application. The proposed framework offers a new way of imitating human motion for a humanoid robot.

In recent years, robotic systems have been playing an increasingly important role in physiotherapy. The aim of these platforms is to aid the recovery process from strokes or muscular damage by assisting patients to perform a number of controlled tasks, thus effectively complementing the role of the physiotherapist. In this paper, we present a novel learning from demonstration framework for cooperative control in robotic-assisted physiotherapy. Unlike other approaches, the aim of the proposed system is to guide the patients to optimally execute a task based on previously learned demonstrations. This allows the generation of patient-specific gestures under the supervision of the expert physiotherapist. The guidance is performed through stiffness control of a compliant manipulator, where the stiffness profile of the generalized trajectory is determined according to the relative importance of each section of the task. In contrast with the traditional learning approach, where the execution of the generalized trajectory by the robot is automated, this cooperative control architecture allows the patients to perform the task at their own pace, while ensuring the movements are executed correctly. Increased performance of the learning framework is accomplished through a novel fast, low-cost multi-demonstration dynamic time warping algorithm used to build the model. Experimental validation of the framework is carried out using an interactive setup designed to provide further guidance through additional visual and sensory feedback based on the task model. The results demonstrate the potential of the proposed framework, showing a significant improvement in the performance of guided tasks compared to unguided ones.

Active constraints are high-level control algorithms providing software-generated force feedback from virtual environments. When applied to surgery, they can assist surgeons in performing complex tasks by guiding their navigation pathways along narrow, possibly convoluted, surgical trajectories. This paper presents a method to generate concave tubular constraints implicitly from pre- or intra-operative data. Patient-specific constraints may be generated efficiently with the proposed scheme and readily deployed in various surgical scenarios. Furthermore, a five degree-of-freedom active constraint framework is proposed, which accounts for the entire tool shaft rather than just the end-effector, and is applicable to both static and dynamic active constraint scenarios. Experimental results on simulated surgical tasks show that this framework can improve safety and accuracy as well as reduce the perceived workload during complex surgical tasks.

This paper introduces the CYCLOPS, a novel robotic tool for single-access and natural-orifice endoscopic surgery. Based on the concept of tendon-driven parallel robots, this highly original design gives the system some of its unique capabilities. Just to name a few, unparalleled force exertion capabilities of up to 65N, large and adjustable workspace, bimanual instrument triangulation. Due to the simplicity and nature of the design, the system could be adapted to an existing laparoscope or flexible endoscope. This promises a more immediate and accelerated route to clinical translation not only through endearing low-cost and adaptive features, but also by directly addressing several major barriers of existing designs.

This paper presents a hand-held microsurgical forceps design with force-feedback capabilities designed for micromanipulation tasks. The device uses a customized force sensor that measures grasping forces over a range of 0-300mN and uses an actuator to exert amplified forces back on to the operator's fingertip in a mechanically-ungrounded setup. This allows perception of low force levels that are otherwise imperceptible to human touch. A customized force sensor design for the forceps grasping measurement is presented and a calibration experiment was conducted to validate its linearity and repeatability. A bench test of the device was conducted to demonstrate its intrinsic force-amplifying capabilities, with amplification factors of up to ×50 reported. A user study was conducted to confirm that the device could significantly improve human perception of grasping forces compared to conventional microsurgical forceps with the results demonstrating an order-of-magnitude improvement in force perception.

In recent years, continuum robots have gained significant momentum in terms of technological maturity and clinical application. Their flexibility allows complex treatment sites to be reached with minimal trauma to the patient. However the reliable control of continuum robots is still an ongoing research issue in the robotics community because their deformable structure makes the modeling of these devices difficult. This motivates the use of external sensors or vision to achieve accurate control. In this paper, a motion control framework based on a vision sensor is proposed in order to perform accurate and controlled movements of a flexible robot that is mounted to an anthropomorphic robotic arm. The vision sensor, which relies on a single camera, provides accurate 3D shape reconstruction and spatial localisation of the flexible robot. This information is used to provide feedback for the real-time control of the flexible robot. The vision sensor detects the robot first in an image stream by modeling its appearance using compressed visual features in an online learning framework. This is combined with the kinematics information from the anthropomorphic robotic arm in order to accurately reconstruct and localise the 3D shape of the flexible robot by minimizing an energy function. Detailed analysis of the framework and a validation are presented in order to demonstrate the practical value of the proposed method.

This paper presents a novel blind collision detection and material characterisation scheme for a compliant robotic arm. By the incorporation of a simple MEMS accelerometer at each joint, the robot is able to detect collision, identify the material of an obstacle, and create a map of the environment. Detailed hardware design is provided, illustrating its value for building a compact and economical robot platform. The proposed method does not require the additional use of vision sensor for mapping the environment, and hence is termed as `blind' collision detection and environment mapping. Based on the shock wave and vibration signals, the proposed algorithm is able to classify a range of materials encountered. Detailed laboratory evaluation was performed with controlled obstacle collision from different orientation and locations with varying force and materials. The proposed method has achieved 98% detection sensitivity while maintaining 77% specificity. Furthermore, by using sound feature extraction and machine learning techniques, the classifier produces an accuracy of 98% for classifying four different impact materials. In this paper, we also demonstrate its use for detailed environment mapping by using the proposed method.

Recent work on probe-based confocal endomicroscopy has demonstrated its potential role for real-time assessment of tumour margins during breast conserving surgery. However, endomicroscope probes tend to have a very small field-of-view, making surveillance of large areas of tissue difficult, and limiting practical clinical deployment. In this paper, a new robotic device for controlled, large area scanning based on a fibre bundle endomicroscope probe is proposed. The prototype uses a 2-DOF mechanism (−90 to +90 degrees bending on one axis, 360 degrees of rotation on a second axis) as well as a passive linear structure to conform to undulating surfaces. Both axes are driven by brushless DC servo motors with computer control, thus facilitating large field-of-view mosaicing. Experimental results have shown good repeatability and low hysteresis of the device, which is able to scan different surface trajectories (e.g. a spiral pattern over a hemi-spherical surface) with consistent tissue contact. Ex vivo human breast tissue results are demonstrated, illustrating a viable scanning approach for breast endomicroscopy.

Precise and repetitive positional control of surgical robots is important to reduce time and risks of surgical procedures. These factors become particularly important when deploying the surgical system through a flexible path to areas with a tight workspace such as the stomach or oesophagus where high dexterity, flexibility, accuracy and stability are required. This paper presents a flexible access robot combining articulated joints and continuum flexible section for both transluminal and endoluminal surgeries. Kinematic model and control strategy for the flexible robot are described in the paper. The experiment simulating a transoral gastric procedure demonstrates great flexibility and dexterity of the device. The results show that good accuracy and repetitive control of the device are achieved, which demonstrate the potential application of the device for transluminal or endoluminal surgery.

This paper proposes a new vasculature reconstruction and catheter modeling scheme based on data fusion from intravascular ultrasound (IVUS) imaging, electromagnetic (EM) tracking and shape sensing for trans-femoral Transcatheter Aortic Valve Implantation (TAVI). The system is suitable for obtaining inner cross sectional images of the aorta with an IVUS probe, reconstructing its 3D virtual model using sensor fusion of the corresponding pose information of IVUS probe from an electromagnetic (EM) sensor, as well as reconstructing the catheter shape based on optical fibers with Fiber Bragg Grating (FBG) sensors. A hybrid probe consisting of an IVUS sensor, an EM sensor and an optical shape sensor has been created and tested on in-vitro silicone aortic phantoms. A practical image processing method based on the gradient vector flow (GVF) snake has been proposed, followed by fusion with pose information from an EM sensor for the anatomical model reconstruction. Demonstration of the proposed method was performed on two aortic phantoms. Preliminary results show how the catheter shape reconstruction is realized by the shape sensor. The proposed method could facilitate intra-operative surgical guidance for valve alignment, improve the precision for positioning, reduce the time of the TAVI procedure, minimize the use of contrast agent, and assess the status of the deployed valve after surgery.

This paper proposes a semi-autonomous navigated master-slave system, for robot assisted remote echography for early trauma assessment. Two RGB-D sensors are used to capture real-time 3D information of the scene at the slave side where the patient is located. A 3D statistical shape model is built and used to generate a customized patient model based on the point cloud generated by the RGB-D sensors. The customized patient model can be updated and adaptively fitted to the patient. The model is also used to generate a trajectory to navigate a KUKA robotic arm and safely conduct the ultrasound examination. Extensive validation of the proposed system shows promising results in terms of accuracy and robustness.

In activity recognition and behaviour profiling studies, wearable inertial sensors are commonly used to monitor the subjects' daily activities. However, the need of carrying the sensing devices in addition to personal belongings may prohibit the widespread use of the technologies. On the other hand, smartphones have become ubiquitous and most smartphones are already equipped with similar inertial sensors. Recent studies have proposed the use of smartphone for quantifying the activity and behaviour of the users. A smartphone based long-term routine profiling system is proposed. To simplify the user interface and facilitate the ubiquitous use of the system, unsupervised and optimized techniques have been developed and integrated into a mobile phone application. By running the application continuously in the background of the phone, the system captures and processes the sensing information to infer the activities of the users, and the results are forwarded to the server for profiling the routines using pattern mining techniques. The proposed system is validated through a study of six users over two weeks. The ability of the proposed system in capturing routine behavior is demonstrated in the results of the study.

To enable continuous monitoring of electrochemical sensors outside the laboratories, there is a significant demand on electrochemical measurement systems to be miniaturized. The paper presents a low-cost, portable miniature device for electrochemical measurements. The device consists of a 35mm × 20mm × 25mm wireless tag, which enables potentiometric and amperometric measurement, and a base station for data acquisition. Potentiometric performance is evaluated using solid contact ion selective electrodes for pH and sodium. High sensitivity, repeatability and fast response time have been achieved. Amperometric measurement of hydrogen peroxide shows that the measured current accuracy and sensitivity are comparable to that of a commercial potentiostat. Moreover, the device can be used for lactate concentration sensing.

Assessing psychological stress is essential for monitoring general health and wellbeing. One key element is the detection of the stimulus, i.e., stressor that evokes a stress response. Visual and verbal stimuli are elementary arousal elements of daily stress responses. The study aim was to discriminate the stress responses from watching videos and speaking using electrodermal activity (EDA) and heart rate variability (HRV) measures. A cohort of 12 subjects completed a laboratory experiment comprising of 4 psychological tasks (watching a relaxing video and a violent video, speaking by counting and speaking on an unknown topic). In total, 17 physiological features were calculated from the EDA and HRV signals. Four classifiers were investigated regarding their ability to discriminate between verbal and visual stimulated stress responses with a maximum accuracy of 92% achieved. This demonstrates that the measured signals have potential for tracking and differentiating the stress responses of watching videos or speaking in real-time by using wearable EDA and HRV devices.

This paper proposes a new approach to gait pattern analysis based on acceleration signals during different walking conditions. Instead of applying traditional classification techniques, the proposed method looks into the characteristics of acceleration signals. Filtering and template matching methods based on singular spectrum analysis (SSA) and longest common subsequence algorithm (LCSS) have been used. The method has been used to discriminate walking downstairs, level walking and walking upstairs using 10 healthy subjects. The results suggest that the proposed method gives new insight into quantitative aspects of gait patterns.

The aim of this work is to provide a humanoid robot that is able to replicate human's upper body movements by using motion capture data acquired from Biomotion+, developed by the Hamlyn Centre. This work proposes an upper limb motion imitation module for a humanoid robot. The module calculates joint angle trajectories, based on motion capture data, and sends these trajectories to a humanoid robot. The experimental results have demonstrated the effectiveness of the module which can achieve reasonable postural similarity of generated robot motions, compared to the captured human movements.

Pervasive human motion capture in the workplace facilitates detailed analysis of the actions of individual subjects and team interaction. It is also important for ergonomic studies for assessing instrument design and workflow analysis. However, a busy, dynamic, team-based environment, such as the operating theatre poses a number of challenges for the currently used marker-based and sensor-based motion capture systems. Occlusions and sensor drift can affect the accuracy of the estimated motion. In this paper, we present a motion capture system that uses a vision-based head detection algorithm and a markerless inertial motion capture for estimating the motion of multiple people. The pose estimation obtained through inertial sensors is combined with location obtained through vision-based tracking to reconstruct the motion of each subject. A multi-target Kalman filter is used to track the movement of each subject. To handle the close proximity of the subjects, visual features associated with the body are used for data association. Experimental results demonstrate the accuracy of the proposed system.

Myoelectric signal analysis provides insight into neural control during muscle contraction and it has been widely used to identify the intention of performing different movements for patients with disabilities. Previous studies have demonstrated that detailed neural control information could be extracted from high-density surface electromyography (EMG) signals. However, this imposes practical constraints for routine applications. In this paper, we present an analysis framework using low-density EMG with example experiments demonstrating the control of forearm functional movement Eight channel surface EMG signals are used with subjects performing 6 different forearm and hand movements. Data analysis consisting of feature selection and pattern classification based on KNN, linear discriminant analysis and support vector machine is then performed. High classification accuracy has been achieved for all the subjects, illustrating the practical value of the method proposed.

Recent technological advances in surgery have permitted cellular and molecular imaging to be carried out intra-operatively. Although optical biopsy techniques such as probe-based confocal laser endomicroscopy (pCLE) have enabled real-time diagnosis and tissue characterisation in vivo, the flexibility of the probe introduces significant challenges under manual control. Examination of large tissue areas is particularly challenging due to micron-scale resolution of the probe and the need for maintaining consistent probe orientation and force contact with the tissue to avoid cellular deformation or damage. The use of a robotic manipulator to perform surface scanning automatically introduces great benefits in terms of positioning repeatability and accuracy. However, pre-programming of such complex task is not realistic due to patient-specific anatomy and constant changes in tissue morphology during the operation. To overcome this problem, a cooperative, in situ microscopic scanning and simultaneous tissue surface reconstruction technique is proposed. The system provides a hands-on, learning-based framework for optimal trajectory coverage from surgeon-demonstrated motions intraoperatively. The position and force information acquired during the scanning are also used to simultaneously reconstruct the surface morphology and combined with the pCLE images to generate a 3D functional map of the tissue.

In this paper we present a new appearance-based localisation system that is able to deal with dynamic elements in the scene. By independently modelling the properties of local features observed in a scene over long periods of time, we show that feature appearances and geometric relationships can be learned more accurately than when representing a location by a single image. We also present a new dataset consisting of a 6 km outdoor path traversed once per month for a period of 5 months, which contains several challenges including short-term and long-term dynamic behaviour, lateral deviations in the path, repetitive scene appearances and strong illumination changes. We show superior performance of the dynamic mapping system compared to state-of-the-art techniques on our dataset.

In this paper we present a new method, Feature Co-occurrence Maps, for appearance-based localisation over the course of a day. We show that by quantising local features in both feature and image space, discriminative statistics can be learned on the co-occurrences of features at different times of the day. This allows for matching at any time, without requiring individual images to be stored representing each time of day, and matching is performed efficiently by simultaneously matching to the entire database. We further show how matching along image sequences can be incorporated into the system and adapt existing methods by allowing for non-zero acceleration. Results on a 20km outdoor dataset show improved performance in precision-recall over state of the art.

This paper presents an Implicit Active Constraints control framework for robot-assisted minimally invasive surgery. It extends on current frameworks by prescribing the external constraints implicitly from the operator motion, forgoing the need for pre-operative imaging; the constraints are defined in situ so as to avoid the use of invasive fiducial markers. A hands-on cooperatively-controlled robotic platform, comprising of a surgical instrument and a compliant manipulator, has been designed for an arthroscopic procedure. The surgical platform is capable of constraining the pose of the instrument so as to ensure it passes through the incision point and does not cause trauma to the surrounding tissue. A flexible arthroscopic instrument is designed and its use is investigated to enlarge reachable and dexterous workspace, increasing the accessibility to the target anatomy. The behaviour of the flexible instrument is analysed. A detailed performance analysis is conducted on a group of subjects for validating the control framework, simulating a minimally invasive arthroscopic procedure. Results demonstrate a statistically significant enhancement in the control ergonomics as well as the accuracy and safety of the procedure.

For mobile robot navigation in crowded environments, hand and body tracking to enable seamless human-robot interaction is a challenging problem. Many existing methods simplify the task with static camera assumptions, initial calibration stages, or ad hoc pose constraints, making them difficult to be applied to assistive robots used for healthcare applications. This paper introduces a method of hand-body association suitable for crowded environments, by incorporating depth cameras. A robust human hand and body detector, optimized for crowded environments, is first introduced. This is followed by a probabilistic framework for associating hands and bodies. Geodesic distances, based on depth information, are employed to isolate points local to a hand, regardless of their Euclidean proximity to points in other regions. This facilitates subsequent hand-body association based on a Bayesian framework with increased association robustness. The accuracy of the proposed method is evaluated using a range of parameters against an existing approach. A public dataset has been created to assess the method's practical value in crowded environments.

This paper introduces a gaze contingent controlled robotic arm for laparoscopic surgery, based on gaze gestures. The method offers a natural and seamless communication channel between the surgeon and the robotic laparoscope. It offers several advantages in terms of reducing on-screen clutter and efficiently conveying visual intention. The proposed hands-free system enables the surgeon to be part of the robot control feedback loop, allowing user-friendly camera panning and zooming. The proposed platform avoids the limitations of using dwell-time camera control in previous gaze contingent camera control methods. The system represents a true hands-free setup without the need of obtrusive sensors mounted on the surgeon or the use of a foot pedal. Hidden Markov Models (HMMs) were used for real-time gaze gesture recognition. This method was evaluated with a cohort of 11 subjects by using the proposed system to complete a modified upper gastrointestinal staging laparoscopy and biopsy task on a phantom box trainer, with results demonstrating the potential clinical value of the proposed system.

We propose a learning-based controller to enable autonomous execution of the eFAST scanning by a lightweight robotic manipulator according to expert demonstrations. The benefits of this approach are two-fold. Firstly, the automatically acquired USS images can be sent to the expert radiologist from a remote location without the need for complex robotic tele-operation. Secondly, the application of learning by demonstration alleviates the complexity of robotic programming and allows extracting operator-specific knowledge in situ in a natural and intuitive way. The provision of incorporating force information can further improve the versatility of the system, allowing easy adaptation to different dynamic environments.

This paper presents an ungrounded, hand-held surgical device that incorporates active constraints and force-feedback. Optical tracking of the device and embedded actuation allow for real-time motion compensation of a surgical tool as an active constraint is encountered. The active constraints can be made soft, so that the surgical tool tip motion is scaled, or rigid, so as to altogether prevent the penetration of the active constraint. Force-feedback is also provided to the operator so as to indicate penetration of the active constraint boundary by the surgical tool. The device has been evaluated in detailed bench tests to quantify its motion scaling and force-feedback capabilities. The combined effects of force-feedback and motion compensation are demonstrated during palpation of an active constraint with rigid and soft boundaries. A user study evaluated the combined effect of motion compensation and force-feedback in preventing penetration of a rigid active constraint. The results have shown the potential of the device operating in an ungrounded setup that incorporates active constraints with force-feedback.

Real-time shape sensing and state acquisition is important for closed-loop control of hyper-redundant snake robots in minimally invasive surgery. Due to the miniaturized size of such minimally invasive surgery robots, it is not feasible to use existing angular sensors involving rotary encoders. With recent advances of the MEMS technology, micro inertial sensors have shown their potential for robot state estimation. Previous studies have demonstrated that accurate joint angles can be estimated for one degree-of-freedom (DoF) joints. However, higher DoF joints of the robot can impose a number of challenges to the current joint angle estimation methods. This paper presents a micro-sensing platform and shape reconstruction algorithm for minimally invasive surgery snake robot with two DoF joints. The method incorporates both gravitational and gyroscopic sensing for calculating the rotation difference between any consecutive robot segments. The gyroscope measurements are first used as the input to predict the rotation difference by direct orientation integration. The orientation difference is then derived from the consecutive acceleration vectors to update the prediction through a complementary filter. To demonstrate the performance of our proposed approach, a robot prototype with two universal joints was fabricated. Detailed experimental results have demonstrated that high accuracy can be achieved by using the proposed method for joint angle estimation.

Despite a wide range of feature detectors developed in the computer vision community over the years, direct application of these techniques to surgical navigation has shown significant difficulties due to the paucity of reliable salient features coupled with free--form tissue deformation and changing visual appearance of surgical scenes. The aim of this paper is to propose a novel probabilistic framework to track affine-invariant anisotropic regions under contrastingly different visual appearances during Minimally Invasive Surgery (MIS). The theoretical background of the affine-invariant anisotropic feature detector is presented and a real-time implementation exploiting the computational power of the GPU is proposed. An Extended Kalman Filter (EKF) parameterization scheme is used to adaptively adjust the optimal templates of the detected regions, enabling accurate identification and matching of the tracked features. For effective tracking verification, spatial context and region similarity have also been incorporated. They are used to boost the prediction of the EKF and recover potential tracking failure due to drift or false positives. The proposed framework is compared to the existing methods and their respective performance is evaluated with in vivo video sequences recorded from robotic-assisted MIS procedures, as well as real-world scenes.

This paper presents an online reinforcement learning framework for medical image segmentation. The concept of context-specific segmentation is introduced such that the model is adaptive not only to a defined objective function but also to the user's intention and prior knowledge. Based on this concept, a general segmentation framework using reinforcement learning is proposed, which can assimilate specific user intention and behavior seamlessly in the background. The method is able to establish an implicit model for a large state-action space and generalizable to different image contents or segmentation requirements based on learning in situ. In order to demonstrate the practical value of the method, example applications of the technique to four different segmentation problems are presented. Detailed validation results have shown that the proposed framework is able to significantly reduce user interaction, while maintaining both segmentation accuracy and consistency.

This paper presents a real-time control framework for a snake robot with hyper-kinematic redundancy under dynamic active constraints for minimally invasive surgery. A proximity query (PQ) formulation is proposed to compute the deviation of the robot motion from predefined anatomical constraints. The proposed method is generic and can be applied to any snake robot represented as a set of control vertices. The proposed PQ formulation is implemented on a graphic processing unit, allowing for fast updates over 1 kHz. We also demonstrate that the robot joint space can be characterized into lower dimensional space for smooth articulation. A novel motion parameterization scheme in polar coordinates is proposed to describe the transition of motion, thus allowing for direct manual control of the robot using standard interface devices with limited degrees of freedom. Under the proposed framework, the correct alignment between the visual and motor axes is ensured, and haptic guidance is provided to prevent excessive force applied to the tissue by the robot body. A resistance force is further incorporated to enhance smooth pursuit movement matched to the dynamic response and actuation limit of the robot. To demonstrate the practical value of the proposed platform with enhanced ergonomic control, detailed quantitative performance evaluation was conducted on a group of subjects performing simulated intraluminal and intracavity endoscopic tasks.

This paper proposes the use of singular spectrum analysis (SSA) to segment and classify human activities in real time by using an ear-worn Activity Recognition (e-AR) sensor. A similarity measure is calculated using SSA to construct a 3D feature vector from the 3 axes of e-AR signal. An algorithm based on the concept of clustering and buffering is then implemented in order to detect activity transition in real time as subjects perform their daily activities. An incremental subspace learning algorithm based on SSA is also proposed for activity classification. The proposed algorithm is applied to a group of five subjects performing daily activities and the results have shown the effectiveness of the method for transition detection and activity classification.

The prevalence of obesity worldwide presents a great challenge to existing healthcare systems. There is a general need for pervasive monitoring of the dietary behaviour of those who are at risk of co-morbidities. Currently, however, there is no accurate method of assessing the nutritional intake of people in their home environment. Traditional methods require subjects to manually respond to questionnaires for analysis, which is subjective, prone to errors, and difficult to ensure consistency and compliance. In this paper, we present a wearable sensor platform that autonomously provides detailed information regarding a subject's dietary habits. The sensor consists of a microphone and a camera and is worn discretely on the ear. Sound features are extracted in real-time and if a chewing activity is classified, the camera captures a video sequence for further analysis. From this sequence, a number of key frames are extracted to represent important episodes during the course of a meal. Results show a high classification rate of chewing activities, and the visual log demonstrates a detailed overview of the subject's food intake that is difficult to quantify from manually-acquired food records.

This paper presents a mixed-signal photo detection architecture that provides DC offset rejection of up to x5 beyond the dynamic range of the front-end amplifier while retaining the DC signal content of the physiological signal being detected. Closed-loop control of the mean input current is used to prevent saturation of the detector's front-end amplifier while frequency modulation of the illumination source enables homodyne detection of the absorption properties of the blood vessels being investigated. As modulation creates a copy of the desired signal at high frequency, the bandwidth of the current feedback loop is allowed to overlap with low frequency physiological signals (e.g. respiration rate) without rejecting them from the homodyne output. Use of lattice wave digital filters enables a photo plethysmography system to be implemented with up to 1,000 samples per second in real-time by a low-power microcontroller. Experimental validation of the dual-mode noise rejection technique shows that it is robust against high static ambient light levels as well as rapid transitions in light levels.

Changes in gait can be caused by a wide range of health complications. As deviations in gait may be an indicator of deteriorating health, abnormalities can be used as a surrogate measure for detecting the onset of certain symptoms. Previous studies have demonstrated the value of wearable sensing for gait analysis. This paper demonstrates the added value of using a depth vision sensor combined with wearable sensors for gait analysis. It also presents a method for extracting a robust set of depth features. The preliminary results from a simulated homecare environment using a three-layer artificial neural network classifier demonstrate the advantages of using a depth sensor for gait analysis.

Detailed motion reconstruction is a prerequisite of biomotion analysis and physical function assessment for a variety of scenarios. For example, biomechanical analysis can be used to assess physical activity to diagnose pathological conditions, to provide an objective measure of biomechanics for peri-operative care, and to monitor patients with mobility issues. Unfortunately, current motion capture systems cannot perform biomechanical analysis continuously in the patient's natural environment. In this paper, a pose estimation scheme from a sparse network of accelerometer-based wearable sensors, which does not impose restrictions upon the patient's daily life, is presented. In the proposed method, a marker-based motion capture system is used for acquiring the 3D motion data, and partial least squares regression (PLSR) is used to establish the implicit model between 3D body pose and the wearable sensor measurements. A linear constant velocity process model and measurement model are designed and a Kalman filter is then deployed to estimate the posture. Experimental results demonstrate the strength of the technique and how it can be used to estimate detailed 3D motion from a sparse set of sensors.

Despite recent advances in modeling the Shape-from-Shading (SFS) problem and its numerical solution, practical applications have been limited. This is primarily due to the lack of perspective SFS models without the assumption of a light source at the camera centre and the non-metric spatial localisation of the reconstructed shape. In this work, we propose a novel formulation of the SFS problem that allows the reconstruction of surfaces lit by a near point light source away from the camera centre. We also show how knowledge of the light source position can enable the recovery of depth information in a metric space by triangulating specular highlights. Validation of the proposed technique is reported on synthetic and endoscopic images.

This paper presents a new hand-held device capable of amplifying delicate micromanipulation forces during minimal invasive surgical tasks. It relays force sensing to the user through a simple sliding feature that is coupled to the surgical tool, which translates relative to the casing of the device held by the operator. This forgoes the need of grounding frames or anchoring mechanisms to the body, allowing the device to be used in general surgical environments without affecting the workflow. The device uses a three-phase linear motor that is compact and capable of generating high forces that allow amplification factors of up to ×15. It features a closed-loop force control scheme to perform the required force amplification in which the force exerted on to the user is measured, forming the feedback in the control loop. The device permits interchangeability of instrumentation through a simple docking feature, and thus can be generalized to a range of surgical instruments for micromanipulation tasks. Detailed bench tests and user trials have been performed to validate the accuracy and practical performance of the device. The results have shown a five times reduction of the minimum force threshold perceived by the subjects and ergonomically sound manipulation advantages.

Minimally invasive endovascular procedures including robotically assisted intervention require effective intraoperative guidance. This is mainly achieved through intraoperative imaging such as fluoroscopy. Concerns over excessive x-ray radiation and nephrotoxicity due to repeated injection of contrast agents have motivated the development of effective catheter navigation schemes based on limited imaging data. This paper presents a catheter navigation technique based on probabilistic fusion of in situ real-time electromagnetic tracking with physically-based simulation of the mechanical characteristics of the catheter. A catheter with multiple electromagnetic sensors placed along its length has been developed. The sensor data and the catheter insertion-length are used as the boundary condition for determining the shape and position of the catheter within the vasculature. A probabilistic framework based on a Kalman Filter is used to combine the information from the catheter motion algorithm and the electromagnetic tracking data. This provides continuous visualization of the catheter within the lumen without the need of continuous fluoroscopy and contrast injection. The proposed approach has been validated with detailed in vitro experiments demonstrating the potential clinical application of the technique.

Accurate recovery of the 3D structure of a deforming surgical environment during minimally invasive surgery is important for intra-operative guidance. One key component of reliable reconstruction is accurate camera pose estimation, which is challenging for monocular cameras due to the paucity of reliable salient features, coupled with narrow baseline during surgical navigation. With recent advances in miniaturized MEMS sensors, the combination of inertial and vision sensing can provide increased robustness for camera pose estimation particularly for scenes involving tissue deformation. The aim of this work is to propose a robust framework for intra-operative free-form deformation recovery based on structure-from-motion. A novel adaptive Unscented Kalman Filter (UKF) parameterization scheme is proposed to fuse vision information with data from an Inertial Measurement Unit (IMU). The method is built on a compact scene representation scheme suitable for both surgical episode identification and instrument-tissue motion modelling. Detailed validation with both synthetic and phantom data is performed and results derived justify the potential clinical value of the technique.

Probe-based confocal laser endomicroscopy (pCLE) provides high resolution imaging of tissue in vivo. Maintaining a steady contact between target tissue and pCLE probe tip is important for image consistency. In this paper, a new prototype hand-held instrument for in vivo pCLE during Minimally Invasive Surgery (MIS) is presented. The proposed instrument incorporates adaptive force sensing and actuation, allowing improved image consistency and force control, thus minimizing tissue deformation and induced micro-structural variations. The performance and accuracy of the contact force control are evaluated in detailed laboratory settings and in vivo validation of the device during transanal microsurgery in a live porcine model further demonstrates the potential clinical value of the device.

The increasing number of degrees-of-freedom involved in new generations of surgical robotics and the need for incorporating active constraints and haptic feedback, require more intuitive and effective ways of robot control. This paper presents a novel manipulator that allows for ergonomic bimanual joint-level control of an anthropomorphic surgical robot. Through the combined use of bidirectional compressed airflow, the manipulator can operate on nearly zero friction and simulate a range of frictional forces. As a generic platform, the system can withstand large payloads and is able to accommodate a wide range of existing haptic manipulators. The performance of the proposed platform is evaluated with detailed experimental tests and proven to provide negligible friction even at high loads. Its dynamic friction is controllable and positional locking can be flexibly applied. Detailed experimental results demonstrate the practical value of the system.

Robotically assisted catheterisation has attracted significant interest in recent years. However, few designs have made full use of the extensive experience and skills that interventional radiologists have acquired from conventional catheter navigation. Additionally, limited research has been conducted in quantifying the effectiveness of haptic feedback for catheterisation procedures. This paper presents a novel master-slave force feedback system for endovascular catheterisation that can be used in a natural setting with enhanced ergonomics. The system has been validated with detailed laboratory experiments and a comprehensive user study. The performance of the system was compared against manual catheterisation by measuring the forces exerted on a vascular phantom. The results showed a 76% reduction in the mean force applied and a 55% reduction in the maximum force over the course of the study, indicating that the force feedback system reduced the magnitude and duration of force exerted during a simulated endovascular procedure. This research provides important insights into the design of compact and ergonomic robotic catheter manipulators incorporating effective real-time force feedback for intraoperative navigation.

This paper describes a multitasking robotic platform for Minimally Invasive Surgery (MIS). The device is designed to be introduced through a standard trocar port. Once the device is inserted to the desired surgical site, it can be reconfigured by lifting an articulated section, and protruding two tendon driven flexible arms. Each of the arms holds an interchangeable surgical instrument. The articulated section features a 2 Degrees-of-Freedom (DoF) universal joint followed by a single DoF yaw joint. It incorporates an on-board camera and LED light source at the distal end, leaving a Ø3mm channel for an additional instrument. The main shaft of the robot is largely hollow, leaving ample space for the insertion of two tendon driven flexible arms integrated with surgical instruments. The ex-vivo and in-vivo experiments demonstrate the potential clinical value of the device for performing surgical tasks through single incision or natural orifice transluminal procedures.

Activity monitoring is important for assessing daily living conditions for elderly patients and those with chronic diseases. Transitions between activities can present characteristic patterns that may be indicative of quality of movement. To detect and analyze transitional activities, a manifold-based approach is proposed in this paper. The proposed method uses a recursive spectral graph-partitioning algorithm to segment transitions in activity. These segments are subsequently mapped to a reference manifold space. Categorization of transitions is performed with the corresponding features in the manifold space. The practical value of the work is demonstrated through data collected under laboratory conditions, as well as patients recovering from total knee replacement operations, demonstrating specific transitions and motion impairment compared to normal subjects.

Gastro-intestinal (GI) endoscopy is a widely used clinical procedure for screening and surveillance of digestive tract diseases ranging from Barrett's Oesophagus to oesophageal cancer. Current surveillance protocol consists of periodic endoscopic examinations performed in 3-4 month intervals including expert's visual assessment and biopsies taken from suspicious tissue regions. Recent development of a new imaging technology, called probe-based confocal laser endomicroscopy (pCLE), enabled the acquisition of in vivo optical biopsies without removing any tissue sample. Besides its several advantages, i.e., noninvasiveness, real-time and in vivo feedback, optical biopsies involve a new challenge for the endoscopic expert. Due to their noninvasive nature, optical biopsies do not leave any scar on the tissue and therefore recognition of the previous optical biopsy sites in surveillance endoscopy becomes very challenging. In this work, we introduce a clustering and classification framework to facilitate retargeting previous optical biopsy sites in surveillance upper GI-endoscopies. A new representation of endoscopic videos based on manifold learning, "endoscopic video manifolds" (EVMs), is proposed. The low dimensional EVM representation is adapted to facilitate two different clustering tasks; i.e., clustering of informative frames and patient specific endoscopic segments, only by changing the similarity measure. Each step of the proposed framework is validated on three in vivo patient datasets containing 1834, 3445, and 1546 frames, corresponding to endoscopic videos of 73.36, 137.80, and 61.84 s, respectively. Improvements achieved by the introduced EVM representation are demonstrated by quantitative analysis in comparison to the original image representation and principal component analysis. Final experiments evaluating the complete framework demonstrate the feasibility of the proposed method as a promising step for assisting the endoscopic expert in retargeting the optical biopsy sites.

The 11 papers in this special issue represent different advances in interventional imaging.

Abstract Wireless sensor networks enable continuous and reliable data acquisition for real‐time monitoring in a variety of application areas. Due to the large amount of data collected and the potential complexity of emergent patterns, scalable and distributed reasoning is preferable when compared to centralised inference as this allows network wide decisions to be reached robustly without specific reliance on particular network components. In this paper, we provide an overview of distributed inference for both wearable and ambient sensing with specific focus on graphical models—illustrating their ability to be mapped to the topology of a physical network. Examples of research conducted by the authors in the use of ambient and wearable sensors are provided, demonstrating the possibility for distributed, real‐time activity monitoring within a home healthcare environment. Copyright © 2010 John Wiley & Sons, Ltd.

This paper presents a new framework for visual place recognition that incrementally learns models of each place and offers adaptability to dynamic elements in a scene. Traditional bag-of-features image-retrieval approaches to place recognition treat images in a holistic manner and are typically not capable of dealing with sub-scene dynamics, such as structural changes to a building facade or the rearrangement of furniture in a room. However, by treating local features as observations of real-world landmarks in a scene that are consistently observed, such dynamics can be accurately modelled at a local level, and the spatio-temporal properties of each landmark can be independently updated online. We propose a framework for place recognition that models each scene by sequentially learning landmarks from a set of images, and in the long term adapts the model to dynamic behaviour. Results on both indoor and outdoor datasets show an improvement in recognition performance and efficiency when compared to the traditional bag-offeatures image retrieval approach.

A progressive improvement in gait following knee arthroplasty surgery can be observed during walking and transitional activities such as sitting/standing. Accurate assessment of such changes traditionally requires the use of a gait lab, which is often impractical, expensive, and labour intensive. Quantifying gait impairment following knee arthroplasty by employing wearable sensors allows for continuous monitoring of recovery. This study employed a recognised protocol of activities both pre-operatively, and at regular intervals up to twenty-four weeks post-total knee arthroplasty. The results suggest that a wearable miniaturised ear-worn sensor is potentially useful in monitoring post-operative recovery, and in identifying patients who fail to improve as expected, thus facilitating early clinical review and intervention.

Human back movement estimation is clinically important for assessing patients with back pain. Most current techniques are limited to simple spinal movement angles without consideration of surrounding muscle movement and backplane rotation and torsion. These three dimensional analysis is fraught with difficulties due to the complex nature of the movement and sensor placement. In this paper, a consistent method based on multiple Body Sensor Network (BSN) nodes for the measurement of 3D bending and twist of the back is proposed. In our method, five BSN nodes, each consisting of a three axis accelerometer, a gyroscope and a magnetometer, are placed at the human back. Euler angles are then defined to represent the orientation for human back segments, kinematics analysis is then derived. An unscented Kalman filter (UKF) is deployed to estimate the defined Euler angles. Detailed experimental results have shown the feasibility and effectiveness of the proposed measurement and analysis framework.

The recognition of a place depicted in an image typically adopts methods from image retrieval in large-scale databases. First, a query image is described as a “bag-of-features” and compared to every image in the database. Second, the most similar images are passed to a geometric verification stage. However, this is an inefficient approach when considering that some database images may be almost identical, and many image features may not repeatedly occur. We address this issue by clustering similar database images to represent distinct scenes, and tracking local features that are consistently detected to form a set of real-world landmarks. Query images are then matched to landmarks rather than features, and a probabilistic model of landmark properties is learned from the cluster to appropriately verify or reject putative feature matches. We present novelties in both a bag-of-features retrieval and geometric verification stage based on this concept. Results on a database of 200K images of popular tourist destinations show improvements in both recognition performance and efficiency compared to traditional image retrieval methods.

Vision-based topological maps for mobile robot localization traditionally consist of a set of images captured along a path, with a query image then compared to every individual map image. This paper introduces a new approach to topological mapping, whereby the map consists of a set of landmarks that are detected across multiple images, spanning the continuous space between nodal images. Matches are then made to landmarks, rather than to individual images, enabling a topological map of far greater density than traditionally possible, without sacrificing computational speed. Furthermore, by treating each landmark independently, a probabilistic approach to localization can be employed by taking into account the learned discriminative properties of each landmark. An optimization stage is then used to adjust the map according to speed and localization accuracy requirements. Results for global localization show a greater positive location identification rate compared to the traditional topological map, together with enabling a greater localization resolution in the denser topological map, without requiring a decrease in frame rate.

This paper introduces an articulated robotic device based on universal joints with embedded micro motors for minimally invasive surgery. The device features an articulated distal tip with seven independently controllable degrees-of-freedom (DoF), arranged as two universal joints (intersecting pitch and yaw) and three single DoF joints (yaw only); two Ø3mm internal channels, one for an on-board camera for visualization and the other for passing interventional instruments. The design allows the robot to explore the entire peritoneal cavity from a chosen single incision point. A trans-vaginal procedure using the device to locate the uterine horn, as a model of a human fallopian tube, and apply an endoscopic clip was carried out during a live porcine trial to demonstrate the potential for performing a Natural Orifice Translumenal Endoscopic Surgery (NOTES) tubal ligation procedure.

In this paper, an auditory attention driven computational model of the auditory midbrain is proposed based on a spiking neural network [17] in order to localize attended sound sources in reverberant environments. Both bottom-up attention driven by sensors and top-down attention driven by the cortex are modeled at the level of an auditory midbrain nucleus - the inferior colliculus (IC). Improvements of the model in [17] is made to increase biological plausibility. First, inter-neuron inhibitions are modeled among the IC neurons which have the same characteristic frequency but different spatial response. This is designed to mimic the precedence effect [15] to produce localization results in reverberate environments. Secondly, descending projections from the auditory cortex (AC) to the IC are model to simulate the top-down attention so that focused sound sources can be better sensed in noise or multiple sound source situations. Our model is implemented on a mobile robot with a manikin head equipped with binaural microphones and tested in a real environment. The results shows that our attention driven model can give more accurate localization results than prior models.

In this paper, we present a new technique for social mobile robot speech recognition based on scene-associated training models. The key contribution of the paper is a real-time framework that reduces the effect of room reverberation and ambient noise, a challenging problem in speech recognition. In classical approaches, anechoic sound is used to train the model, with the main focus on removing reverberation or noise from the sound. Our technique differs in that we train a number of speech recognizers directly from the reverberated sound, by associating each recognizer with a unique visual scene, to deal with the varying reverberation properties of different rooms. By extracting local features from a captured image and recognizing a scene, the robot can use the appropriate speech recognizer that is trained for the particular structural properties of that scene. We tested our method by using a baseline speech recognition model (HTK) across a variety of rooms and different levels of background noise. The results show that the association between a visual scene and a corresponding speech recognizer greatly improves the robot's speech recognition accuracy, together with increasing the computational speed of recognition, compared to competing techniques.

This paper introduces a modular, mechatronic joint design for a flexible access minimally invasive surgical platform. The design features a hybrid tendon-micromotor actuation scheme coupled via an internal “ring gear”. This configuration transmits the rotary motion of the embedded micromotor to joint rotation in an efficient manner while also providing a hollow lumen through the centre of the joint unit. Detailed analyses of the joint design including tendon path length, positional accuracy and force/torque transmission characteristics are presented. The design has been incorporated into a dexterous, seven degree-of-freedom flexible access platform and its clinical efficacy for performing a Natural Orifice Translumenal Endoscopic Surgery (NOTES) diagnostic peritoneoscopy in an in-vivo environment is presented.

Patients' functional recovery at home following surgery may be evaluated by monitoring their activities of daily living. Existing tools for assessing these activities are labor-intensive to administer and rely heavily on recall. This study describes the use of a wireless ear-worn activity recognition sensor to monitor postoperative activity levels continuously using a Bayesian activity classification framework. The device was used to monitor the postoperative recovery of five patients following abdominal surgery. Activity was classified into four groups ranging from very low (level 0) to high (level 3). Overall, patients were found to be undertaking a higher proportion of level 0 activities on postoperative day 1 which was gradually replaced by higher-level activities over the next 3 days. This study demonstrates how a pervasive healthcare technology can objectively monitor functional recovery in the unsupervised home setting. This may be a useful adjunct to existing postoperative monitoring systems.

This paper presents a novel approach to shape extraction and interpretation in 4-D cardiac magnetic resonance imaging data. Statistical modeling of spatiotemporal interlandmark relationships is performed to enable the decomposition of global shape constraints and subsequently of the image analysis tasks. The introduced descriptors furthermore provide invariance to similarity transformations and thus eliminate pose estimation errors in the presence of image artifacts or geometrical inconsistencies. A set of algorithms are derived to address key technical issues related to constrained boundary tracking, dynamic model relaxation, automatic initialization, and dysfunction localization. The proposed framework is validated with a relatively large dataset of 50 subjects and compared to existing statistical shape modeling methods. The results indicate increased adaptation to spatiotemporal variations and imaging conditions.

Activities of daily living are important for assessing changes in physical and behavioural profiles of the general population over time, particularly for the elderly and patients with chronic diseases. Although accelerometers are widely integrated with wearable sensors for activity classification, the positioning of the sensors and the selection of relevant features for different activity groups still pose interesting research challenges. This paper investigates wearable sensor placement at different body positions and aims to provide a framework that can answer the following questions: (i) What is the ideal sensor location for a given group of activities? (ii) Of the different time-frequency features that can be extracted from wearable accelerometers, which ones are most relevant for discriminating different activity types?

Due to the natural aging process, the risks associated with falling can increase significantly. For the elderly, this usually marks a rapid deterioration of their health. While there are identified strategies that can be adopted to reduce the number of falls, it is still not possible to prevent all falls. Clinically, the Tinetti Gait and Balance Assessment has been widely used to assess the risk of falls in elderly by examining balance and gait. This paper presents our initial results of using an ear-worn BSN sensor to detect aspects of the Tinetti Gait and Balance Assessment to predict the risk of falls compared to a healthy control cohort. For this study, data was collected from a control cohort of 12 healthy volunteers and a cohort of 16 elderly fallers of varying degrees of risk. The results derived have shown that it is possible to directly detect some aspects of the Tinetti Gait and Balance Assessment and the Timed Up and Go test, demonstrating the potential value of using the platform for continuous assessment in a home environment.

Changing demographics in the western society require the provision of resources to enable independent living for the aging population. This includes guiding research and development into technologies and interventions that would assist the elderly to live safely and independently in their own homes. It is also essential that policy regarding privacy and security is addressed, both on a regional, national and international scale. This paper will provide an introduction to the European Commission’s CAPSIL project, aimed at studying the requirements for enabling independent living, with an emphasis on the role of Body Sensor Networks (BSN). The objective of the project will be described including a description of the CAPSIL Wiki, an online tool for the dissemination of the project findings, how BSNs can be used to support independent living of the elderly, key design aspect of BSN systems for the elderly and the current gaps and challenges that will need to be addressed.

With the development of miniaturized wireless wearable and implantable medical devices, pervasive monitoring is becoming a clinical reality. With an increasing drive for minimizing power utilization, optimal antenna design and radio wave propagation are important topics for BSN (body sensor networks) research. To this end, subject-specific modeling is essential for achieving a truly personalized and optimized sensor design. In this paper, a Volumetric Graph Laplacian method is used for subject-specific whole-body mesh warping for finite-difference time-domain (FDTD) simulations. Validation of the method is shown with data from both phantom and MR studies with FDTD simulations.

The recent maturity of body sensor networks has enabled a wide range of applications in sports, well-being and healthcare. In this paper, we hypothesise that a single unobtrusive head-worn inertial sensor can be used to infer certain biomotion details of specific swimming techniques. The sensor, weighing only seven grams is mounted on the swimmer's goggles, limiting the disturbance to a minimum. Features extracted from the recorded acceleration such as the pitch and roll angles allow to recognise the type of stroke, as well as basic biomotion indices. The system proposed represents a non-intrusive, practical deployment of wearable sensors for swimming performance monitoring.

This paper investigates the feasibility of a ratio metric approach to compensating for ambient light and motion artefacts in a reflective photoplethysmography (PPG) sensor suitable for wearable applications. A low-power, discrete-time pulse-oximeter development platform is used to capture both infra-red (IR) and red photoplethysmograms, as well as the ambient light level, such that the data used for analysis has noise levels representative of what a true body sensor network (BSN) device would experience. The performance of a ratiometric artefact reduction technique is tested with both simulated multiplicative noise sources, as well as real motion artefacts to best determine where the limitations of the system lie.

Despite the great diversity of teleoperator designs and applications, their underlying control systems have many similarities. These similarities can be exploited to enable inter-operability between heterogeneous systems. We have developed a network data specification, the Interoperable Telerobotics Protocol, that can be used for Internet based control of a wide range of teleoperators. In this work we test interoperable telerobotics on the global Internet, focusing on the telesurgery application domain. Fourteen globally dispersed telerobotic master and slave systems were connected in thirty trials in one twenty four hour period. Users performed common manipulation tasks to demonstrate effective master-slave operation. With twenty eight (93%) successful, unique connections the results show a high potential for standardizing telerobotic operation. Furthermore, new paradigms for telesurgical operation and training are presented, including a networked surgery trainer and upper-limb exoskeleton control of micro-manipulators.

Accurate, efficient and robust location recognition is a fundamental task for any mobile robot. This paper presents a new approach using visual features to efficiently represent a series of locations along a path in an indoor environment. In the training stage, local features which are detected across multiple images from a single tour are combined to represent a real-world landmark, modelled by the expected variance of its descriptor. Those landmarks which represent the scene in the most efficient and discriminative manner are then retained, and this selection is optimized with respect to the scale of the environment. In the recognition stage, features detected in an image are matched to the landmarks in memory, based upon a novel similarity measure drawing from feature co-occurrence statistics.

Home monitoring plays an important role within pervasive healthcare, particularly for monitoring the elderly and patients with chronic disease. For assessing activities of daily living, one of the most challenging problems for research remains that of accurate transition detection and characterisation. Early detection of a change in these transitions, such as difficulty getting up from a seated position, can be an indicator of further complications which often precede a fall. Such changes can also accompany early stage neurological disorders which can be treated effectively to improve quality of life. In this paper, we present a system for the accurate characterisation of motion based upon the fusion of ambient and wearable sensors. A probabilistic, privacy respectful method for the extraction of detailed 3D posture information is proposed and fusion with an ear-worn accelerometer and gyroscope is discussed. We present results detailing high accuracy in the recognition of complex motions over four subjects.

This paper introduces a stereoscopic fibroscope imaging system for minimally invasive surgery (MIS) and examines the feasibility of utilizing images transmitted from the distal fibroscope tip to a proximally mounted CCD camera to recover both camera motion and 3D scene information. Fibre image guides facilitate instrument miniaturization and have the advantage of being more easily integrated with articulated robotic instruments. In this paper, twin 10,000 pixel coherent fibre bundles (590mum diameter) have been integrated into a bespoke laparoscopic imaging instrument. Images captured by the system have been used to build a 3D map of the environment and reconstruct the laparoscope's 3D pose and motion using a SLAM algorithm. Detailed phantom validation of the system demonstrates its practical value and potential for flexible MIS instrument integration due to the small footprint and flexible nature of the fibre image guides.

As medical robotics gathers increasing attention, the ergonomics of the surgical-console design becomes an important issue. Motivated by the need of augmenting the surgeon mastery, we explore the capabilities of a near infrared brain-computer interface as a complementary input modality to enhance the human-robot interaction at the robotic console. A multistage analysis framework is proposed and evaluated by an exploratory off-line synchronous study. The three stages of the data processing flow, namely dimensionality reduction, solution to binary problems and aggregation into multi-class decision are examined to address key challenges during the pattern recognition step. Early experimental results endorse near infrared based brain-computer interface as a suitable additional communication modality between the surgeon and the robotic console.

This paper presents a human-robot interface with perceptual docking to allow for the control of multiple microbots. The aim is to demonstrate that real-time eye tracking can be used for empowering robots with human vision by using knowledge acquired in situ. Several micro-robots can be directly controlled through a combination of manual and eye control. The novel control environment is demonstrated on a virtual biopsy of gastric lesion through an endoluminal approach. Twenty-one subjects were recruited to test the control environment. Statistical analysis was conducted on the completion time of the task using the keyboard control and the proposed eye tracking framework. System integration with the concept of perceptual docking framework demonstrated statistically significant improvement of task execution.

This paper presents a novel structure learning algorithm for the creation of distributed Bayesian networks over static and mobile Vision Sensor Network (VSN) nodes. These compose an assistive, intelligent environment for activity recognition. We provide results demonstrating a higher level of accuracy in the recognition of fine motor tasks when the environment is augmented with a mobile robot and show the ability of our learning algorithm to reduce VSN communication compared to a nai¿ve, greedy structure learning technique.

For robotic assisted minimally invasive surgery, recovering the 3D soft-tissue shape and morphology in vivo is important for providing image-guidance, motion compensation and applying dynamic active constraints. In this paper, we propose a practical method for calibrating the illumination source position in monocular and stereoscopic laparoscopes. The method relies on using the geometric constraints from specular reflections obtained during the laparoscope camera calibration process. By estimating the light source position, the method forgoes the common assumption of coincidence with the camera centre and can be used to obtain constraints on the normal of the surface geometry during surgery from specularities. We demonstrate the effectiveness of the proposed approach with numerical simulations and by qualitative analysis of real stereo-laparoscopic calibrations.

Determining human intention is a challenging task for establishing affective human robot interaction. The aim of this paper is to provide a vision based framework to achieve a level of understanding about people in an environment before engaging in active communication or interaction. The proposed method combines multiple cues in a Bayesian framework to identify people in the scene and determine potential intentions. To improve the system performance, contextual feedback is used, which allows the Bayesian network to evolve and adjust itself according to the surrounding environment. Our results demonstrate the effectiveness of the technique in dealing with human-robot interaction in a relatively crowded environment.

New approaches to chronic disease management within a home or community setting offer patients the prospect of more individually focused care and improved quality of life. This paper investigates the use of a light-weight ear worn activity recognition device combined with wireless ambient sensors for identifying common activities of daily living. A two-stage Bayesian classifier that uses information from both types of sensors is presented. Detailed experimental validation is provided for datasets collected in a laboratory setting as well as in a home environment. Issues concerning the effective use of the relatively limited discriminative power of the ambient sensors are discussed. The proposed framework bodes well for a multi-dwelling environment, and offers a pervasive sensing environment for both patients and care-takers.

The use of sensor networks for healthcare, well-being, and working in extreme environments has long roots in the engineering sector in medicine and biology community. With the maturity of wireless sensor networks, body area networks (BANs), and wireless BANs (WBANs), recent efforts in promoting the concept of body sensor networks (BSNs) aim to move beyond sensor connectivity to adopt a system-level approach to address issues related to biosensor design, interfacing, and embodiment, as well as ultralow-power processing/communication, power scavenging, autonomic sensing, data mining, inferencing, and integrated wireless sensor microsystems. As a result, the system architecture based on WBAN and BSN is becoming a widely accepted method of organization for ambulatory and ubiquitous monitoring systems. This editorial paper presents a snapshot of the current research and emerging applications and addresses some of the challenges and implementation issues.

Laparoscopic surgery is a challenging task in minimally invasive surgery, which involves complex instrument control, extensive manual dexterity, and hand-eye coordination. This requires a greater attention to training and skills evaluation. In order to provide a more objective skills assessment method, this paper presents a wireless sensor platform for the capture of laparoscopic hand gesture data and a hidden-Markov-model-based analysis framework for optimal sensor selection and placement. Detailed experimental validation is provided to illustrate how the proposed method can be used to assess surgical performance improvement over repeated training.

The levator ani is the main muscular support of the pelvic floor organs and damage caused by childbirth can affect its function. The full functionality of this muscle group is still unknown but it is essential for effective surgical planning. To elucidate its functional significance, a physical-based statistical shape model was built from the levator ani surfaces of 15 subjects scanned in an open access scanner. Simulation of dynamic exercises was performed on the resulting surfaces with finite element analysis. Statistical shape modeling was performed on the training set consisting of the original and simulated shapes along with thickness and strain distributions. Simulation results are presented on 15 subjects. The statistical shape model shows good correspondence to inter- and intra-subject shape variability, with the modes of variation highlighting movement in the posterior of the levator ani as well as in the levator arms. Strain distribution plots and the modes of variation show results that correspond to clinical findings. Further validation of the technique and a repeatability test were performed on four subjects with internal global pressure readings taken from a perineometer and five patients suffering from minor pelvic floor disorders due to obstructed defaecation. A Mann-Whitney nonparametric test was used to compare the normal model fitting to the two subject groups.

Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is an important tool for detecting subtle kinetic changes in cancerous tissue. Quantitative analysis of DCE-MRI typically involves the convolution of an arterial input function (AIF) with a nonlinear pharmacokinetic model of the contrast agent concentration. Parameters of the kinetic model are biologically meaningful, but the optimization of the nonlinear model has significant computational issues. In practice, convergence of the optimization algorithm is not guaranteed and the accuracy of the model fitting may be compromised. To overcome these problems, this paper proposes a semi-parametric penalized spline smoothing approach, where the AIF is convolved with a set of B-splines to produce a design matrix using locally adaptive smoothing parameters based on Bayesian penalized spline models (P-splines). It has been shown that kinetic parameter estimation can be obtained from the resulting deconvolved response function, which also includes the onset of contrast enhancement. Detailed validation of the method, both with simulated and in vivo data, is provided.

This paper introduces a novel technique for controlling an articulated robotic device through the eyes of the surgeon during minimally invasive surgery. The system consists of a binocular eye-tracking unit and a robotic instrument featuring a long, rigid shaft with an articulated distal tip for minimally invasive interventions. They have been integrated into a daVinci surgical robot to provide a seamless and non-invasive localization of eye fixations of the surgeon. By using a gaze contingent framework, the surgeonpsilas fixations in 3D are converted into commands that direct the robotic probe to the desired location. Experimental results illustrate the ability of the system to perform real-time gaze contingent robot control and opens up a new avenue for improving current human-robot interfaces.

MOTIVATION: The quest for high-throughput proteomics has revealed a number of challenges in recent years. Whilst substantial improvements in automated protein separation with liquid chromatography and mass spectrometry (LC/MS), aka 'shotgun' proteomics, have been achieved, large-scale open initiatives such as the Human Proteome Organization (HUPO) Brain Proteome Project have shown that maximal proteome coverage is only possible when LC/MS is complemented by 2D gel electrophoresis (2-DE) studies. Moreover, both separation methods require automated alignment and differential analysis to relieve the bioinformatics bottleneck and so make high-throughput protein biomarker discovery a reality. The purpose of this article is to describe a fully automatic image alignment framework for the integration of 2-DE into a high-throughput differential expression proteomics pipeline. RESULTS: The proposed method is based on robust automated image normalization (RAIN) to circumvent the drawbacks of traditional approaches. These use symbolic representation at the very early stages of the analysis, which introduces persistent errors due to inaccuracies in modelling and alignment. In RAIN, a third-order volume-invariant B-spline model is incorporated into a multi-resolution schema to correct for geometric and expression inhomogeneity at multiple scales. The normalized images can then be compared directly in the image domain for quantitative differential analysis. Through evaluation against an existing state-of-the-art method on real and synthetically warped 2D gels, the proposed analysis framework demonstrates substantial improvements in matching accuracy and differential sensitivity. High-throughput analysis is established through an accelerated GPGPU (general purpose computation on graphics cards) implementation. AVAILABILITY: Supplementary material, software and images used in the validation are available at http://www.proteomegrid.org/rain/.

The postgenomics era has witnessed a rapid change in biological methods for knowledge elucidation and pharmacological approaches to biomarker discovery. Differential expression of proteins in health and disease holds the key to early diagnosis and accelerated drug discovery. This approach, however, has also brought an explosion of data complexity not mirrored by existing progress in proteome informatics. It has become apparent that the task is greater than that can be tackled by individual laboratories alone and large-scale open collaborations of the new human proteome organization (HUPO) have highlighted major challenges concerning the integration and cross-validation of results across different laboratories. This paper describes the state-of-the-art proteomics workflows (two-dimensional gel electrophoresis, liquid chromatography, and mass spectrometry) and their utilization by the participants of the HUPO initiatives towards comprehensive mapping of the brain, liver, and plasma proteomes. Particular emphasis is given to the limitations of the underlying data analysis techniques for large-scale collaborative proteomics. Emerging paradigms including statistical data normalization, direct image registration, spectral libraries, and high-throughput computation with Web-based bioinformatics services are discussed. It is envisaged that these methods will provide the basis for breaking the bottleneck of large-scale automated proteome mapping and biomarker discovery.

Visual realism has been a major objective of computer graphics since the inception of the field. However, the perception of visual realism is not a well-understood process and is usually attributed to a combination of visual cues and image features that are difficult to define or measure. For highly complex images, the problem is even more involved. The purpose of this paper is to present a study based on eye tracking for investigating the perception of visual realism of static images with different visual qualities. The eye-fixation clusters helped to define salient image features corresponding to 3D surface details and light transfer properties that attract observers' attention. This enabled the definition and categorization of image attributes affecting the perception of photorealism. The dynamics of the visual behavior of different observer groups were examined by analyzing saccadic eye movements. We also demonstrated how the different image categories used in the experiments were perceived with varying degrees of visual realism. The results presented can be used as a basis for investigating the impact of individual image features on the perception of visual realism. This study suggests that post-recall or simple abstraction of visual experience is not accurate and the use of eye tracking provides an effective way of determining relevant features that affect visual realism, thus allowing for improved rendering techniques that target these features.

This paper describes a decision support system for determining salient features for CT lung nodule detection using an eye-tracking based machine learning technique. The method first analyses the scan paths of expert radiologists during normal examination. The underlying features are then used to highlight salient regions that may be of diagnostic relevance by merging visual features learned from different experts with a weighted probability function. The framework has been evaluated using data from CT lung nodule examination and the results demonstrate the potential clinical value of the proposed technique, which can also be generalized to other diagnostic applications.

This paper presents a new outlier handling method for volumetric segmentation with three-dimensional (3-D) active shape models. The method is based on a shape metric that is invariant to scaling, rotation and translation by using the ratio of interlandmark distances as a local shape dissimilarity measure. Tolerance intervals for the descriptors are calculated from the training samples and used as a statistical tolerance model to infer the validity of the feature points. A replacement point is then suggested for each outlier based on the tolerance model and the position of the valid points. A geometrically weighted fitness measure is introduced for feature point detection, which limits the presence of outliers and improves the convergence of the proposed segmentation framework. The algorithm is immune to the extremity of the outliers and can handle a highly significant presence of erroneous feature points. The practical value of the technique is validated with 3-D magnetic resonance (MR) segmentation tasks of the carotid artery and myocardial borders of the left ventricle.

This paper presents an image-based method for virtual bronchoscope with photo-realistic rendering. The technique is based on recovering bidirectional reflectance distribution function (BRDF) parameters in an environment where the choice of viewing positions, directions, and illumination conditions are restricted. Video images of bronchoscopy examinations are combined with patient-specific three-dimensional (3-D) computed tomography data through two-dimensional (2-D)/3-D registration and shading model parameters are then recovered by exploiting the restricted lighting configurations imposed by the bronchoscope. With the proposed technique, the recovered BRDF is used to predict the expected shading intensity, allowing a texture map independent of lighting conditions to be extracted from each video frame. To correct for disocclusion artefacts, statistical texture synthesis was used to recreate the missing areas. New views not present in the original bronchoscopy video are rendered by evaluating the BRDF with different viewing and illumination parameters. This allows free navigation of the acquired 3-D model with enhanced photo-realism. To assess the practical value of the proposed technique, a detailed visual scoring that involves both real and rendered bronchoscope images is conducted.

This paper presents a nonrigid registration two-dimensional/three-dimensional (2-D/3-D) framework and its phantom validation for subject-specific bronchoscope simulation. The method exploits the recent development of five degrees-of-freedom miniaturized catheter tip electromagnetic trackers such that the position and orientation of the bronchoscope can be accurately determined. This allows the effective recovery of unknown camera rotation and airway deformation, which is modelled by an active shape model (ASM). ASM captures the intrinsic variability of the tracheo-bronchial tree during breathing and it is specific to the class of motion it represents. The method reduces the number of parameters that control the deformation, and thus greatly simplifies the optimisation procedure. Subsequently, pq-based registration is performed to recover both the camera pose and parameters of the ASM. Detailed assessment of the algorithm is performed on a deformable airway phantom, with the ground truth data being provided by an additional six degrees-of-freedom electromagnetic (EM) tracker to monitor the level of simulated respiratory motion.

Eye movements provide important insight into the cognitive processes underlying the visual search tasks. For image understanding, although the visual search patterns of different observers while studying the same scene bear some common characteristics, the idiosyncrasy associated with individual observers provides both research opportunities and challenges. The aim of this paper is to study the spatial characteristics of visual search, together with the intrinsic visual features of the fixation points for comparing different visual search strategies. An analysis framework based on earth mover's distance (EMD) in normalized anatomical space is proposed, and the results are demonstrated with high resolution computed tomography (HRCT) images of the lungs. The study shows that through the effective use of both spatial and feature space representation, it is possible to untangle what appear to be uncorrelated fixation distribution patterns to reveal common visual search behaviors.

This paper proposes a new method for estimating kinetic parameters of dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) based on adaptive Gaussian Markov random fields. Kinetic parameter estimates using neighboring voxels reduce the observed variability in local tumor regions while preserving sharp transitions between heterogeneous tissue boundaries. Asymptotic results for standard errors from likelihood-based nonlinear regression are compared with those derived from the posterior distribution using Bayesian estimation with and without neighborhood information. Application of the method to the analysis of breast tumors based on kinetic parameters has shown that the use of Bayesian analysis combined with adaptive Gaussian Markov random fields provides improved convergence behavior and more consistent morphological and functional statistics.

In this paper, we propose a practical method for removing specular artifacts on the epicardial surface of the heart in robotic laparoscopic surgery while preserving the underlying image structure. We use freeform temporal registration of the non-rigid surface motion to recover chromatic information saturated by highlights. The diffuse and specular image components are then separated by shifting pixel intensities with respect to chromaticity gathered from the spatio-temporal volume. Results on in vivo data and reconstructions of 3D structure from the diffuse images show the potential value of the technique.

Understanding of the dynamic behaviour of the levator ani is important to the assessment of pelvic floor dysfunction. Whilst shape modelling allows the depiction of 3D morphological variation of the levator ani between different patient groups, it is insufficient to determine the underlying behaviour of how the muscle deforms during contraction and strain. The purpose of this study is to perform a subject specific finite element analysis of the levator ani with open access magnetic resonance imaging. The method is based on a Mooney-Rivlin hyperelastic model and permits dynamic study of subjects under natural physiological loadings. The value of the proposed modelling framework is demonstrated with dynamic 3D data from nulliparous, female subjects. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Photo-realistic rendering combined with vision techniques is an important trend in developing next generation surgical simulation devices. Training with simulator is generally low in cost and more efficient than traditional methods that involve supervised learning on actual patients. Incorporating genuine patient data in the simulation can significantly improve the efficacy of training and skills assessment. In this paper, a photo-realistic simulation architecture is described that utilises patient-specific models for training in minimally invasive surgery. The datasets are constructed by combining computer tomographic images with bronchoscopy video of the same patient so that the three dimensional structures and visual appearance are accurately matched. Using simulators enriched by a library of datasets with sufficient patient variability, trainees can experience a wide range of realistic scenarios, including rare pathologies, with correct visual information. In this paper, the matching of CT and video data is accomplished by using a newly developed 2D/3D registration method that exploits a shape from shading similarity measure. Additionally, a method has been devised to allow shading parameter estimation by modelling the bidirectional reflectance distribution function (BRDF) of the visible surfaces. The derived BRDF is then used to predict the expected shading intensity such that a texture map independent of lighting conditions can be extracted. Thus new views can be generated that were not captured in the original bronchoscopy video, thus allowing free navigation of the acquired 3D model with enhanced photo-realism.

Displays combining both 2D and 3D views have been shown to support higher performance on certain visualization tasks. However, it is not clear how best to arrange a combination of 2D and 3D views spatially in a display. In this study, we analyzed the eyegaze strategies of participants using two arrangements of 2D and 3D views to estimate the relative position of objects in a 3D scene. Our results show that the 3D view was used significantly more often than individual 2D views in both displays, indicating the importance of the 3D view for successful task completion. However, viewing patterns were significantly different between the two displays: transitions through centrally-placed views were always more frequent, and users avoided saccades between views that were far apart. Although the change in viewing strategy did not result in significant performance differences, error analysis indicates that a 3D overview in the center may reduce the number of serious errors compared to a 3D overview placed off to the side.

This paper presents a new technique for extracting visual saliency from experimental eye tracking data. An eye-tracking system is employed to determine which features that a group of human observers considered to be salient when viewing a set of video images. With this information, a biologically inspired saliency map is derived by transforming each observed video image into a feature space representation. By using a feature normalisation process based on the relative abundance of visual features within the background image and those dwelled on eye tracking scan paths, features related to visual attention are determined. These features are then back projected to the image domain to determine spatial areas of interest for unseen video images. The strengths and weaknesses of the method are demonstrated with feature correspondence for 2D to 3D image registration of endoscopy videos with computed tomography data. The biologically derived saliency map is employed to provide an image similarity measure that forms the heart of the 2D/3D registration method. It is shown that by only processing selective regions of interest as determined by the saliency map, rendering overhead can be greatly reduced. Significant improvements in pose estimation efficiency can be achieved without apparent reduction in registration accuracy when compared to that of using a non-saliency based similarity measure.

The levator ani is vulnerable to injury during childbirth and effective surgical intervention requires full knowledge of the morphology and mechanical properties of the muscle structure. The shape of the levator ani and regional thickening during different levels of physiological loading can provide an indication of pelvic floor dysfunction. This paper describes a coupled approach for shape and thickness statistical modelling based on harmonic embedding for volumetric analysis of the levator ani. With harmonic embedding, the dynamic information extracted by the statistical modelling captures shape and thickness variation of the levator ani across different subjects and during varying levels of stress. With this study, we demonstrate that the derived model is compact and physiologically meaningful, demonstrating the practical value of the technique. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Respiratory-induced cardiac deformation is a major problem for high-resolution cardiac imaging. This paper presents a new technique for predictive cardiac motion modeling and correction, which uses partial least squares regression to extract intrinsic relationships between three-dimensional (3-D) cardiac deformation due to respiration and multiple one-dimensional real-time measurable surface intensity traces at chest or abdomen. Despite the fact that these surface intensity traces can be strongly coupled with each other but poorly correlated with respiratory-induced cardiac deformation, we demonstrate how they can be used to accurately predict cardiac motion through the extraction of latent variables of both the input and output of the model. The proposed method allows cross-modality reconstruction of patient specific models for dense motion field prediction, which after initial modeling can be used for real-time prospective motion tracking or correction. Detailed numerical issues related to the technique are discussed and the effectiveness of the motion and deformation modeling is validated with 3-D magnetic resonance data sets acquired from ten asymptomatic subjects covering the entire respiratory range.

Due to its static nature, the inference capability of Bayesian networks (BNs) often deteriorates when the basis of input data varies, especially in video processing applications where the environment often changes constantly. This paper presents an adaptive BN where the network parameters are adjusted in accordance to input variations. An efficient retraining method is introduced for updating the parameters and the proposed network is applied to shadow removal in video sequence processing with quantitative results demonstrating the significance of adapting the network with environmental changes.

In surgery, virtual and augmented reality are increasingly being used as new ways of training, preoperative planning, diagnosis and surgical navigation. Further development of virtual and augmented reality in medicine is moving towards photorealistic rendering and patient specific modeling, permitting high fidelity visual examination and user interaction. This coincides with the current development in computer vision and graphics where image information is used directly to render novel views of a scene. These techniques require extensive use of geometric information about the scene and provide a comprehensive review of the underlying techniques required for building patient specific models with photorealstic rendering. It also highlights some of the opportunities that image based modeling and rendering techniques can offer in the context of minimally invasive surgery.

This paper presents a new framework for capturing intrinsic visual search behavior of different observers in image understanding by analysing saccadic eye movements in feature space. The method is based on the information theory for identifying salient image features based on which visual search is performed. We demonstrate how to obtain feature space fixation density functions that are normalized to the image content along the scan paths. This allows a reliable identification of salient image features that can be mapped back to spatial space for highlighting regions of interest and attention selection. A two-color conjunction search experiment has been implemented to illustrate the theoretical framework of the proposed method including feature selection, hot spot detection, and back-projection. The practical value of the method is demonstrated with computed tomography image of centrilobular emphysema, and we discuss how the proposed framework can be used as a basis for decision support in medical image understanding.

This paper presents a new method of knowledge gathering for decision support in image understanding based on information extracted from the dynamics of saccadic eye movements. The framework involves the construction of a generic image feature extraction library, from which the feature extractors that are most relevant to the visual assessment by domain experts are determined automatically through factor analysis. The dynamics of the visual search are analyzed by using the Markov model for providing training information to novices on how and where to look for image features. The validity of the framework has been evaluated in a clinical scenario whereby the pulmonary vascular distribution on Computed Tomography images was assessed by experienced radiologists as a potential indicator of heart failure. The performance of the system has been demonstrated by training four novices to follow the visual assessment behavior of two experienced observers. In all cases, the accuracy of the students improved from near random decision making (33%) to accuracies ranging from 50% to 68%.

This paper presents a virtual tagging framework for measuring, as well as visualising, myocardial deformation using magnetic resonance (MR) velocity imaging. Tagging grids are allocated artificially according to the deformation gradient with varying shapes and densities. The control points are then deformed such that the difference between the induced deformation velocity and that of actually measured MR data is minimum. A full three-dimensional implementation of the technique combined with the mass conservation constraint is provided. Numerical considerations of applying the proposed framework and different optimization strategies have been investigated with both simulated and phantom experiments. The accuracy of the technique in terms of following material deformation is compared with that of conventional tagging technique.

The identification of bronchi on Computed Tomography (CT) images of the lungs provides valuable clinical information in patients with suspected airways diseases including bronchiectasis, emphysema, or constrictive obliterative bronchiolitis. The automated recognition of the airways is, therefore, an important part of a diagnosis aid system for resolving potential ambiguities associated with intensity-based feature extractors. On CT images, near-perpendicular cross sections of bronchi normally appear as elliptical rings and this paper presents a novel technique for their recognition. The proposed method, the edge-radius-symmetry (ERS) transform, is based on the analysis of the distribution of edges in local polar coordinates. Pixels are ranked according to local edge (E) strength, radial (R), uniformity and local symmetry (S). A discrete implementation of the technique is provided which reduces the computational cost of the ERS transform by using a geometric approximation of the intensity patterns. The identification of the adjacent pulmonary vessels with template matching then allows for the automated measurement of bronchial dilatation and bronchial wall thickening. Computationally, the method compares favorably with other methods such as the Hough transform. Noise-sensitivity of the technique was evaluated on a set of synthetic images and nine patients under investigation for suspected airways disease. Agreement for the automated scoring of the presence and severity of bronchial abnormalities was demonstrated to be comparable to that of an experienced radiologist (kappa statistics kappa > 0.5 ).

We present a new algorithm for the robust and accurate tracking of the aorta in cardiovascular magnetic resonance (MR) images. First, a rough estimate of the location and diameter of the aorta is obtained by applying a multiscale medial-response function using the available a priori knowledge. Then, this estimate is refined using an energy-minimizing deformable model which we define in a Markov-random-field (MRF) framework. In this context, we propose a global minimization technique based on stochastic relaxation, Simulated annealing (SA), which is shown to be superior to other minimization techniques, for minimizing the energy of the deformable model. We have evaluated the performance and robustness of the algorithm on clinical compliance studies in cardiovascular MR images. The segmentation and tracking has been successfully tested in spin-echo MR images of the aorta. The results show the ability of the algorithm to produce not only accurate, but also very reliable results in clinical routine applications.

Bronchiolar obstruction is commonly manifested in computed tomography (CT) images as areas of decreased attenuation relative to adjacent normal lung parenchyma. The certain identification of such areas is difficult in practice, particularly if they are poorly marginated. This paper presents a novel approach to the enhancement of feature differences between normal and diseased lung parenchyma so that reliable visual assessment can be made. The method relies on a hybrid structural filtering technique which removes pulmonary vessels appearing in the CT cross-sectional images without affecting intrinsic subtle intensity details of the lung parenchyma. In order to restore possible structural distortions introduced by the hybrid filter, a feature localization process based on wavelet reconstruction of feature extrema is used. After contrast enhancement the resultant images are used to delineate region borders of the diseased areas and quantification is made with regard to the extent of the disease.

Blood movement investigated by magnetic resonance (MR) velocity mapping is generally presented in the form of velocity components in one or more chosen velocity encoding directions. By viewing these components separately, it is difficult for MR practitioners to conceptualize and comprehend the underlying flow structures, especially when the image data have strong background noise. A flow visualization technique that adapts the idea of particle tracing used in classical fluid dynamics for visualizing flow is presented. The flow image processing relies on the strong correlation between the principal flow direction estimated from the distribution of the modulus of the velocity field and the direction derived from the raw image data. By correlation calculation, severe background noise can be eliminated. Flow pattern rendering and animation provide an efficient way for representing internal flow structures.>

The widespread prevalence of atherosclerotic vascular disease has given rise to the need for a simple, noninvasive imaging examination of the cardiovascular performance of patients. The potential of using Magnetic Resonance (MR) Imaging to quantify flow in vivo has for reaching possibilities for the future of preventive medicine. In this paper we address the problem of using MR velocity imaging to analyse the flow boundaries in human arteries which are of great importance to the early diagnosis of occlusive diseases. A flow related enhancement process is introduced in this paper. It is designed to suppress the residuals and the noisy background of the MR velocity images caused by misregistration, tissue movement and uneven magnetic field and provide great improvement in signal to noise ratio. From the enhanced image, the main flow areas can be delineated by a thresholding process which defines the kernel of the flow. The boundaries of the kernel region are then dynamically guided by a defined flow boundary localization process to their final positions. The results of the application of this coarse to fine process show its robustness and effectiveness for the determination of the blood blow boundaries form very low quality MR velocity images.