Surface Electromyography (sEMG) is a non-invasive signal that is used in the recognition of hand movement patterns, the diagnosis of diseases, and the robust control of prostheses. Despite the remarkable success of recent end-to-end Deep Learning approaches, they are still limited by the need for large amounts of labeled data. To alleviate the requirement for big data, we propose utilizing a feature-imitating network (FIN) for closed-form temporal feature learning over a 300ms signal window on Ninapro DB2, and applying it to the task of 17 hand movement recognition. We implement a lightweight LSTM-FIN network to imitate four standard temporal features (entropy, root mean square, variance, simple square integral). We observed that the LSTM-FIN network can achieve up to 99% R2 accuracy in feature reconstruction and 80% accuracy in hand movement recognition. Our results also showed that the model can be robustly applied for both within- and cross-subject movement recognition, as well as simulated low-latency environments. Overall, our work demonstrates the potential of the FIN modeling paradigm in data-scarce scenarios for sEMG signal processing.

In the past decade, there has been significant advancement in designing wearable neural interfaces for controlling neurorobotic systems, particularly bionic limbs. These interfaces function by decoding signals captured noninvasively from the skin’s surface. Portable high-density surface electromyography (HD-sEMG) modules combined with deep learning decoding have attracted interest by achieving excellent gesture prediction and myoelectric control of prosthetic systems and neurorobots. However, factors like small electrode size and unstable electrode-skin contacts make HD-sEMG susceptible to pixel electrode drops. The sparse electrode-skin disconnections rooted in issues such as low adhesion, sweating, hair blockage, and skin stretch challenge the reliability and scalability of these modules as the perception unit for neurorobotic systems. This paper proposes a novel deep-learning model providing resiliency for HD-sEMG modules, which can be used in the wearable interfaces of neurorobots. The proposed 3D Dilated Efficient CapsNet model trains on an augmented input space to computationally ‘force’ the network to learn channel dropout variations and thus learn robustness to channel dropout. The proposed framework maintained high performance under a sensor dropout reliability study conducted. Results show conventional models’ performance significantly degrades with dropout and is recovered using the proposed architecture and the training paradigm.

Stroke is a life-threatening medical condition that could lead to mortality or significant sensorimotor deficits. Various machine learning techniques have been successfully used to detect and predict stroke-related outcomes. Considering the diversity in the type of clinical modalities involved during management of patients with stroke, such as medical images, bio-signals, and clinical data, multimodal machine learning has become increasingly popular. Thus, we conducted a systematic literature review to understand the current status of state-of-the-art multimodal machine learning methods for stroke prognosis and diagnosis. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines during literature search and selection, our results show that the most dominant techniques are related to the fusion paradigm, specifically early, joint and late fusion. We discuss opportunities to leverage other multimodal learning paradigms, such as multimodal translation and alignment, which are generally less explored. We also discuss the scale of datasets and types of modalities used to develop existing models, highlighting opportunities for the creation of more diverse multimodal datasets. Finally, we present ongoing challenges and provide a set of recommendations to drive the next generation of multimodal learning methods for improved prognosis and diagnosis of patients with stroke.

Development of advance surface Electromyogram (sEMG)-based Human-Machine Interface (HMI) systems is of paramount importance to pave the way towards emergence of futuristic Cyber-Physical-Human (CPH) worlds. In this context, the main focus of recent literature was on development of different Deep Neural Network (DNN)-based architectures that perform Hand Gesture Recognition (HGR) at a macroscopic level (i.e., directly from sEMG signals). At the same time, advancements in acquisition of High-Density sEMG signals (HD-sEMG) have resulted in a surge of significant interest on sEMG decomposition techniques to extract microscopic neural drive information. However, due to complexities of sEMG decomposition and added computational overhead, HGR at microscopic level is less explored than its aforementioned macroscopic-level, DNN-based counterparts. In this regard, we propose the HYDRA-HGR framework, which is a hybrid model for HGR that simultaneously extracts a set of temporal and spatial features through its two independent Vision Transformer (ViT)-based parallel architectures (the so called Macro and Micro paths). The Macro Path is trained directly on the pre-processed HD-sEMG signals, while the Micro path is fed with the p-to-p values of the extracted Motor Unit Action Potentials (MUAPs) of each source. Extracted features at macroscopic and microscopic levels are then coupled via a Fully Connected (FC) fusion layer for final gesture classification. We evaluate the proposed hybrid HYDRA-HGR framework through a recently released HD-sEMG dataset, and show that it significantly outperforms its stand-alone counterparts. The proposed HYDRA-HGR framework achieves average accuracy of 94.86% for the 250 ms window size, which is 5.52 % and 8.22 % higher than that of the Macro and Micro paths, respectively.

Electroadhesive clutches have attracted a great deal of interest in the last decade as semi-active actuators for human-robot interaction due to their lightweight, low power consumption, and tunable high-torque output capability. However, because of the complexity of their dynamics, in most cases, they are utilized in an ON/OFF -control strategy. In this regard, the non-autonomous (time-dependent) degradation of electroadhesive behavior is an inherent challenge that injects unpredictability and uncertainty into the behavior of this family of semi-active clutches. We propose a novel approach to preventing degradation of electroadhesion using a segmented electrode design that modulates the electrical field on the dielectric surface while using a direct current signal and securing low power consumption. This paper, for the first time, presents an optimization process based on a novel analytic model of the proposed actuator. It also develops a data-driven model augmentation using a hybrid shallow learning approach composed of a long short-term memory (LSTM) architecture which is combined with the analytical model. The performance of the proposed semi-active clutch and the data-driven hybrid model is experimentally validated in this paper.

Communication latency in any delicate telerobotic operation (such as remote surgery over distance) would impose a significant challenge due to the temporal degradation of visual perception and can substantially affect the outcomes. Less is known, however, about the neurophysiological basis of how operators adapt/react to delayed visual feedback. Identification of such neural markers might provide novel ways for future applications to monitor the mental workload (MW). In this study, we recorded electroencephalography (EEG) data from nine users while performing a peg transfer task using the da Vinci Research Kit with three levels of induced visual delay in the video feedback. Our results suggest that spectral EEG-based features can provide markers of the operator's MW modulated by arbitrary visual delay. We also show that the exposure to different visual delays could be successfully classified/detected solely from EEG data, using a Riemannian geometry-based classifier, which highlights the utility of EEG signals for detecting the effect of visual delay on brain activity.

The intrinsic biomechanical characteristic of the human upper limb plays a central role in absorbing the interactive energy during physical human-robot interaction (pHRI). We have recently shown that based on the concept of “Excess of Passivity (EoP),” from nonlinear control theory, it is possible to decode such energetic behavior for both upper and lower limbs [1], [2]. The extracted knowledge can be used in the design of controllers (such as [2]-[5]) for optimizing the transparency and fidelity of force fields in human-robot interaction and in haptic systems. In this paper, for the first time, we investigate the frequency behavior of the passivity map for the upper limb when the muscle co-activation was controlled in real- time through visual electromyographic feedback. Five healthy subjects (age: 27±5) were included in this study. The energetic behavior was evaluated at two stimulation frequencies at eight interaction directions over two controlled muscle co-activation levels. Electromyography (EMG) was captured using the Delsys Wireless Trigno system. Results showed a correlation between EMG and EoP, which was further amplified by decreasing the frequency. The proposed energetic behavior is named the Geometric MyoPassivity (GMP) map. The findings indicate that the GMP map has the potential to be used in real-time to quantify the absorbable energy, thus passivity margin of stability for upper limb interaction during pHRI.

This paper proposes a smart handheld textural sensing medical device with complementary Machine Learning (ML) algorithms to enable on-site Colorectal Cancer (CRC) polyp diagnosis and pathology of excised tumors. The proposed unique handheld edge device benefits from a unique tactile sensing module and a dual-stage machine learning algorithms (composed of a dilated residual network and a t-SNE engine) for polyp type and stiffness characterization. Solely utilizing the occlusion-free, illumination-resilient textural images captured by the proposed tactile sensor, the framework is able to sensitively and reliably identify the type and stage of CRC polyps by classifying their texture and stiffness, respectively. Moreover, the proposed handheld medical edge device benefits from internet connectivity for enabling remote digital pathology (boosting the diagnosis in operating rooms and promoting accessibility and equity in medical diagnosis).

The human limb possesses a remarkable capacity to absorb energy during physical human-robot interaction (pHRI), which can be quantified as the biomechanical “Excess of Passivity” (EoP) using non-linear control theory. This biome-chanical passivity index can be used to reduce conservatism and increase the transparency of pHRI stabilizers. Previous work on EoP has used system identification techniques to compute EoP offline. However, for use in real-time controllers, an instantaneous method for EoP estimation would be desired. This paper hypothesizes that muscle fatigue can potentially be a complicating factor which can cumulatively affect the ability of human biomechanics to absorb mechanical energy over time during physical interaction with robots. In this work, we focused on the energetic behavior of the human wrist during pHRI, and, for the first time, we investigated the effect of fatigue on EoP. The EoP for five participants was computed throughout one hundred-second trials of high-frequency wrist perturbations in four directions. Subjects maintained a stiff and consistent grip throughout each trial, causing an accumulation of fatigue in the forearm muscles. Muscle activity was recorded using an array of sixteen sEMG sensors. It was found that the EoP degraded (in a statistically significant manner) with increased muscle fatigue in all directions, even when the level of muscle co-contraction was controlled consistently through a visual myofeedback mechanism. 100% of the subjects exhibited this decline in energy absorption capacity in all directions studied. The median drop in EoP after one-hundred seconds of perturbation was 11% for trials in the abduction and adduction directions and 22% in the pronation and supination directions. These results indicate a need for more robust estimation methods or new modalities to account for muscle fatigue in the control architectures of physical human-robot interaction.

This study proposes a novel diagnosis framework to decrease the early detection miss rate of colorectal cancer (CRC) polyps by using a hypersensitive vision-based tactile sensor (HySenSe) and a deep residual neural network. The HySenSe generates high-resolution 3D textural images of 160 realistic polyp phantoms for accurate classification via the proposed deep learning (DL) architecture. The DL module explores lightweight dilated convolutions, residual neural network architecture, and transfer learning to overcome the challenge of a small dataset of 229 images. Results show that the proposed architecture outperforms state-of-the-art DL models (i.e., EfficientNet and DenseNet) with a 94% accuracy, offering a promising solution for improving early detection of CRC polyps. The proposed framework can be used as a diagnostic module within tele-assessment medical robots, highlighting the potential of advanced technology and deep learning to revolutionize the early detection and treatment of CRC.

In the context of physical human-(tele)robot interaction, passivity-based stabilizers have been used to guarantee the physical or (tele) physical stability. In most of these examples, human biomechanics is considered an inherently passive system that dissipates energy. This assumption may not hold true when the interaction is implemented in the force-position domain, even though such a setting would be needed to boost positional accuracy and avoid the common kinematic drifts in the force-velocity domains. The aforementioned topic is examined in this paper using the concept of shortage versus excess of passivity index for human biomechanics in the force-position domain. We also investigate the compounding effect of the frequency of interaction. The outcomes of this paper will be imperative for the design of force-position domain pURI stabilizers when the classical assumption of passivity of human biomechanics can lead to serious safety issues. In this work, for the first time, we quantitatively present the passivity margin and, thus, the energetic behavior of the human arm's biomechanics under various interaction scenarios in the Force-Position domain. The outcome of this work includes a three-dimensional passivity index map (3DPiM) that is validated on five healthy participants. The goal is to illustrate the passivity margin of the human upper limb biomechanics for two distinct levels of muscle co-contractions, as indicated by the Electromyography (EMG) signal, across four interaction frequencies and eight geometric directions. This outcome enables the future development of biomechanics-aware stabilizers in the force-position domain, quantifying the passivity margin in real-time and thus significantly reducing the stabilizer's conservatism while ensuring the safety of human-robot interactions.

Characterization of fatigue using surface electromyography (sEMG) data has been motivated for rehabilitation and injury-preventative technologies. Current sEMG-based models of fatigue are limited due to (a) linear and parametric assumptions, (b) lack of a holistic neurophysiological view, and (c) complex and heterogeneous responses. This paper proposes and validates a data-driven non-parametric functional muscle network analysis to reliably characterize fatigue-related changes in synergistic muscle coordination and distribution of neural drive at the peripheral level. The proposed approach was tested on data collected in this study from the lower extremities of 26 asymptomatic volunteers (13 subjects were assigned to the fatigue intervention group, and 13 age/gender-matched subjects were assigned to the control group). Volitional fatigue was induced in the intervention group by moderate-intensity unilateral leg press exercises. The proposed non-parametric functional muscle network demonstrated a consistent decrease in connectivity after the fatigue intervention, as indicated by network degree, weighted clustering coefficient (WCC), and global efficiency. The graph metrics displayed consistent and significant decreases at the group level, individual subject level, and individual muscle level. For the first time, this paper proposed a non-parametric functional muscle network and highlighted the corresponding potential as a sensitive biomarker of fatigue with superior performance to conventional spectrotemporal measures.

The excess of passivity (EoP) of the human biomechanics plays an imperative role in absorbing the interaction energy during physical human–(tele)robot interaction and can be exploited by controllers used for stabilization of human-centered (tele)robotic systems. However, the first generation of nonlinear EoP-based stabilizers loaded the force reflection channel resulting in degradation of the force profile. This will challenge applications that are heavily dependent on the quality of force reflection, such as telerobotic rehabilitation. This article explores the possibility of developing a nonlinear stabilizer that modifies the reflected velocity to the follower-side operator based on the corresponding EoP map. As an applied benefit in the context of telerehabilitation, the proposed stabilizer does not require information about the EoP of all patients; instead, it would require that for the individual therapist who works with the group of patients. The article provides the mathematical derivation and stability proof of the nonlinear design of the stabilizer named “power-based velocity-domain variable structure passivity signature control (PV-VSPSC).” The proposed nonlinear stabilizer is evaluated through systematic experiments and systematic grid simulation studies in this paper.

Advances in biosignal signal processing and machine learning, in particular Deep Neural Networks (DNNs), have paved the way for the development of innovative Human-Machine Interfaces for decoding the human intent and controlling artificial limbs. DNN models have shown promising results with respect to other algorithms for decoding muscle electrical activity, especially for recognition of hand gestures. Such data-driven models, however, have been challenged by their need for a large number of trainable parameters and their structural complexity. Here we propose the novel Temporal Convolutions-based Hand Gesture Recognition architecture (TC-HGR) to reduce this computational burden. With this approach, we classified 17 hand gestures via surface Electromyogram (sEMG) signals by the adoption of attention mechanisms and temporal convolutions. The proposed method led to 81.65% and 80.72% classification accuracy for window sizes of 300 ms and 200 ms, respectively. The number of parameters to train the proposed TC-HGR architecture is 11.9 times less than that of its state-of-the-art counterpart.

Going beyond the traditional sparse multi-channel peripheral human-machine interface that has been used widely in neurorobotics, high-density surface electromyography (HD-sEMG) has shown significant potential for decoding upper-limb motor control. We have recently proposed heterogeneous temporal dilation of LSTM in a deep neural network architecture for a large number of gestures (>60), securing spatial resolution and fast convergence. However, several fundamental questions remain unanswered. One problem targeted explicitly in this paper is the issue of “electrode shift,” which can happen specifically for high-density systems and during doffing and donning the sensor grid. Another real-world problem is the question of transient versus plateau classification, which connects to the temporal resolution of neural interfaces and seamless control. In this paper, for the first time, we implement gesture prediction on the transient phase of HD-sEMG data while robustifying the human-machine interface decoder to electrode shift. For this, we propose the concept of deep data augmentation for transient HD-sEMG. We show that without using the proposed augmentation, a slight shift of 10mm may drop the decoder's performance to as low as 20%. Combining the proposed data augmentation with a 3D Convolutional Neural Network (CNN), we recovered the performance to 84.6% while securing a high spatiotemporal resolution, robustifying to the electrode shift, and getting closer to large-scale adoption by the end-users, enhancing resiliency.

This article presents an overview of the current state of research and application of haptic (primarily kinesthetic) feedback and force-based teleoperation in the context of surgical robotics. Telerobotic surgery provides an approach for transferring the sensorimotor skills of a surgeon through a robotic platform to perform surgical intervention inside a patient’s body. Integration of advanced sensing and haptic technologies in telerobotic surgery can help to enhance the sensory awareness and motor accuracy of the surgeon, thereby leading to improved surgical procedures and outcomes for patients. The primary mode of sensory feedback has been through 3-D visual observation using stereo endoscopes. However, until recently, the sense of touch, i.e., haptics, has been missing in the commercial telesurgery robots approved for use in the operating room despite over two decades of research and development in the field of haptics for teleoperated systems (“telehaptics”). Research has shown that high-fidelity force feedback can enhance the performance of telesurgery and potential outcomes by enabling the surgeon to have a more natural feel of interaction between surgical tools and tissue as normally experienced during open surgery. Interaction forces, such as those generated during palpation of tissue, insertion of a needle, unintentional (and potentially unsafe) exertion of force by a tool, suture breakage, needle slippage, or tool interaction, are replaced by indirect (virtual) sensations, termed visual haptics, which provides an alternative to sensory compensation. Although there is a significant amount of literature supporting this benefit, there are still several important technical challenges in introducing haptics in telesurgery, including instrumentation, fidelity (transparency), stability, and modalities for force reflection, e.g., direct or indirect. This article examines these challenges and discusses recent work on haptics-based teleoperated surgical robotic systems.

While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience. By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.

Bilateral telerobotic systems have attracted a great deal of interest during the last two decades. The major challenges in this field are the transparency and stability of remote force rendering, which are affected by network delays causing asynchrony between the actions and the corresponding reactions. In addition, the overactivation of stabilizers further degrades the fidelity of the rendered force field. In this article, a real-time frequency-based delay compensation approach is proposed to maximize transparency while reducing the activation of the stabilization layer. The algorithm uses a regulated bound-limited multiple Fourier linear combiner to extract the dominant frequency of force waves. The estimated weights are used in conjunction with the relatively phase-lead harmonic kernels to reconstruct the signal and generate a compensated wave to reduce the effect of the delay. The reconstructed force will then pass through a modulated time-domain passivity controller to guarantee the stability of the system. We will show that the proposed technique will reduce the force-tracking error by 40% and the activation of the stabilizer by 79%. It is shown, for the first time, that through the utilization of online adaptive frequency-based prediction, the asynchrony between transmitted waves through delayednetworks can be significantly mitigated while stability can be guaranteed with less activation of the stabilization layer.

This work is motivated by the recent advancements of Deep Neural Networks (DNNs) for myoelectric prosthesis control. In this regard, hand gesture recognition via surface Electromyogram (sEMG) signals has shown a high potential for improving the performance of myoelectric control prostheses. Although the recent researches in hand gesture recognition with DNNs have achieved promising results, they are still in their infancy. The recent literature uses traditional supervised learning methods that usually have poor performance if a small amount of data is available or requires adaptation to a changing task. Therefore, in this work, we develop a novel hand gesture recognition framework based on the formulation of FewShot Learning (FSL) to infer the required output given only one or a few numbers of training examples. Thus in this paper, we proposed a new architecture (named as FHGR which refers to "Few-shot Hand Gesture Recognition") that learns the mapping using a small number of data and quickly adapts to a new user/gesture by combing its prior experience. The proposed approach led to 83.99% classification accuracy on new repetitions with few-shot observations, 76.39% accuracy on new subjects with few-shot observations, and 72.19% accuracy on new gestures with few-shot observations.

The time-domain passivity approach has been proposed in the literature in a variety of formats to guarantee the stability of teleoperation leader-follower systems. The conventional use of the proposed technique utilizes the control effort at the follower side as the force feedback to be sent back to the user at the leader’s side. However, this has resulted in transparency problems, especially when the follower dynamics are not negligible. On the other hand, four-channel and three-channel Lawrence architectures have been investigated widely in the literature to maximize the transparency of the system when, in most advanced cases, stability is guaranteed using wave-variables. However, wave-variables are historically known for their transparency deterioration problems. In this paper, we propose a two-layer approach taking advantage of the fusion of (a) a more optimal derivation of Lawrence telerobotic architecture (utilizing only two channels), and (b) a two-port time-domain passivity stabilizer while comparing the performance with a one-port passivity stabilizer. The two-channel derivation of the Lawrence architecture allows for implementing a two-port time domain passivity approach, which is investigated in this paper. The performance of this is compared systematically through a multi-objective approach by analyzing dissipated energy besides force and velocity errors for a wide range of time delays and frequencies of excitation. The paper gives a comprehensive view of the efficacy of two-port versus one-port time-domain passivity control when combined with the two-channel derivation of Lawrence architecture.

Capitalizing on the goal of addressing identified shortcomings of recent solutions developed for recognition tasks via sparse multichannel surface Electromyography (sEMG) signals, the paper proposes a novel deep learning model, referred to as the XceptionTime architecture. The proposed innovative XceptionTime architecture is designed by integration of depthwise separable convolutions, adaptive average pooling, and a novel no-linear normalization technique. At the hearth of the proposed architecture is several XceptionTime modules concatenated in series fashion designed to captures both temporal and spatial information-bearing contents of the sparse multichannel sEMG signals without the need for data augmentation and manual design of feature extraction. In addition to instruction of the new XceptionTime module, by integration of adaptive average pooling, instead of fully connected layers, and utilization of a novel non-linear normalization approach, the proposed architecture is less prone to overfitting, more robust to temporal translation of the input, and more importantly is independent from the input window size, i.e., there is no need to change/reconfigure the architecture by changing the size of the input sequence. Finally, by utilizing the depthwise separable convolutions, the XceptionTime network has far less parameters resulting in less complex network.

This paper presents a novel light-weight back-drivable inherently-safe robotic mechanism for delivering ankle rehabilitation therapies. The robot is designed to be used as the ankle module of a multi-purpose lower-limb rehabilitation robot. A novel friction-based safety feature has been introduced that enables mechanical adjustment of the maximum amount of allowable transfer forces and torques to the patient's limb. The design procedure, mathematical modeling and experimental validations are provided to demonstrate the performance of the proposed system.

Tremor extraction techniques are considered as the central component of several rehabilitative and compensatory robotic technologies, and the accuracy of such filters can directly affect the performance of the aforementioned technologies. Motivated by this fact, the paper proposes an adaptive estimation framework, referred to as Multiple Adaptive Reduced-order Kalman filtering (KFE-BMFLC), for extraction of pathological hand tremors. The proposed KFE-BMFLC framework is designed with the goal of improving the performance of an existing state-of-the-art filtering technique, i.e. Enhanced Band-limited Fourier Linear Combiner (E-BMFLC), which has shown a promising potential in extracting involuntary hand motions but uses embedded least mean square (LMS) estimation approach. The proposed technique is capable of reducing the computational overhead in comparison to that of the conventional BMFLC technique, while increasing the estimation accuracy.

This paper presents a novel multimodal training platform integrated with hand-over-hand (HOH) haptic guidance for dual-console surgical robotic systems such as the da Vinci Si system. The expert-in-the-loop (EIL) framework incorporates a fuzzy interface system in order to provide a trainee with adaptive authority over the procedure as well as hand-over-hand haptic guidance adjusted in real time based on the proficiency level of the trainee. The EIL expertise-oriented framework enables performance of a surgical procedure by an expert surgeon on a patient, while simultaneously providing a trainee at any stage of the motor-skills development with multimodal training without jeopardizing patient safety. Closed-loop stability of the system is investigated using the circle criterion and it is shown that the proposed architecture is unconditionally stable. Experimental evaluations are presented in support of the proposed platform through the implementation of a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA, USA) and the dV-Trainer master console (Mimic Technology, Inc., Seattle, WA, USA). To the best of our knowledge, the implemented setup is the first research platform for dual-console studies involving the classic da Vinci surgical system.

In this paper, the design of a novel bilateral telerobotic architecture for rehabilitation purposes is proposed and the related feasibility, stability, and control challenges are studied. The objective is to incorporate the supervision of a local/remote human physiotherapist into haptics-enabled rehabilitation systems and allow the therapist to provide nonpassive nonlinear assistive/resistive forces in response to the patient's movements. This can address a challenge of conventional software-based rehabilitation systems, i.e., limited capability in adjusting the therapy. To guarantee human-robot interaction safety, a new design framework and a stabilizing controller are developed based on the small-gain approach. System stability and transparency are analyzed in the presence of the nonpassive, nonlinear, and nonautonomous behavior of the terminals (the therapist and the patient) and time-varying delays for the case of remote and cloud-based therapy. Several practical considerations have been taken into account to match the clinical needs and minimize the implementation cost. Simulation studies, practical implementation, and experimental evaluations are presented.

This paper presents a Therapist-in-the-Loop (TIL) framework for robotics-assisted mirror rehabilitation therapy integrated with adaptive Assist-as-Needed (ANN) training, to be adjusted based on the impairment and disability level of the patient's affected limb. Closed-loop system stability has been investigated using a combination of the Circle Criterion and the Small-Gain Theorem to account both for time-delay and the time-varying adaptive ANN training. Experiments to investigate the performance of the proposed framework are reported.

In this paper, a new passivity-based technique is proposed to analyze and guarantee the stability of haptics-enabled telerobotic rehabilitation systems where there is a possibility of having more sources of non-passivity than communication delays. In practice, the difficulty of therapeutic exercises should be tuned taking into account the stage of physical disability. However, tuning the difficulty and intensity should not violate the stability of patient-robot interaction. This usually puts conservative prefixed limits on the allowable exercise intensity. In this paper, patient-robot interaction safety is studied in the context of Strong Passivity Theory (SPT). Our goal is to ultimately relax the limitation on the allowable robotic therapies while preserving system stability. The proposed stabilizing scheme does not try to make the entire non-passive component passive. This allows the therapist to have freedom in injecting energy into the system for assistive therapies while ensuring safe patient-robot interaction. In this paper, the case of telerobotic rehabilitation is considered. Experimental implementation and evaluation are presented to support the proposed theory.

This paper proposes an expertise-oriented training platform for robotics-assisted minimally invasive surgery. The framework builds on previous work of the authors and makes use of dual-user teleoperation scenario, allowing the presence of an expert in the training loop. A Fuzzy-Logic (FL) methodology is proposed, which specifies the level/mode of the training required for the trainee according to his/her level of proficiency over the task. A major advantage of the proposed FL approach is that, having the expert in the loop, it can specify the trainee's proficiency level relative to that of the expert in real-time. Moreover, based on the relative skills assessment, the proposed FL approach decides if or to what extent the trainee should receive a haptic guidance force based on Virtual Fixtures or the environment force from the interaction between the surgical instrument and tissue at the slave side. In addition to the level/mode of the haptics-enabled training required for the trainee, the proposed FL framework specifies the authority level of the trainees over the operation in real-time, according to their proficiency levels over the task. Stability of the overall closed-loop teleoperated system is also investigated using the small-gain theorem, resulting in a sufficient condition to guarantee stability in the presence of constant communication delays. Finally, experimental results are given to evaluate the design and feasibility of the proposed framework.

Concentric-tube robots can offer a suitable compromise between force and curvature control. In a previous study by the authors, a real-time trajectory tracking scheme for an unloaded concentric-tube robot was developed. One of the practical barriers to the use of a concentric-tube robot in medical applications is compensation for the impact of environmental forces which can cause drastic deterioration in tracking performance. In this paper, by modifying the robot's forward kinematics and Jacobian, a new method is developed to facilitate tip tracking in real-time while accounting for an external load at the robot's tip. By considering the tip deflection resulting from the external load, a novel dual-layer control architecture is proposed to compensate for this deflection during trajectory tracking. In order to measure the force exerted on the tip position of the robot, a new technique is proposed that can move the sensing system from the distal tip to the proximal base. Experimental results are given to illustrate the effectiveness of the proposed method.

In this paper, a safety concern arising from pathological tremors in patients interacting with haptics-enabled rehabilitation robots is analyzed and the issue of tremor amplification for assistive/coordinative robotic rehabilitation is investigated. In order to deal with this issue, a control architecture is proposed to dissipate the extra energy of the system and guarantee its stability and safety of the patient. For this purpose, (a) first, a multilayer adaptive filter is proposed to estimate high-frequency components of hand motions (corresponding to involuntary movements); (b) then a resistive force field is generated and applied by the robot to attenuate the tremor; and (c) simultaneously the residual low-frequency voluntary actions are amplified/coordinated to deliver appropriate therapy. Stability analysis and a stabilization scheme are developed to guarantee safe interaction regardless of variations in the patient's dynamics and tremor kinematics. The ultimate goal is to make it possible for patients with pathological tremors to take advantage of non-passive robotic assistive/coordinative therapy. This would not be possible using conventional systems due to the possibility of tremor amplification. Experimental results are presented.

In this paper, a novel robotics-assisted rehabilitation framework is proposed for bilateral mirror-image therapy. For this purpose, a customized dual-user teleoperation architecture is designed incorporating Guidance Virtual Fixtures (GVFs) to deliver the appropriate therapeutic movements to the patient's impaired limb by providing an assist-as-needed treatment strategy. In addition, the therapist is provided with informative haptic feedback that is generated based on the patient's movements, allowing the therapist to decide in real-time on the level and format of the therapy required for the patient. Stability of the closed-loop system is also investigated using the small gain theorem, in the presence of communication time delays, facilitating the case of remote tele-rehabilitation. Experimental results are given to validate the performance of the proposed platform.

In this paper, a robotic solution is proposed to deal with the challenges caused by lung motion during needle insertion. To accomplish this goal, a macro-micro robotic tool is designed to compensate for tissue motion using the macro part, while performing the needle insertion independently with the micro part. The main application of this work is for robotics-assisted lung tumor biopsy, where the combined motions of respiration and heartbeat may compromise success. An impedance-based controller keeps the macro reference coordinate in contact with the moving soft tissue using measurements from small pressure sensors mounted at the tip of the macro shaft. The micro part, mounted at the end of the macro robot, manipulates the needle in the harmonized reference coordinate system. Preoperative identification of ex vivo soft tissue is performed to estimate the dynamic behavior of the tissue. The controller is then synthesized based on the identified model. The effects of identification error and high frequency uncertainty are addressed in the control design. A prototype was built to evaluate the proposed approach using: 1) two Mitsubishi PA-10 robots, one for manipulating the macro part and the other for mimicking tissue motion, 2) one motorized linear stage to handle the micro part, and 3) a Phantom Omni haptic device for remote manipulation. Experimental results demonstrate the performance of the motion compensation system.

This paper proposes a teleoperated dual-user system incorporating Virtual Fixtures (VFs) that allows concurrent performance of a robotic surgical task by an expert and a trainee. In order to guide the trainee through the procedure, an adaptive VF is created in the trainee's workspace according to the motion generated by the expert who is performing the surgery at the same time. The VF gets adaptively adjusted based on the level of expertise the trainee shows during the surgery. In addition, the trainee's level of expertise is used to adaptively adjust the dual-user dominance factor in an online fashion, which gives the trainee some authority over the task based on his/her skill level. To quantify the trainee's expertise level, a performance measure is proposed, based on the force generated by the VF. Three performance measures from the literature are also used. To satisfy the desired objectives of the proposed system, an impedance-based control methodology is adopted. Stability of the closed-loop system is investigated using the small-gain theorem. A sufficient stability condition is derived that guarantees stability in the presence of time-varying communication delay. Experimental results are given to validate the performance of the system.

The problem of designing of a haptics-enabled teleoperated rehabilitation system in the presence of communication delays is addressed. In a teleoperated rehabilitation system, communication delays introduce phase shift which may result in the task inversion phenomenon. To overcome the task inversion, a new type of projection-based force reflection algorithm is proposed which is suitable for assistive/resistive therapy in the presence of irregular communication delays. Additionally, algorithms for augmented therapy are introduced which combine the projection-based force reflection with a delay-free local virtual therapist. A small-gain design is developed which guarantees stability of the proposed schemes for both assistive and resistive modes of the therapy. Simulations and experimental results are presented which confirm the improvement achieved by the proposed methods.

In master-slave teleoperator systems used for tele-rehabilitation purposes, the passivity can be violated because of the assistive actions of a therapist; moreover, any type of passivation approach to the design of such a system would defeat the purpose of the assistive therapy. In this paper, a design framework is presented that does not rely on passivity considerations. A number of results are given that deal with stability and transparency properties of the tele-rehabilitation system in the case of intrinsically non-passive behaviour of the therapist. In particular, a stabilization scheme for the networked tele-rehabilitation system is proposed which guarantees stability regardless of the specific actions of the therapist. Simulation results are presented which confirm the theoretical developments.

This paper focuses on control challenges caused by the slave-arm flexibility in master-slave telerobotic systems. Apart from time delay, the nonlinear non-minimum phase deflection of flexible-link slave manipulators creates extra challenges for control of telerobotic systems. These challenges have forced most of the prior research to use basic simplifications. In this paper, we try to remove most of the simplifications using a more realistic model for slave deflections. The degrading effects of flexibility on stability and performance of conventional telerobotics architectures are analyzed. Then, the standard Extended Lawrence Four-Channel (ELFC) control architecture is modified to obtain the stability condition and to enhance performance. Finally, the input-to-output stability (IOS) Small-Gain Theorem is used to generalize the stability criteria for varying delay and to remove the restrictive assumption of deflection-linearity. The proposed architecture consists of a local Partial Feedback Linearization scheme embedded into a modified ELFC architecture. This study is motivated by the use of light-weight cable driven tools in telerobotic surgery.

In this paper, a novel decentralized multilateral structure is proposed for the dual user systems in the presence of communication delay. The proposed structure utilizes adaptive impedance control approach in order to overcome the destructive effect of the time-delay on system desired-objectives, which is a disregarded issue in the previous studies on dual user system. The proposed control strategy, which utilizes three desired impedance surfaces defined in the paper, satisfactorily brings the system hybrid matrix close to the ideal one that guarantees the system stability and transparency. The controller is designed in a way that eliminates the necessity of the delay estimation as one of its outstanding characteristics; consequently, the unknown communication time-delay can be handled via this structure while previous studies have disregarded the issue of time delay in dual user system. Furthermore, the adaptive structure of the controller promises to overcome the uncertainties on robot's dynamics. In addition, the efficiency of the controller in guaranteeing the system stability in the presence of unknown communication delay is investigated through passivity theory and the presented analysis illustrates complete independency of the closed-loop system stability on time delay value applying the proposed controller. Experimental results performed on a delayed dual user system demonstrate validity of the proposed scheme.