In this article, we present a novel approach for continuous operator authentication in teleoperated robotic processes based on Hidden Markov Models (HMM). While HMMs were originally developed and widely used in speech recognition, they have shown great performance in human motion and activity modeling. We make an analogy between human language and teleoperated robotic processes (i.e., words are analogous to a teleoperator’s gestures, sentences are analogous to the entire teleoperated task or process) and implement HMMs to model the teleoperated task. To test the continuous authentication performance of the proposed method, we conducted two sets of analyses. We built a virtual reality (VR) experimental environment using a commodity VR headset (HTC Vive) and haptic feedback enabled controller (Sensable PHANToM Omni) to simulate a real teleoperated task. An experimental study with 10 subjects was then conducted. We also performed simulated continuous operator authentication by using the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). The performance of the model was evaluated based on the continuous (real-time) operator authentication accuracy as well as resistance to a simulated impersonation attack. The results suggest that the proposed method is able to achieve 70% (VR experiment) and 81% (JIGSAWS dataset) continuous classification accuracy with as short as a 1-second sample window. It is also capable of detecting an impersonation attack in real-time.

Drawing tasks, such as completing an Archimedes spiral, are commonly used to diagnose and evaluate tremor severity in individuals presenting with tremor-like symptoms. Clinical evaluation of these tasks is generally restricted to having a clinician examine and rate performance using one of several rating scales. Although this method is relatively effective in tremor assessment, it restricts tremor assessment to clinical settings and requires each assessment be evaluated individually, placing considerable time demands on both patients and clinicians. Here is presented the first multi-class approach to evaluation of tremor severity based on data remotely recorded by a mobile- or tablet-based drawing application. Taking a divergent approach from previously published work, which focuses solely on binary diagnostic capacity of similar systems, our work seeks to differentiate between healthy subjects, essential tremor patients receiving deep brain stimulation treatment, and those same patients with treatment disabled. Our classification algorithm was highly effective, with overall accuracy of 97.04%. Of note is that all erroneously classified samples within this set were ”treated” samples mistakenly labelled as ”healthy” samples, implying that DBS treatment is capable of completely eliminating tremor. Future work will focus on adapting these findings into a regression algorithm capable of automatically evaluating tremor severity on an established tremor rating scale, allowing fully automated remote assessment of tremor severity for the coming era of telemedicine.

Chronic electrocorticography (ECoG) studies will be necessary to understand how end-users of assistive brain-computer interface (BCI) technologies will control their devices on a long-term scale. A BCI platform consisting of a deep brain stimulator (DBS) and a cortical electrode strip, theoretically suitable for use over multi-year periods, is validated in a single human subject with essential tremor (ET). At the time of this writing, stable ECoG recordings have been demonstrated for four months after implantation. Additionally, the patient has performed a BCI cursor-control task with better-than-chance accuracy using both (1) an overt movement control scheme and (2) a motor imagery control scheme. These results validate the sensing-enabled DBS with cortical strip system as an exciting platform for chronic ECoG BCI experiments and as a scientific tool for long-term cortical recordings in different patient populations.

This paper presents a method of augmenting streaming point cloud data with pretouch proximity sensor information for the purposes of teleoperated grasping of transparent targets. When using commercial RGB-Depth (RGB-D) cameras, material properties can significantly affect depth measurements. In particular, transparent objects are difficult to perceive with RGB images and commercially available depth sensors. Geometric information of such objects needs to be gathered with additional sensors, and in many scenarios, it is of interest to gather this information without physical contact. In this work, a non-contact pretouch sensor fixed to the robot end effector is used to sense and explore physical geometries previously unobserved. Thus, the point cloud representation of an unknown, transparent grasp target, can be enhanced through telerobotic exploration in real-time. Furthermore, real-time haptic rendering algorithms and haptic virtual fixtures used in combination with the augmented streaming point clouds assist the teleoperator in collision avoidance during exploration. Theoretical analyses are performed to design virtual fixtures suitable for pretouch sensing, and experiments show the effectiveness of this method to gather geometry data without collision and eventually to successfully grasp a transparent object.

Haptic technologies have made it possible for human users to interact with cyber systems not only via traditional interfaces like keyboards and mice but also by applying force and motion. With these extra information channels, how a user haptically interacts with a system potentially presents unique user dependent features and can thus be used for authentication purposes. In this paper, we propose a new biometric technology based on haptic interaction. Our technique leverages artificial neural network (ANN) based wavelet analysis to perform user identification. Identification and authentication are done in two steps: a discrete wavelet transform (DWT) is applied to extract features, and then the neural network is used to perform identification and authentication. The performance of the model is evaluated based on identification and authentication accuracies. The results show that our proposed haptic password system has a high identification accuracy and that it is resistant to forgery attacks.

This paper presents a constraint-based method for haptic interaction between an arbitrary voxelized polygon tool and a streaming point cloud derived from a depth sensor. Using the presented method, a user can interact with both dynamic as well as static point cloud representations of real objects captured in real-time. Every depth image frame is filtered and surface normals are calculated in real-time. For movement of the virtual tool, a `quasi-static' simulation is used. The innovations of this work include the extension of haptic rendering from streaming point clouds to six degrees of freedom. This is appropriate for co-robotic tasks where haptic feedback to the user is combined with remote control of a robot.

This paper investigates automation of soft tissue compression for robot-assisted surgery. This is a fundamental task in surgery and includes interaction with a variety of tissues with unknown properties. In addition, due to sterilization and size constraints the use of contact force and position sensors are often avoided in surgical applications. We propose an Adaptive Model Predictive Control approach for execution of given tool trajectories in contact with unknown tissues in the absence of contact measurements. The Unscented Kalman Filter is employed in advance of system operation to identify the dynamics of a cable driven manipulator. These dynamics are then used to estimate contact force and position in free motion and in contact with tissue. An optimal control problem for automating tissue compression is formulated and is solved in real-time using Differential Dynamic Programming with Automatic Differentiation. The proposed methods are evaluated in experiments on an artificial tissue sample with unknown properties.

Cable driven power transmissions are used in applications such as haptic devices, surgical robots etc. The use of flexible cable based power transmission often causes relative motion between the motor actuator and mechanism joint during operation due to the elasticity of the cable. State-space control methods can be used to improve performance, but may require state estimates. For nonlinear systems, the Unscented Kalman Filter (UKF) provides a computationally efficient way to obtain state estimates. The UKF is applied here to a simulation of a minimially invasive surgical robot, to study the state estimation for a cable driven system with nonlinear dynamics. State estimates from the UKF are compared with the known states available from the simulation. These state estimates are also utilized by two different controllers interacting with the simulation to test the UKF performance under closed loop control. We tested the UKF performance with error perturbations in the system model's cable stiffness parameter.

Several established methods for remote touching using non-contact sensors exist. Applications for these methods are primarily within the field of robotic teleoperation. In surgical robotics it would be useful to not only touch, but also be able to maintain a distance from a certain organ. The latter can be done using non-contact sensors such as cameras. The novelty in this paper is the idea of combining forbidden-region virtual fixtures with haptic rendering from streaming point clouds. This is then used to protect as well as remotely touch a beating heart without any a priori knowledge of the heart geometry (such as from CT/MR scans).

This paper proposes a novel algorithm for haptic rendering from time varying point clouds captured using an Xbox Kinect RGB-D camera. Existing methods for point-based haptic rendering using a proxy can not directly be applied in this situation since no information about the underlying objects is given. This paper extends the notion of proxy to point clouds. The haptic algorithm presented here renders haptic forces from point clouds captured in real-time representing both static and dynamic objects.

The empirical mode decomposition (EMD) has seen widespread use for analysis of nonlinear and nonstationary time-series. Despite some practical success, it lacks a firm theoretical foundation. This work addresses two important theoretical properties. The original EMD algorithm is slightly modified, in a way that facilitates this analysis. For periodic, band-limited, signals the convergence and time scale separation of the algorithm are proved.

The empirical mode decomposition (EMD) has seen widespread use for analysis of nonlinear and nonstationary time-series. Despite some practical success, it lacks a firm theoretical foundation. This work addresses this. The original EMD algorithm is slightly modified, in a way that facilitates its analysis. We prove three theorems that give conditions for the convergence and time scale separation of the EMD for stationary, band-limited signals.

In robot-assisted surgery, surgical tools interact with tissues that have nonlinear mechanical properties. For situations where a pre-specified trajectory of tool positions (or applied forces) is desired, there are many controller designs that might be used. Four candidates are comparatively evaluated here, via computer simulation involving a nonlinear model of soft tissue behavior during grasping actions. The parameters for this model were obtained experimentally (in earlier work). The four candidate controllers are: (1) a well- tuned PID controller; (2) feedback linearization in combination with deadbeat control; (3) an optimal open-loop control law obtained via minimization of a quadratic cost function; and (4) a model predictive controller. Simulation trials are used to compare the transient performance of these candidate controllers under different assumptions regarding input and output noises. The conditions where each of the candidates is best are characterized.

The Vocal Joystick is a novel human-computer interface mechanism designed to enable individuals with motor impairments to make use of vocal parameters to control objects on a computer screen (buttons, sliders, etc.) and ultimately electro-mechanical instruments (e.g., robotic arms, wireless home automation devices). We have developed a working prototype of our "VJ-engine" with which individuals can now control computer mouse movement with their voice. The core engine is currently optimized according to a number of criterion. In this paper, we describe the engine system design, engine optimization, and user-interface improvements, and outline some of the signal processing and pattern recognition modules that were successful. Lastly, we present new results comparing the vocal joystick with a state-of-the-art eye tracking pointing device, and show that not only is the Vocal Joystick already competitive, for some tasks it appears to be an improvement.

Modeling is essential to integrating knowledge of human physiology. Comprehensive self-consistent descriptions expressed in quantitative mathematical form define working hypotheses in testable and reproducible form, and though such models are always "wrong" in the sense of being incomplete or partly incorrect, they provide a means of understanding a system and improving that understanding. Physiological systems, and models of them, encompass different levels of complexity. The lowest levels concern gene signaling and the regulation of transcription and translation, then biophysical and biochemical events at the protein level, and extend through the levels of cells, tissues and organs all the way to descriptions of integrated systems behavior. The highest levels of organization represent the dynamically varying interactions of billions of cells. Models of such systems are necessarily simplified to minimize computation and to emphasize the key factors defining system behavior; different model forms are thus often used to represent a system in different ways. Each simplification of lower level complicated function reduces the range of accurate operability at the higher level model, reducing robustness, the ability to respond correctly to dynamic changes in conditions. When conditions change so that the complexity reduction has resulted in the solution departing from the range of validity, detecting the deviation is critical, and requires special methods to enforce adapting the model formulation to alternative reduced-form modules or decomposing the reduced-form aggregates to the more detailed lower level modules to maintain appropriate behavior. The processes of error recognition, and of mapping between different levels of model complexity and shifting the levels of complexity of models in response to changing conditions, are essential for adaptive modeling and computer simulation of large-scale systems in reasonable time.

Fuzzy system identification was applied to a biomedical system for classification purposes. Gait achieved through functional electrical stimulation (FES) of paraplegics was divided using sensor measurements of kinematic variables as inputs to five discrete events. Two identification algorithms were used to estimate the system model. Both max-min and max-product composition were used. Membership functions were either trapezoidal or triangular and all membership functions in a particular universe of discourse had the same shape and size. The universe of discourse was varied by altering the overlap between membership functions. The classification performance was assessed quantitatively, by measuring the percentage of time steps in which the correct event was found, and qualitatively, by observing types of errors. The identification algorithm affected system performance. No difference in classification was found between max-min and max-product composition. The performance was dependent on membership function overlap. A comparison of the classification found using the fuzzy rule base versus that found using a traditional look-up table demonstrated that the fuzzy approach was superior. It is speculated that the use of fuzzy logic decreased errors stemming from sensor noise and/or small variations in the input signals. The performance of this approach was compared to that of a feedforward neural network and the fuzzy system is found superior.

A bounded sample path control strategy, based on the idea of minimizing an upper bound of the possible costs to go, is formulated. The finite and infinite versions of the JLQBSP (jump linear quadratic bounded sample path) problem are solved. A set of sufficient conditions for the existence and uniqueness of steady-state solutions that stabilize the controlled system with certainty (i.e. on any sample path) is also presented. The resulting costs are finite. The sufficient conditions are based on concepts of absolute controllability and observability of the jump linear system. This JLQBSP controller requires less precise information about form transition probabilities (only the directed interaction matrix is needed) and provides reliable control in all circumstances. Consequently, sometimes it is more appropriate for potential applications than JLQ control algorithms that are optimal in only an average sense.>

The authors adopt a control-systems perspective in reviewing past applications of functional neuromuscular stimulation for providing lower extremity motor function in paralyzed individuals. Specifically, their approach emphasizes direct computer-controlled electrical stimulation of paralyzed muscle rather than triggering reflexes. In experimental settings it provides paralyzed individuals with the ability to do functional tasks while standing, to walk short distances on varying surfaces, to negotiate obstacles, and to climb and descend stairs.>