The papers in this special section present the most recent research work that showcases the state-of-the-art of human-centered computing and its potential applications in developing truly smart home automation systems.

Redundant tendon-driven systems such as the human hand or the ACT robotic hand are high-dimensional and nonlinear systems that make traditional control strategies ineffective. The synergy hypothesis from neuroscience suggests that employing dimensionality reduction techniques can simplify the system without a major loss in function. We define a dimensionality reduction framework consisting of separate observation and activation synergies, a first-order model, and an optimal controller. The framework is implemented for two example tasks: adaptive control of thumb posture and hybrid position/force control to enable dynamic handwriting.

We apply path integral reinforcement learning to a biomechanically accurate dynamics model of the index finger and then to the Anatomically Correct Testbed (ACT) robotic hand. We illustrate the applicability of Policy Improvement with Path Integrals (PI2) to parameterized and non-parameterized control policies. This method is based on sampling variations in control, executing them in the real world, and minimizing a cost function on the resulting performance. Iteratively improving the control policy based on real-world performance requires no direct modeling of tendon network nonlinearities and contact transitions, allowing improved task performance.

We present a novel and simple experimental method called physical human interactive guidance to study human-planned grasping. Instead of studying how the human uses his/her own biological hand or how a human teleoperates a robot hand in a grasping task, the method involves a human interacting physically with a robot arm and hand, carefully moving and guiding the robot into the grasping pose, while the robot's configuration is recorded. Analysis of the grasps from this simple method has produced two interesting results. First, the grasps produced by this method perform better than grasps generated through a state-of-the-art automated grasp planner. Second, this method when combined with a detailed statistical analysis using a variety of grasp measures (physics-based heuristics considered critical for a good grasp) offered insights into how the human grasping method is similar or different from automated grasping synthesis techniques. Specifically, data from the physical human interactive guidance method showed that the human-planned grasping method provides grasps that are similar to grasps from a state-of-the-art automated grasp planner, but differed in one key aspect. The robot wrists were aligned with the object's principal axes in the human-planned grasps (termed low skewness in this paper), while the automated grasps used arbitrary wrist orientation. Preliminary tests show that grasps with low skewness were significantly more robust than grasps with high skewness (77-93%). We conclude with a detailed discussion of how the physical human interactive guidance method relates to existing methods to extract the human principles for physical interaction.

Muscle synergies are hypothesized as a way to reduce the control space of redundant biological neuromuscular control. Robotics researchers are starting to use grasping synergies to simplify the task of coordinating many joints for complex manipulators. This is especially useful for hand movement control as there are many joints and muscles to control to achieve even simple tasks. This paper uses the Anatomically Correct Testbed (ACT) Robotic Hand to understand how task-specific synergies may be formed and used for complex robotic manipulation tasks such as writing. A comparison is made between synergies formed from (1) general hand movements, (2) task-specific demonstration of generic writing, and (3) practice of writing a specific letter. Results showed that performance using task specific demonstration synergies outperforms performance using general-purpose synergies in terms of completion time, energy expenditure, and trajectory error. Performance using practiced synergy outperforms the performance using demonstration synergies in trajectory error, even though there was no statistical difference in completion time and energy expenditure. These results indicate that task specific synergies from demonstration and practice allow a robotic hand to write better than using more generic synergies that may work for other tasks. How general/specific these synergies should be to optimize the performance of different complex tasks without learning too many specific synergies is an interesting topic for the future.

In order to further understand what physiological characteristics make a human hand irreplaceable for many dexterous tasks, it is necessary to develop artificial joints that are anatomically correct while sharing similar dynamic features. In this paper, we address the problem of designing a two degree of freedom metacarpophalangeal (MCP) joint of an index finger. The artificial MCP joint is composed of a ball joint, crocheted ligaments, and a silicon rubber sleeve which as a whole provides the functions required of a human finger joint. We quantitatively validate the efficacy of the artificial joint by comparing its dynamic characteristics with that of two human subjects' index fingers by analyzing their impulse response with linear regression. Design parameters of the artificial joint are varied to highlight their effect on the joint's dynamics. A modified, second-order model is fit which accounts for non-linear stiffness and damping, and a higher order model is considered. Good fits are observed both in the human (R2= 0.97) and the artificial joint of the index finger (R2= 0.95). Parameter estimates of stiffness and damping for the artificial joint are found to be similar to those in the literature, indicating our new joint is a good approximation for an index finger's MCP joint.

In the past, the music community conducted research on what makes music more musical or expressive. Much of this work has focused on the manipulation of phrasing, articulation and rubato to make music more expressive. However, it has been difficult to study neuromuscular control used by experts to create such musical music. This paper took a first step toward this effort by using the Anatomically Correct Testbed (ACT) Robotic Hand to mimic the way expert humans play when they are instructed to perform "musically" or "robotically." Results from 22 human subjects showed that musical expression contained a larger range of dynamics and different articulation than robotic expression, while there was no difference in the use of rubato. The ACT Hand was controlled to the level of precision that allowed the replication of expert expressive performance. Its performance was then rated by 17 human listeners against music played by a human expert to show that the ACT Hand could play as musically as an expert human. Furthermore, articulation, phrasing, and rubato were tested in isolation to determine the importance of articulation over phrasing and rubato. This type of study will lead to understanding how to implement future robots to perform musically without pre-programming them, finding ways to teach novice pianists strategies in controlling their muscles to become expressive musicians more quickly, and understanding why humans feel expressiveness or even emotion in music.

Humans are adept at grasping different objects robustly for different tasks. Robotic grasping has made significant progress, but still has not reached the level of robustness or versatility shown by human grasping. It would be useful to understand what parameters (called grasp measures) humans optimize as they grasp objects, how these grasp measures are varied for different tasks, and whether they can be applied to physical robots to improve their robustness and versatility. This paper demonstrates a new way to gather human-guided grasp measures from a human interacting haptically with a robotic arm and hand. The results revealed that a human-guided strategy provided grasps with higher robustness on a physical robot even under a vigorous shaking test (91%) when compared with a state-of-the-art automated grasp synthesis algorithm (77%). Furthermore, orthogonality of wrist orientation was identified as a key human-guided grasp measure, and using it along with an automated grasp synthesis algorithm improved the automated algorithm's results dramatically (77% to 93%).

With the goal of developing human-like dextrous manipulation, we investigate how the central nervous system uses the redundant control space of the human hand to perform tasks with force-stiffness requirements. Specifically, while the human hand is actuated by several muscles with varying mechanical advantage (called the moment arm), it is unclear how each muscle is used. Using the anatomically correct testbed (ACT) robotic hand to compute the control solution space and human-subject experiments with surface electromyography to measure biological control strategy, we identified that there is significant redundancy in the control spaces of both muscles with large moment arms and muscles with small moment arms. However, the central nervous system was selective about the solution for muscles with large moment arms, while it chose to span large regions of the available control space for muscles with small moment arms. Furthermore, the biological solution used low-moment-arm muscles at relatively high actuation levels. We summarize by making inferences on why the central nervous system chooses such a strategy and how this can help robotic manipulation.

In this paper we present our low-level software framework that allows human neuromuscular properties and behaviors to be programmed modularly onto biologically inspired prosthetic devices. Specifically this software framework is used at the lowest level of the anatomically correct testbed (ACT) hand. It abstracts away from the robotic details and allows users including biologists to program individual muscle physiology, multiple muscle synergistic activities, neural commands, etc. independently. Our system has been designed to be fault-tolerant with the idea of safe and continued operation of prostheses if a new prototype behavior were to perform incorrectly or outright fail. Example coding of Hills model is shown.

Human hands are capable of many dexterous grasping and manipulation tasks. To understand human levels of dexterity and to achieve it with robotic hands, we constructed an anatomically correct testbed (ACT) hand which allows for the investigation of the biomechanical features and neural control strategies of the human hand. This paper focuses on developing control strategies for the index finger motion of the ACT Hand. A direct muscle position control and a force-optimized joint control are implemented as building blocks and tools for comparisons with future biological control approaches. We show how Gaussian process regression techniques can be used to determine the relationships between the muscle and joint motions in both controllers. Our experiments demonstrate that the direct muscle position controller allows for accurate and fast position tracking, while the force-optimized joint controller allows for exploitation of actuation redundancy in the finger critical for this redundant system. Furthermore, a comparison between Gaussian processes and least squares regression method shows that Gaussian processes provide better parameter estimation and tracking performance. This first control investigation on the ACT hand opens doors to implement biological strategies observed in humans and achieve the ultimate human-level dexterity.

Safety is a critical factor when designing a robotic rehabilitation environment. Whole-limb or life-size haptic interaction would allow virtual robotic rehabilitation of daily living activities such as sweeping or shelving. However, it has been too dangerous to implement such an environment with conventional active robots that use motor, hydraulic, or pneumatic actuation. To address this issue, a life-size 6-degree-of-freedom (DOF) brake-actuated manipulator (BAM) was designed and constructed. This paper details the BAM's system models including mechanisms, kinematics, and dynamics, as well as detailed input and friction models. In addition, a new system-identification technique that utilizes human input to excite the robot's dynamics with unscented Kalman filtering was employed to identify system parameters. Noise sources are discussed, and the model is validated through force estimation with inverse dynamics. Model parameters and performance are compared with other commercially available haptic devices. The BAM shows a significantly larger workspace, maximum force, and stiffness over other devices exhibiting its promise toward rehabilitative applications.

With the goal of developing biologically inspired manipulation strategies for an anthropomorphic hand, we investigated how the human central nervous system utilizes the hands redundant neuromusculoskeletal biomechanics to transition between conditions. Using a experiment protocol where subjects were asked to transit between control states with equal end-effector force but different stiffness requirements, we observed that (1) some subjects used the same muscle synergy for both conditions by maintaining the same synergy throughout the transition, and (2) other subjects used two different muscle synergies to execute two conditions by transiting from one synergy to another rapidly. We hypothesize that humans typically try to use the same muscle synergy to execute two tasks when it is possible to optimize on the simplicity and speed over energy. This is a different control strategy from the way robots have been controlled in the past, and it provides a new direction in controlling an anthropomorphic robotic hand.

The properties of dissipative haptic displays allow larger workspaces that permit a whole body interaction useful for sports, rehabilitation, and large-scale object design applications. To that end we designed and constructed the brake actuated manipulator (BAM) with 2m3workspace. Dissipative devices are capable of simulating virtual objects through resistance analogous to active devices. However, the challenge remains for path guidance paradigms because neither impedance nor admittance control can be used to actively steer limb movements. Here we first define a new way to create and track a path during path guidance with a twinned vector field to allow bilateral motion. Using this new path definition three controllers, velocity ratio, force cancelling, and force mapping are compared with and without visual feedback. The results indicate that both force controllers provide better guidance over velocity control; the force mapping technique resulted in the smoothest limb trajectory. The presence of visual feedback was found to be a critical factor for path guidance using dissipative devices.

Our previous work has shown that visual distortion of subject-controlled goals can be used to manipulate force production and movement distance. Thus, visual distortion may be a helpful addition to robotic rehabilitation for brain-injured and stroke patients. In this work, we investigated the effects of distortion on unimpaired subjects learning a two-finger coordinated pinch task. Rather than distorting the subject's goal, we enhanced the apparent error attributable to one or both fingers. Distortion affected the distribution of attention between the two fingers; when the error enhancement was given to one finger, the performance of that finger improved relative to the other. This result indicates that enhanced visual error could be used to manipulate the performance and the coordination pattern between fingers to achieve desired movements during rehabilitation

The thumb is essential to the hand's function in grasping and manipulating objects. Previous anthropomorphic robot hands have thumbs that are biologically-inspired but kinematically-simplified. In order to study the biomechanics and neuromuscular control of hand function, an anatomical robotic model of the human thumb is constructed for the anatomically-correct testbed (ACT) hand. This paper presents our ACT thumb kinematic model that unifies a number of studies from biomechanical literature. We also validate the functional consistency (i.e. the nonlinear moment arm values) between the cadaveric data and the ACT thumb. This functional consistency preserves the geometric relationship between muscle length and joint angles, which allows robotic actuators to imitate human muscle functionality

Artificial skin research has focused on three primary characteristics: high density, flexible sensing arrays, and the ability to decompose all three components of an applied force vector. Previous research has accomplished some but not all of these characteristics. In this paper, we present a new flexible, high density sensor array and a triaxial force discernment algorithm that can be used with commercially available flexible high density sensor arrays. Together, the sensing array and algorithm achieve all three of the critical skin characteristics. With an addition of thin silicone pillar arrays, the triaxial force discernment algorithm provides a direct solution for nearly all possible force directions. Where the solution is ambiguous, our algorithm is able to reduce the uncertainty to a small range of known values. To confirm the validity of the algorithm, a testbed for a single sensing element was developed. Initial results show a resolution of less than 0.01 radians and calculated force components within the uncertainty of the ground truth for most situations. Our approach opens doors for many applications including robotic manipulation, haptic interfaces, human-robot interaction, and medical diagnoses

We create a robotic rehabilitation environment that distorts the visual feedback of a movement representation to restore lost movements. The use of visual-feedback distortion produces a perceptual gap between the perceived (visual) and actual somatosensory experiences, in which movements can be manipulated without patients' knowledge. Previously, we reported the smallest amount of distortion that is imperceptible by measuring just noticeable difference (JND), the distortion size that is tolerable without detection, and the invariance of perceived physical effort under visual-feedback distortion. In this paper, we report the performance changes under visual-feedback distortion in a robotic rehabilitation environment. By interleaving trials with no visual feedback, we showed that the internal somatosensory representation of the movement goal changed along with visual distortion for unimpaired and motor-impaired subjects. In addition, a gradual change of visual-feedback distortion within one experiment allowed movement changes significantly larger than the JND. These changes were robust under sudden lack of visual feedback and even when subjects were informed of possible distortion ahead of time. Finally, preliminary results of a six-week rehabilitation paradigm are reported. Improvements in the patient's hand condition were found, particularly in the range of motion of the index and middle fingers

Scaled teleoperation is increasingly prevalent in medicine, as well as in other applications of robotics. Visual feedback in such systems is essential and should make maximal use of natural hand-eye coordination. This paper describes a new method of visual feedback for scaled teleoperation in which the operator manipulates the handle of a remote tool in the presence of a registered virtual image of the target in real time. The method adapts a concept already used successfully in a new medical device called the Sonic Flashlight, which permits direct in situ visualization of ultrasound during invasive procedures. The Sonic Flashlight uses a flat-panel monitor and a half-silvered mirror to merge the visual outer surface of a patient with a simultaneous ultrasound scan of the patient's interior. Adapting the concept to scaled teleoperation involves removing the imaging device and the target to a remote location and adding a master-slave control device. This permits the operator to see his hands, along with what appears to be the tool, and the target, merged in a workspace that preserves natural hand-eye coordination. Three functioning prototypes are described, one based on ultrasound and two on light microscopy. The limitations and potential of the new approach are discussed.

The neural-based control of a robotic hand has many clinical and engineering applications. Current approaches to this problem have been limited due to a lack of understanding of the relationship between neural signals and dynamic finger movements. Here, we present a technique to predict index finger joint angles from neural signals recorded from the associated muscles. The neural signals are converted to a torque estimate (EBTE) and then input to artificial neural networks. The networks predict the finger position more closely when the input to the networks are torque estimates rather than neural signals. Furthermore, the networks trained with the EBTE signals could predict the joint angles for different phases of finger movements (i.e. dynamic reaching and positioning task) while networks trained with the neural signals could not. Our results indicate that (1) similar finger movements are executed with different synergistic strategies and (2) different phases of finger movements employ different neural strategies. Through these results, we have demonstrated the first concrete technique to control a hand prosthetic device or dexterous tele-manipulator using natural neural control signals.

In order to design a robotic rehabilitation environment using visual feedback distortion, we investigated in this study the limits and effects of visual feedback distortion for the elderly and the motor-impaired. To determine the minimum imperceptible amount of visual distortion, we measured the Just Noticeable Differences (JNDs) for force and position for elderly, unimpaired subjects; values of 31.0% (0.619 N) and 16.1% (5.01 mm), respectively, were obtained. JNDs of 46.0% (0.920 N) and 45.0% (14.8 mm) were measured for a motor-impaired individual. These JNDs were larger than corresponding measurements previously taken with young subjects, showing a decrease in discrimination ability with age and impairment. Visual distortion based on these values caused elderly subjects and the motor-impaired individual to increase their force production levels by 72.5% and 97.7%, respectively. These results were similar to those obtained with young subjects, but differences were observed on interspersed trials with no visual feedback. Poor discrimination abilities in elderly and impaired subjects and visual dominance in our environment for this subject group support our hypothesis that visual distortion can be an effective tool for rehabilitation in a robotic environment.

To restore dexterity to paralyzed hands, we have designed and constructed a lightweight, low-profile orthotic exoskeleton controlled by the user's residual electromyography (EMG) signals. In this paper, we compared several simple strategies to control the orthotic device for a quadriplegic (C5/C6) subject. When contralateral arm control was employed, we found that a simple on/off strategy allowed for faster interaction with objects, while variable control provided more controlled interactions, especially with deformable objects. Furthermore, we designed a control strategy that allowed for a natural reaching and pinching sequence without the use of the contralateral arm. We validated that the EMG signal from the ipsilateral biceps could be used to develop an extremely reliable natural reaching and pinching algorithm. This evaluation showed that different control strategies may be appropriate for different situations, and further investigation on the natural algorithm is crucial.

We are constructing an anatomically-correct testbed (ACT) of the human hand to understand its mechanisms, function, and neural control. As a part of this effort, we have created an actuator that mimics both the active and passive behaviors of the human muscles. To use it as the actuator of the ACT Hand, it must conserve the tendon-driven structure so that the size or the weight of the hand does not have to be compromised. A custom-made spring composite is used to closely simulate the human's nonlinear passive muscle stiffness (R/sup 2/=0.99). A coreless DC motor is used to simulate the active contraction. The passive and active components interact in parallel and they are attached to a cable tendon that inserts into the bones. In order to study the neural control of hand movements, we incorporate Hill's model in the active contraction component. Therefore, the computer program can simply specify the muscle activation level, and the appropriate tendon tension can be provided given the current muscle length. This muscle actuator allows an accurate realization of neural activation based control of the ACT Hand, and provides a foundation for intimate brain-machine interface.

Robotic hands built for manipulation are often anthropomorphic but not anatomically accurate. We are constructing an anatomically-correct testbed (ACT) of the human hand to understand its mechanisms, function, and control. We have previously demonstrated that an accurate model of the extensor mechanism in the ACT Hand is crucial in realizing human-like finger movements. Here, we present the design of the bones and joints that form the skeletal structure for the ACT Hand. The bones are machined from human bone data, and are accurate in surface shape, mass, and center-of-gravity, while the joints have been designed to match both degrees-of-freedom and passive stiffness. Our evaluation of the assembled index finger confirms the anatomic properties, and reveals the function of some of the peculiar shapes of the finger bones, the necessity of matching the joint passive stiffness properties, and the connections of the extensor mechanism.

Direct electrical communication between living nervous systems and external devices would allow a wide variety of clinical and engineering applications. This neuro-robotic interface has not yet been fully achieved because existing interface systems cannot establish sufficiently close contact between neurons and communicating electrodes. We present here the synthesis and testing of a new class of conductive polymer that can be used to coat metal electrodes and achieve the necessary close contact. The polymer coatings can be made to incorporate biological adhesion proteins to maximize biocompatibility. Neurons extracted from mouse and rat brains were able to attach to the coatings, survive beyond five days, and grow out long, communicating processes. These new polymer films offer the potential for a new level of communication between robotic devices and the nervous system and may eventually make a high-bandwidth interface a reality.

Real-time accurate human posture tracking in unconstrained environments provides an enabling technology for physicians and other care providers to monitor the movements of their patients in real-life situations. Constructing a posture tracking system with the form factor suitable for human wear requires the development of miniature units that can be attached to the limb segments of interest in an unobtrusive way. Simultaneously, fast algorithms that can produce real-time posture estimates at sufficient rates are needed. In this paper, we focus on the development of efficient and accurate algorithms that compute the human posture information from low-cost miniature inertial and magnetic sensors. We present a new technique that computes posture estimates from the sensor data 23.8 times faster than the most efficient previously proposed technique, and simultaneously increases the accuracy of the estimates.

The human finger possesses a structure called the extensor mechanism, a web-like collection of tendinuous material that lies on the dorsal side of each finger and connects the controlling muscles to the bones of the finger. In past robotic hand designs, extensor mechanisms have generally not been employed due in part to their complexity and a lack of understanding of their utility. This paper presents our first design and analysis effort of an artificial extensor mechanism. The goal of our analysis is to provide an understanding of the extensor mechanism's functionality so that we can extract the crucial features that need to be mimicked to construct an anatomical robotic hand. With the inclusion of an extensor mechanism, we believe that all possible human finger postures can be achieved using four cable driven actuators, we identified that this extensor mechanism gives independent control of the metacarpo-phalangeal (MCP) joint and acts not only as an extensor but also as a flexor, abductor, adductor, or rotator depending on the finger's posture.