Fingernail imaging has been proven to be effective in prior works [1], [2] for estimating the 3D fingertip forces with a maximum RMS estimation error of 7%. In the current research, fingernail imaging is used to perform unconstrained grasp force measurement on multiple fingers to study human grasping. Moreover, two robotic arms with mounted cameras and a visual tracking system have been devised to keep the human fingers in the camera frame during the experiments. Experimental tests have been conducted for six human subjects under both constrained and unconstrained grasping conditions, and the results indicate a significant difference in force collaboration among the fingers between the two grasping conditions. Another interesting result according to the experiments is that in comparison to constrained grasping, unconstrained grasp forces are more evenly distributed over the fingers and there is less force variation (more steadiness) in each finger force. These results validate the importance of measuring grasp forces in an unconstrained manner in order to study how humans naturally grasp objects.

This paper presents an automated routine for calibrating a fingernail imaging system, with the intent of predicting fingerpad forces. The system uses a magnetic levitation haptic device to apply forces to the human fingerpad while recording images of the nail and surrounding skin. A novel force controller is implemented to interact stably with the human fingerpad. The data are used to calibrate a principal component regression model relating pixel intensity to 3-D force. Using data from this automated routine, this model simultaneously predicts 3-D force with an RMS error of 0.56 ± 0.03 N (7.7% of the full range of forces).

This paper investigates the effect of two different parameters on the registration and force prediction accuracy of using Active Appearance Models (AAM) to align fingernail images. First, the color channel used to form the AAM is varied between (1) an averaged grayscale image, (2) the red channel, (3) the green channel and (4) the blue channel. Second, the number of landmark points used to create the AAM is varied between 6 and 75. The color channel is found to have an effect on the registration accuracy and the force prediction error. The green, blue and grayscale images are approximately equivalent, while the red images have a larger error across all metrics used. The number of landmark points may be reduced to 25 with no significant effect on either the registration accuracy or the force prediction error, though further reduction has shown some effect. With this information, a simpler registration model can be used that requires fewer calculations.

This paper demonstrates the effectiveness of registering fingernail images using Active Appearance Models (AAM). A shape model is formed using two contours, one to trace the edge of the fingernail and a second to outline the edge of the finger. This model is used to create an AAM, which is then used to register several hundred images. Using the images and associated force measurements, three different models relating pixel intensity to finger contact force are formed. This process is repeated for eight subjects. These image-force models could then be used to measure the finger contact force without obstructing the haptic sense or instrumenting the object being touched, allowing for more natural interaction between human and robot. Two of the models are found to predict force with an RMS error of 0.3N normal force (4% of the full range) and 0.23N shear force (3% of the full range). These two models are also shown to be nearly independent of image resolution, with no significant difference in the results for resolutions between 10-by-10 and 50-by-50. The simpler of the two models relies on the Texture Parameters generated by the AAM. Since these parameters are already calculated as part of the registration process, this removes an additional calculation step and could lead to improved real-time processing.

Shape memory alloy (SMA) actuators are compact and have high force-to-weight ratios, making them strong candidates to actuate robots, exoskeletons, and prosthetics. To optimize speed and energy consumption, SMA actuators have been embedded in an NxN vascular network that can deliver electric and thermofluidic energy to the each actuator. The scalable architecture of the vascular network allows for 2N control devices (valves, transistors) to be shared amongst N2actuators, so that as the number of actuators increases, the number of required control devices scales at a smaller rate. This Network Array Architecture (NAA) allows for each actuator to be controlled individually or in discrete subarrays. However, not all combinations of actuators can be activated simultaneously; therefore in general, a sequence of control commands will be need to be executed in order to achieve the desired actuation. By treating each actuator as having a binary state, the combined states of the actuator array can be represented by graph theory, where states are nodes and the transitions between states are graph edges. By properly weighting the costs of the transitions, graph search techniques can be used to find optimal sets of control commands for desired state changes. This paper formulates the control of NAA actuators systems as a graph theory problem, and characterizes the ability of search algorithms to optimize a weighted combination of speed and energy usage, while minimizing computational cost.

Increased demand for powered wheelchairs and their inherent mobility limitations have prompted the development of omnidirectional wheelchairs. These wheelchairs provide improved mobility in confined spaces, but can be more difficult to control and impact the ability of their users to embody the wheelchair. We hypothesize that control and embodiment of omnidirectional wheelchairs can be significantly improved by providing intuitive control with 3-DOF haptic feedback that directly corresponds to the degrees of freedom of an omnidirectional wheelchair. This paper introduces a novel 3-DOF Haptic Joystick designed for the purpose of controlling omnidirectional wheelchairs. When coupled with range finders, it is able to provide the user with feedback that improves the operator's awareness of the area surrounding the vehicle and assists the driver in obstacle avoidance. The haptic controller design and a stability analysis of the coupled wheelchair joystick systems are presented. Experimental results of the coupled systems validate the ability of the controller to influence the trajectory of the wheelchair and assist in obstacle avoidance.

Shape memory alloy (SMA) actuators are compact and have high force-to-weight ratios, making them strong candidates to actuate robots, exoskeletons, and prosthetics. However, these actuators are thermomechanical in nature and slow cooling rates can limit their performance. To improve the convective cooling, SMA wires have been embedded in vascular networks allowing cold fluid to pass across the actuators to cool/extend them. The vascular network can also deliver hot fluid to heat and contract the wire. In addition to the fluidic network, an electrical network operating in parallel can resistively heat the SMA to produce contraction. To minimize the weight and size of the vascular and electrical networks, a scalable N[N architecture has been implemented that allows for 2N control devices (valves, transistors) to be shared amongst N2actuators. This Network Array Architecture (NAA) allows each actuator to be controlled individually or in discrete subarrays. However, this architecture does not allow all combinations of actuators to be activated simultaneously; therefore in general a sequence of control commands will need to be executed in order to achieve the complete actuation. In order to find an optimal sequence of control commands, graph theory algorithms have been implemented. By treating each actuator's state as binary (fully contracted or extended), the collected states of an actuator array can be represented as nodes of the graph and the control commands as the graph edges. By properly weighting the costs of the graph edges, search algorithms can be used to find an optimal set of control commands for desired state changes. NAA results in a multi-graph that has 2NA—Nnodes and is highly interconnected. While initial work in this area established the basic validity of a graph search approach for NAA control, this article presents a scalable method of determining the optimal set of controls to minimize the operating cost (time and energy) and computational search cost.

This paper presents the design, optimization, and experimental analysis of a biologically inspired wet shape memory alloy (SMA) actuated heart for robotic systems. Just as the human heart provides energy to the muscles in the body, the robotic SMA heart distributes thermofluidic energy to arrays of SMA actuators that function as robotic muscles. Furthermore, the robotic heart draws from its own fluidic output to assist in the actuation of its own internal SMA actuators, just as a portion of the blood pumped by the human heart supplies energy to its own muscles. Dynamic modeling and simulation of the SMA actuators and pumping system are used to assist in optimizing the output of the heart. The effects of changing various parameters such as actuator length and actuation timing were explored via simulation and experiment. The current prototype of the SMA heart is capable of pumping 2.1 times more fluid than is required to sustain its own actuation. This is the first successful implementation of such a robotic heart, such that it has a net positive thermofluidic output to provide to other actuators while sustaining its own actuation via thermofluidic feedback. Furthermore, the SMA heart is capable of pumping a net output of 66 mL/min, which is two orders of magnitude larger than the output of other SMA “micro” pumps.

This paper presents a method of imaging the coloration pattern in the fingernail and surrounding skin to infer fingertip force direction (which includes four major shear-force directions plus normal force) during planar contact. Nail images from 15 subjects were registered to reference images with random sample consensus (RANSAC) and then warped to an atlas with elastic registration. With linear discriminant analysis, common linear features corresponding to force directions, but irrelevant to subjects, are automatically extracted. The common feature regions in the fingernail and surrounding skin are consistent with observation and previous studies. Without any individual calibration, the overall recognition accuracy on test images of 15 subjects was 90%. With individual training, the overall recognition accuracy on test images of 15 subjects was 94%. The lowest imaging resolution, without sacrificing classification accuracy, was found to be between 10-by-10 and 20-by-20 pixels.

This paper presents a method of imaging the coloration pattern in the fingernail and surrounding skin to infer fingertip force direction during planar contact. Nail images from 7 subjects were registered to reference images with RANSAC and then warped to an atlas with elastic registration. Recognition of fingertip force direction, based on linear discriminant analysis, shows that there are common color pattern features in the fingernail and surrounding skin for different subjects. Based on the common features, the overall recognition accuracy is 92%.

The matrix vasoconstriction device (MVD) is a scalable mechanism than can control the fluid flow through a vascular network with n2wet shape memory alloy (SMA) actuators using 2n+2 constrictors (control elements). This vascular network delivers hot and cold fluid to conductively heat and cool shape memory alloy (SMA) muscles embedded in compliant vessels. The MVD mimics smooth muscles found in biological systems by constricting fluidic vessels to prevent flow. When released, the MVD does not add any fluidic resistance to the system, which has reduced the effective fluid resistance of the vascular network to 20% of the previous vascular system controlled by solenoid valves, therefore increasing the flow rate by a factor of 5. With the MVD's increased flow rates, a cycling rate of one Hz has been achieved for a single actuator. The MVD has (2n - 1)2possible configurations, which allow fluid flow through a single or multiple vessels at the intersection of one or more released row and column constrictors. As the number of released vessels increases, the total fluid resistance of the system decreases and the total flow through the system increases. Releasing all the constrictors, the flow through a 4 times 4 array can produce 3.5 actuations per unit time (where the unit time is the time necessary to drive a single actuator). This paper examines the operation of the MVD as it controls the ternary fluid flow (hot, cold, no flow) to an n times n array of wet SMA actuators operating independently and operating columns of actuators that are connected mechanically series with each other

This paper presents a technique termed eigennails to classify fingertip force during contact based on the coloration patterns in the fingernail and surrounding skin. Fingertip force is classified into six directions: no force, normal force only, two directions (left/right) of lateral shear force, and two directions (forward/backward) of longitudinal shear forces. Based on the face recognition technique eigenfaces, a small number of eigennails are sufficient to express the color pattern features for shear force direction classification. Results show that 98% of 960 fingernail images of 8 different subjects are correctly classified. The lowest imaging resolution without sacrificing classification accuracy is found to be 10-by-10.

As an extension of our previous work on estimating fingertip forces by imaging the fingernail (Y. Sun, et al., 2006), the dynamic features of the coloration response of different parts of the fingernail and surrounding skin to different force levels are studied. The effect of the cardiovascular state on measurable coloration is also characterized. The accuracy of normal force estimated by generalized least squares is presented. A time compensation method for fast force estimation is presented, based on a knowledge of the time constants from the dynamic response of individual fingernail regions

This paper presents a new control method to drive an array of wet Shape Memory Alloy actuators utilizing a Matrix Manifold and Valve system (MMV). The MMV architecture allows a vast DOF robotic system to be controlled using limited number of resources. Using biological inspiration, the robotic system contains a network of “blood vessels” transporting fluid to the various “muscle” actuators. In general, an array of N2wet actuators can be controlled by 2N+2 fluidic valves, resulting in a scalable architecture. This would allow robots to contain a vast number of actuators like the human body, and be controlled in a scalable manner. The initial prototype contains a 4x4 array of 16 actuators controlled by 10 solenoid valves, where 25% of the actuators can be activated at any one time. Since only a subset of the actuators can be activated at any time, various methods of scheduling the resources (i.e. valves) are investigated, as well as various ways to define the error or difference between the desired and actual states of the array. The fluidic impedance of the system is taken into account in order to optimize the control. Results of simulation show that new scheduler options and definitions of error improve the performance of the system by a factor of 10 when operating the array near its maximum capacity.

Long hours of driving cause the tissues to be pressurized for an extended period and result in considerable discomfort and driver fatigue. Periodically stimulating the tissues in contact with the car seat, as well as providing adequate ventilation and pressure relief may alleviate these problems. In this paper, we describe the design of a robotic car seat capable of actively redistributing the contact pressure on the tissue and thus providing relief to the weary driver. The distributed nature of the surface actuation and space limitations necessitate the use of a large number of actuators that must be confined to a small volume. We design and use Shape Memory Alloy actuator units, which can accomplish the above goals by virtue of their high power to weight ratio. Matrix architecture is used for the drive amplifier that can drive N/sup 2/ actuator units using only 2N switches and is thus suitable for vast degree of freedom systems in terms of scalability. The prototype car seat uses 16 SMA actuator units, which are driven in a matrix architecture using 8 switches. The actuators are compactly housed under the car seat and the force and displacements are transmitted to the flexible seat surface through a novel routing scheme. We create a distributed lifting motion of the seat surface in order to stimulate the tissue. A complementary distributed sinking motion of the seat surface is created in order to provide pressure relief and ventilation.

When the human fingertip is pressed against a surface or bent, the hemodynamic state of the fingertip is altered due to mechanical interactions between the fingernail and bone. Normal force, shear force, and finger extension/flexion all result in different patterns of blood volume beneath the fingernail. This phenomenon has been exploited in order to detect finger forces and finger posture by creating a photoplethysmograph "fingernail sensor," which measures the two-dimensional pattern of blood volume beneath the fingernail. In this paper, a filter is designed to predict the normal force, lateral shear force, longitudinal shear force, and bending angle based on readings from the fingernail sensor. Linear, polynomial, and neural network models relating the bending angle and touch forces to optical sensor outputs are developed and tested. A method is developed to uniformly calibrate the predictor for each user. Calibration experiments are performed to train and validate the predictor for seven human subjects. Results show that on average, shear forces can be predicted with 0.5 N root mean square (rms) error, normal force with 1 N rms error, and posture angle with 10 degrees rms error. Applications and methods for improving performance are discussed.

A new type of actuator is presented where shape memory alloy (SMA) wires are embedded within artificial "blood vessels." Fluid flowing through vessels allows the SMA wires to be rapidly cooled by convection, resulting in greater bandwidth than ordinarily possible. Combinations of electric, fluidic, and thermal inputs can be used to control the contraction/extension of the SMA wires within the compliant vessels. These wet vascular SMA actuators can be used as robotic muscle or even embedded in a compliant rubber material to create an active vasculated robotic flesh. This paper begins by proposing a type of biomimetics where robots and machines are imbued with a vasculature or network of blood vessels. The concept of a wet SMA actuator is then introduced as an immediate application of a vasculated robot. A wet SMA actuator is then designed and implemented. Initial prototypes are 3 mm in diameter and are capable of 2% strain at 2 Hz by pulsing 2 A of current and 3 mL/s of water. Reservoirs of hot and cold water are also used to recycle the thermal energy and allow the actuators to exert static force with no electric current. Finally, multiple vascular SMA actuators are imbedded within a 4 mm thick rubber sheet to implement the active vasculated robotic flesh.

A new concept for an actuator array is presented where a vast number of actuators are contained within a small volume and are controlled in a scalable sense. This vast degree-of-freedom system utilizes an array of shape memory alloy wires embedded within a network of fluidic vessels. A matrix manifold and valve (MMV) system routes fluid from hot and cold reservoirs in order to control an array of N/sup 2/ SMA actuators using only 2N valves. A prototype MMV system containing a 4 by 4 array of wet SMA actuators is designed and implemented. Control strategies suitable for controlling MMV system that uses 2N valves to control N/sup 2/ actuators are presented. Two different protocols are presented and evaluated. The system error increases without bound under certain input conditions, since all the actuators cannot be activated at the same time. Simulations show that system error starts to go out of bound when the individual actuator is activated 25% of the time for the 4 by 4 arrays of the actuators.

When the human fingertip is pressed against a surface or bent, the hemodynamic state of the fingertip is altered due to mechanical interactions between the fingernail and bone. Normal force, shear force, and finger extension/flexion all result in different patterns of blood volume beneath the fingernail. This phenomenon has been exploited in order to detect finger forces and finger posture by creating a photoplethysmograph "fingernail sensor," which measures the two-dimensional pattern of blood volume beneath the fingernail. In this paper, a filter is designed to predict the normal force, lateral shear force, longitudinal shear force, and bending angle based on readings from the fingernail sensor. Linear, polynomial and neural network models are proposed as candidates for the predictor framework. A method is developed to uniformly calibrate the predictor for each user. Calibration experiments are performed to train and validate the predictor for seven human subjects. Results indicate that a simple linear model performs best, predicting shear forces with an average of 0.5 N r.m.s. error and normal force with an average of 1.0 N r.m.s. error.

A new method for measuring both the posture of human fingers and shear force at human fingertips is presented. Instead of using a traditional electronic glove with bending sensors embedded along the finger and shear sensors embedded beneath the fingertip, a wearable fingernail sensor is used to measure resulting changes in coloration of the fingernail. Since the sensor is mounted on the fingernail, finger posture can be measured without wearing a glove that hampers the motion of the fingers. Similarly, shear forces can be measured without covering the finger pad and obstructing the human's natural haptic sense. In the past, fingernail sensors with a one-dimensional array of photodetectors have been used to measure normal touching forces at the fingertip. A new fingernail sensor with a two-dimensional spatial array of photodetectors is constructed in order to measure finger posture and shear forces. Experiments are performed in order to measure the outputs of the photodetectors in response to changes in finger posture and applied normal and shear forces. Analysis is then performed to correlate the outputs to the inputs and provide a means of estimating normal forces, shear forces, and changes in finger posture.

A new type of touch sensor for detecting contact pressure at human fingertips is presented. This new sensor allows the fingers to directly contact the environment without obstructing the human's natural haptic senses. The finger touch force is detected by measuring changes in the coloration of the fingernail; hence the sensor is mounted on the fingernail rather than on the fingertip. Specifically, the fingernail is instrumented with miniature light emitting diodes (LEDs) and photodetectors in order to measure changes in the reflection intensity when the fingertip is pressed against a surface. The changes in intensity are then used to determine changes in the blood volume under the fingernail, a technique termed "reflectance photoplethysmography". A hemodynamic model is used to investigate the dynamics of the blood volume at two locations under the fingernail. A miniaturized prototype nail sensor is designed, built, and tested. The theoretical analysis is verified through experiment and simulation.

Advancements on a new type of touch sensor for detecting contact pressure at human fingertips are presented. A fingernail is instrumented with miniature LEDs and photodetectors in order to measure changes in the nail color when the fingertip is pressed against a surface. The fingernail sensor allows the fingers to directly contact the environment without obstructing the human's natural haptic senses. Reflectance photoplethysmography is used to measure the pattern of nail color, i.e., the blood content under the fingernail. Hemodynamic modeling, based on actual fingertip anatomy and physiology, is used to investigate the dynamics of the change in blood volume at multiple locations under the fingernail. An optical model is created to relate the blood volume to the light intensity measured by the photodetectors. The theoretical analysis is verified through model simulation and experimentation using a prototype fingernail sensor.

A novel human-machine interface using wearable sensors is presented. Fingernail sensors that detect touch forces at the fingertip are used as a means to acquire human intentions of pressing buttons and switches. The sensor is worn on the fingernail (and looks like an ornamental fingernail). Combined with a magnetic tracker detecting the position of the human hand, it identifies which switch the human wishes to push and when the switch has been pushed. This allows traditional physical switches, embedded in a wall, control panel, etc., to be replaced by "virtual switches" that contain no electrical circuits or mechanical parts, but are merely pictures showing the location of the switch. In the virtual switch system, the location and functionality of switches are determined by software and can thereby be changed flexibly depending on the progress of task performance, environmental conditions, and context. The virtual switch method is combined with a "hyper manual" that stores task-procedure and operational information on a computer. Monitoring the human task performance allows the digital hyper manual to better guide the human, detect errors, and provide a safer environment.

A new type of touch sensor for detecting contact pressure at human fingertips is presented. A fingernail is instrumented with miniature LEDs and photodetectors in order to measure changes in the nail color pattern when the fingertip is pressed against a surface. Unlike traditional electronic gloves, in which sensor pads are placed between the fingers and the environment surface, this new sensor allows the fingers to directly contact the environment without obstructing the human's natural haptic senses. The finger touch force is detected by measuring changes in the nail color, hence the sensor is mounted on the fingernail rather than on the fingertip. Photo-reflective plethysmography is used to measure the nail color. Haemodynamic modeling is used to investigate the dynamics of the change in blood volume under the fingernail. The model is simulated and then evaluated by comparison with the experimental dynamic response of the sensor. Applications to human-machine interaction are discussed.

A method for docking a vehicle with a fixture is developed and applied to a hybrid wheelchair/bed system. A powered wheelchair is docked to a bed portion and reconfigured to a flat stationary bed so that the bedridden person does not have to change seating when transferring between the chair and bed. A holonomic omnidirectional vehicle with a ball wheel mechanism is used for the wheelchair. The high maneuverability and holonomic nature of the vehicle allow the wheelchair to be docked precisely with a fixture. An instrumented bumper is developed to detect contacts with the fixture and ensure rider comfort. Using sensor information from the bumpers, the contact state of the vehicle is recognized at all times. Moreover, the orientation of the fixture is estimated from the contact information, and using a framework of discrete event control, the vehicle is aligned and guided towards the fixture despite large misalignments. A prototype system was designed and tested. The vehicle was successfully docked to the bed despite a small clearance ratio as well as large lateral and angular initial misalignments.

An approach to interactive control of human-robot systems using dual Petri nets is presented. A human and robot(s) work side by side by sharing the task goal and part of the process state. The human is instrumented with a type of data glove so that the robot can monitor the human state and coordinate its action with the human. First, the class of interactive tasks performed by a human and robot is described as a dual-agent, concurrent, event-driven system, and represented by two Petri nets interacting to each other. One Petri net represents the human side task process, while the other represents the robot side. Both sides proceed with their assigned tasks concurrently through state transitions within each Petri net. They observe the partner's state and coordinate each step of task performance as specified by the conditional state transition involved in the Petri nets. A distributed control system is directly derived from the dual Petri net representation. As an exemplary case study, a cable connection task is considered. The task is first decomposed into two subtasks, which are assigned to the human and the robot. A data glove worn by the human is developed for measuring the hand location as well as the force and position of each finger tip. A decentralized controller is built and a proof-of-concept experiment is described.

A hybrid wheelchair/bed system for bedridden persons is developed and tested. A powered wheelchair can be docked to a bed portion and reconfigured to a flat stationary bed so that the bedridden person does not have to change seating when transferring between the chair and bed. Moreover the wheelchair can also be docked to a toilet directly and automatically. A holonomic omnidirectional vehicle with a ball wheel mechanism is used for the wheelchair. The high maneuverability and holonomic nature of the vehicle allow the chair to be docked easily and precisely against a fixture. The wheelchair is equipped with teleconferencing facility so that the bedridden patient may communicate face-to-face with a distal caregiver, friends and relatives. This paper describes the basic concept of the reconfigurable holonomic omnidirectional mobile bed with unified seating (RHOMBUS). Issues on the mechanical design of the vehicle, chair, and bed are addressed, followed by control issues. A force-guided docking control method using force sensors embedded in the bumper of the vehicle is developed. A prototype system is designed and tested.