Robust fingertip force detection from fingernail image is a critical strategy that can be applied in many areas. However, prior research fixed many variables that influence the finger color change. This paper analyzes the effect of the finger joint on the force detection in order to deal with the constrained finger position setting. A force estimator method is designed: a model to predict the fingertip force from finger joints measured from 2D cameras and 3 rectangular markers in cooperation with the fingernail images are trained. Then the error caused by the color changes of the joint bending can be avoided. This strategy is a significant step forward from a finger force estimator that requires tedious finger joint setting. The approach is evaluated experimentally. The result shows that it increases the accuracy over 10% for the force in conditions of the finger joint free movement. The estimator is used to demonstrate lifting and replacing objects with various weights.

Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

We demonstrate a simple approach with which finger force can be measured from nail coloration. By automatically extracting features from nail images of a finger-mounted CCD camera, we can directly relate these images to the force measured by a force-torque sensor. The method automatically corrects orientation and illumination differences. Using Gaussian processes, we can relate prepro-cessed images of the finger nail to measured force and torque of the finger, allowing us to predict the finger force at a level of 95%–98% accuracy at force ranges up to 10N, and torques around 90% accuracy, based on training data gathered in 90s.

How can an animal learn from experience? How can it train sensors, such as the auditory or tactile system, based on other sensory input such as the visual system? Supervised spike-timing-dependent plasticity (supervised STDP) is a possible answer. Supervised STDP trains one modality using input from another one as "supervisor." Quite complex time-dependent relationships between the senses can be learned. Here we prove that under very general conditions, supervised STDP converges to a stable configuration of synaptic weights leading to a reconstruction of primary sensory input.