Tactile sensing has been increasingly utilized in robot control of unknown objects to infer physical properties and optimize manipulation. However, there is limited understanding about the contribution of different sensory modalities during interactive perception in complex interaction both in robots and in humans. This study investigated the effect of visual and haptic information on humans' exploratory interactions with a ‘cup of coffee’, an object with nonlinear internal dynamics. Subjects were instructed to rhythmically transport a virtual cup with a rolling ball inside between two targets at a specified frequency, using a robotic interface. The cup and targets were displayed on a screen, and force feedback from the cup-and-ball dynamics was provided via the robotic manipulandum. Subjects were encouraged to explore and prepare the dynamics by “shaking” the cup-and-ball system to find the best initial conditions prior to the task. Two groups of subjects received the full haptic feedback about the cup-and-ball movement during the task; however, for one group the ball movement was visually occluded. Visual information about the ball movement had two distinctive effects on the performance: it reduced preparation time needed to understand the dynamics and, importantly, it led to simpler, more linear input-output interactions between hand and object. The results highlight how visual and haptic information regarding nonlinear internal dynamics have distinct roles for the interactive perception of complex objects.

Humans are adept at a wide variety of motor skills, including the handling of complex objects and using tools. Advances to understand the control of voluntary goal-directed movements have focused on simple behaviors such as reaching, uncoupled to any additional object dynamics. Under these simplified conditions, basic elements of motor control, such as the roles of body mechanics, objective functions, and sensory feedback, have been characterized. However, these elements have mostly been examined in isolation, and the interactions between these elements have received less attention. This study examined a task with internal dynamics, inspired by the daily skill of transporting a cup of coffee, with additional expected or unexpected perturbations to probe the structure of the controller. Using optimal feedback control (OFC) as the basis, it proved necessary to endow the model of the body with mechanical impedance to generate the kinematic features observed in the human experimental data. The addition of mechanical impedance revealed that simulated movements were no longer sensitively dependent on the objective function, a highly debated cornerstone of optimal control. Further, feedforward replay of the control inputs was similarly successful in coping with perturbations as when feedback, or sensory information, was included. These findings suggest that when the control model incorporates a representation of the mechanical properties of the limb, that is, embodies its dynamics, the specific objective function and sensory feedback become less critical, and complex interactions with dynamic objects can be successfully managed.

Modern computer algorithms easily beat world champions in chess or Go, but state-of-the-art robots are still outperformed by two-year-old’s in manipulating the pieces, let alone interacting with more complex objects. This work studied human behavior when moving an underactuated object, a cup with a ball rolling inside creating internal dynamics like sloshing coffee in a cup. The objective was to develop a control model that could replicate human behavior. Human movement data were collected for transporting this cup-and-ball system, both with and without external perturbations. The existing models in the human control literature, including maximum smoothness, optimal feedback control with minimum effort, and dynamic primitives with impedance were revisited for this challenging task. As these control models were primarily developed for unconstrained reaching movements, they could replicate human trajectories when transporting a rigid object. However, they fell short when the object introduced complex interaction forces due to its internal dynamics. Therefore, this study extended the framework of dynamic primitives and used an optimal controller to generate a maximally smooth zero-force trajectory for the impedance operator when interacting with perturbations from the object or the environment. Given the challenges that robot control still faces when interacting with complex objects, these findings may inform the development of bio-inspired controllers for robotic manipulation.

Stroke rehabilitation technologies have focused on reducing treatment cost while improving effectiveness. Rehabilitation robots are generally developed for home and clinical usage to: 1) deliver repetitive practice to post-stroke patients, 2) minimize therapist interventions, and 3) increase the number of patients per therapist, thereby decreasing the associated cost. The control of rehabilitation robots is often limited to black-or gray-box approaches; thus, safety issues regarding the human-robot interaction are not easily considered. To overcome this issue, controllers working with physics-based models gain more importance. In this study, we have developed an efficient two dimensional (2D) human-robot interaction model to implement a model-based controller on a planar end-effector-type rehabilitation robot. The developed model was used within a nonlinear model predictive control (NMPC) structure to control the rehabilitation robot. The GPOPS-II optimal control package was used to implement the proposed NMPC structure. The controller performance was evaluated by simulating the human-robot rehabilitation system, modeled in MapleSim®. In this system, a musculoskeletal model of the arm interacting with the robot is used to predict movement and muscle activation patterns, which are used by the controller to provide optimal assistance to the patient. In simulations, the controller achieved desired performance and predicted muscular activities of the dysfunctional subject with a good accuracy. In our future work, a structure exploiting the NMPC framework will be developed for the real-time control of the rehabilitation robot.