Tactile sensing has been increasingly utilized in robot control of unknown objects to infer physical properties and optimize manipulation. However, there is limited understanding about the contribution of different sensory modalities during interactive perception in complex interaction both in robots and in humans. This study investigated the effect of visual and haptic information on humans' exploratory interactions with a ‘cup of coffee’, an object with nonlinear internal dynamics. Subjects were instructed to rhythmically transport a virtual cup with a rolling ball inside between two targets at a specified frequency, using a robotic interface. The cup and targets were displayed on a screen, and force feedback from the cup-and-ball dynamics was provided via the robotic manipulandum. Subjects were encouraged to explore and prepare the dynamics by “shaking” the cup-and-ball system to find the best initial conditions prior to the task. Two groups of subjects received the full haptic feedback about the cup-and-ball movement during the task; however, for one group the ball movement was visually occluded. Visual information about the ball movement had two distinctive effects on the performance: it reduced preparation time needed to understand the dynamics and, importantly, it led to simpler, more linear input-output interactions between hand and object. The results highlight how visual and haptic information regarding nonlinear internal dynamics have distinct roles for the interactive perception of complex objects.

Humans are adept at a wide variety of motor skills, including the handling of complex objects and using tools. Advances to understand the control of voluntary goal-directed movements have focused on simple behaviors such as reaching, uncoupled to any additional object dynamics. Under these simplified conditions, basic elements of motor control, such as the roles of body mechanics, objective functions, and sensory feedback, have been characterized. However, these elements have mostly been examined in isolation, and the interactions between these elements have received less attention. This study examined a task with internal dynamics, inspired by the daily skill of transporting a cup of coffee, with additional expected or unexpected perturbations to probe the structure of the controller. Using optimal feedback control (OFC) as the basis, it proved necessary to endow the model of the body with mechanical impedance to generate the kinematic features observed in the human experimental data. The addition of mechanical impedance revealed that simulated movements were no longer sensitively dependent on the objective function, a highly debated cornerstone of optimal control. Further, feedforward replay of the control inputs was similarly successful in coping with perturbations as when feedback, or sensory information, was included. These findings suggest that when the control model incorporates a representation of the mechanical properties of the limb, that is, embodies its dynamics, the specific objective function and sensory feedback become less critical, and complex interactions with dynamic objects can be successfully managed.

Modern computer algorithms easily beat world champions in chess or Go, but state-of-the-art robots are still outperformed by two-year-old’s in manipulating the pieces, let alone interacting with more complex objects. This work studied human behavior when moving an underactuated object, a cup with a ball rolling inside creating internal dynamics like sloshing coffee in a cup. The objective was to develop a control model that could replicate human behavior. Human movement data were collected for transporting this cup-and-ball system, both with and without external perturbations. The existing models in the human control literature, including maximum smoothness, optimal feedback control with minimum effort, and dynamic primitives with impedance were revisited for this challenging task. As these control models were primarily developed for unconstrained reaching movements, they could replicate human trajectories when transporting a rigid object. However, they fell short when the object introduced complex interaction forces due to its internal dynamics. Therefore, this study extended the framework of dynamic primitives and used an optimal controller to generate a maximally smooth zero-force trajectory for the impedance operator when interacting with perturbations from the object or the environment. Given the challenges that robot control still faces when interacting with complex objects, these findings may inform the development of bio-inspired controllers for robotic manipulation.

Humans dexterously interact with a variety of objects, including those with complex internal dynamics. Even in the simple action of carrying a cup of coffee, the hand not only applies a force to the cup, but also indirectly to the liquid, which elicits complex reaction forces back on the hand. Due to underactuation and nonlinearity, the object's dynamic response to an action sensitively depends on its initial state and can display unpredictable, even chaotic behavior. With the overarching hypothesis that subjects strive for predictable object-hand interactions, this study examined how subjects explored and prepared the dynamics of an object for subsequent execution of the target task. We specifically hypothesized that subjects find initial conditions that shorten the transients prior to reaching a stable and predictable steady state. Reaching a predictable steady state is desirable as it may reduce the need for online error corrections and facilitate feed forward control. Alternative hypotheses were that subjects seek to reduce effort, increase smoothness, and reduce risk of failure. Motivated by the task of 'carrying a cup of coffee', a simplified cup-and-ball model was implemented in a virtual environment. Human subjects interacted with this virtual object via a robotic manipulandum that provided force feedback. Subjects were encouraged to first explore and prepare the cup-and-ball before initiating a rhythmic movement at a specified frequency between two targets without losing the ball. Consistent with the hypotheses, subjects increased the predictability of interaction forces between hand and object and converged to a set of initial conditions followed by significantly decreased transients. The three alternative hypotheses were not supported. Surprisingly, the subjects' strategy was more effortful and less smooth, unlike the observed behavior in simple reaching movements. Inverse dynamics of the cup-and-ball system and forward simulations with an impedance controller successfully described subjects' behavior. The initial conditions chosen by the subjects in the experiment matched those that produced the most predictable interactions in simulation. These results present first support for the hypothesis that humans prepare the object to minimize transients and increase stability and, overall, the predictability of hand-object interactions.

Relatively little work in human and robot control has examined the control of underactuated objects with internal dynamics, such as transporting a cup of coffee, a task that presents little problems for humans. This study examined how humans move a `cup of coffee' with a view to identify principles that may be useful for robot control. The specific focus was on how humans choose initial conditions to safely reach a steady state. We hypothesized that subjects choose initial conditions that minimized the transient duration to reach the steady state faster, as it presented more predictable dynamics. In the experiment, the cup of coffee was reduced to a 2-D cup with a sliding ball inside which was simulated in a virtual environment. Human subjects interacted with this virtual object via a robotic manipulandum that provided haptic feedback. Participants moved the cup between two targets without losing the ball; they were instructed to explore different initial conditions before initiating the continuous interaction. Results showed that subjects converged to a small set of initial conditions that decreased their transient durations and achieved a predictable steady state faster. Simulations with a simple feedforward controller and inverse dynamics calculations confirmed that these initial conditions indeed led to shorter transients and less complex interaction forces. These results may inform robot control of objects with internal dynamics where the effects of initial conditions need further investigation.