Manipulating complex objects is ubiquitous in our daily activities, such as donning a jacket or carrying a cup of coffee. However, such non-rigid objects easily become unstable: When carrying a cup of coffee, the coffee could slosh unpredictably and spill out of the cup. It remains unclear what motor control strategies ensure stability, especially when the physical properties of the object, like the amount of liquid in the cup, are unknown. The task of transporting a 'cup of coffee' was simplified to transporting a virtual cup with a sliding ball inside, modeled as a cart-pendulum system. Participants were instructed to 'jiggle' the cup in one dimension to prepare the cup and ball states for the ensuing continuous rhythmic movement. To introduce uncertainty regarding the object's properties, the pendulum's length was manipulated either to 1) change randomly from trial to trial, or to 2) remain constant across trials. We measured the ball's angle at the end of preparation and the cup's oscillation frequency during the rhythmic portion of the trial. Grip force on the robot handle served as proxy for mechanical impedance of the arm. The results supported three predictions: 1) When dynamic uncertainty was high, object preparation was important to stabilize the transient dynamics; stability increased during preparation and humans prepared longer when the dynamics was uncertain. 2) Humans maximized dynamic stability by flexibly covarying system initialization and cup frequency; dynamic stability matched participant behavior better than magnitude or smoothness of force. 3) Humans increased their arm impedance to accommodate uncertain dynamics, while the net force applied on the cup remained the same. Feedforward simulations using an impedance controller and stochastic open-loop optimal control corroborated these findings, further revealing that participants' selection of preparation and interaction frequencies also minimized mechanical impedance. In sum, humans used preparation and interaction strategies to optimize the mechanical impedance and dynamic stability of the hand-object interactions. These results may inform approaches in robotic control and rehabilitation.

Humans are skillful at manipulating objects that possess nonlinear underactuated dynamics, such as clothes or containers filled with liquids. Several studies suggested that humans implement a predictive model-based strategy to control such objects. However, these studies only considered unconstrained reaching without any object involved or, at most, linear mass-spring systems with relatively simple dynamics. It is not clear what internal model humans develop of more complex objects, and what level of granularity is represented. To answer these questions, this study examined a task where participants physically interacted with a nonlinear underactuated system mimicking a cup of sloshing coffee: a cup with a ball rolling inside. The cup and ball were simulated in a virtual environment and subjects interacted with the system via a haptic robotic interface. Participants were instructed to move the system and arrive at a target region with both cup and ball at rest, 'zeroing out' residual oscillations of the ball. This challenging task affords a solution known as 'input shaping', whereby a series of pulses moves the dynamic object to the target leaving no residual oscillations. Since the timing and amplitude of these pulses depend on the controller's internal model of the object, input shaping served as a tool to identify the subjects' internal representation of the cup-and-ball. Five simulations with different internal models were compared against the human data. Results showed that the features in the data were correctly predicted by a simple internal model that represented the cup-and-ball as a single rigid mass coupled to the hand impedance. These findings provide evidence that humans use simplified internal models along with mechanical impedance to manipulate complex objects.

An important component for the effective collaboration of humans with robots is the compatibility of their movements, especially when humans physically collaborate with a robot partner. Following previous findings that humans interact more seamlessly with a robot that moves with human-like or biological velocity profiles, this study examined whether humans can adapt to a robot that violates human signatures. The specific focus was on the role of extensive practice and real-time augmented feedback. Six groups of participants physically tracked a robot tracing an ellipse with profiles where velocity scaled with the curvature of the path in biological and non-biological ways, while instructed to minimize the interaction force with the robot. Three of the 6 groups received real-time visual feedback about their force error. Results showed that with 3 daily practice sessions, when given feedback about their force errors, humans could decrease their interaction forces when the robot's trajectory violated human-like velocity patterns. Conversely, when augmented feedback was not provided, there were no improvements despite this extensive practice. The biological profile showed no improvements, even with feedback, indicating that the (non-zero) force had already reached a floor level. These findings highlight the importance of biological robot trajectories and augmented feedback to guide humans to adapt to non-biological movements in physical human-robot interaction. These results have implications on various fields of robotics, such as surgical applications and collaborative robots for industry.

Tactile sensing has been increasingly utilized in robot control of unknown objects to infer physical properties and optimize manipulation. However, there is limited understanding about the contribution of different sensory modalities during interactive perception in complex interaction both in robots and in humans. This study investigated the effect of visual and haptic information on humans' exploratory interactions with a ‘cup of coffee’, an object with nonlinear internal dynamics. Subjects were instructed to rhythmically transport a virtual cup with a rolling ball inside between two targets at a specified frequency, using a robotic interface. The cup and targets were displayed on a screen, and force feedback from the cup-and-ball dynamics was provided via the robotic manipulandum. Subjects were encouraged to explore and prepare the dynamics by “shaking” the cup-and-ball system to find the best initial conditions prior to the task. Two groups of subjects received the full haptic feedback about the cup-and-ball movement during the task; however, for one group the ball movement was visually occluded. Visual information about the ball movement had two distinctive effects on the performance: it reduced preparation time needed to understand the dynamics and, importantly, it led to simpler, more linear input-output interactions between hand and object. The results highlight how visual and haptic information regarding nonlinear internal dynamics have distinct roles for the interactive perception of complex objects.

Humans are adept at a wide variety of motor skills, including the handling of complex objects and using tools. Advances to understand the control of voluntary goal-directed movements have focused on simple behaviors such as reaching, uncoupled to any additional object dynamics. Under these simplified conditions, basic elements of motor control, such as the roles of body mechanics, objective functions, and sensory feedback, have been characterized. However, these elements have mostly been examined in isolation, and the interactions between these elements have received less attention. This study examined a task with internal dynamics, inspired by the daily skill of transporting a cup of coffee, with additional expected or unexpected perturbations to probe the structure of the controller. Using optimal feedback control (OFC) as the basis, it proved necessary to endow the model of the body with mechanical impedance to generate the kinematic features observed in the human experimental data. The addition of mechanical impedance revealed that simulated movements were no longer sensitively dependent on the objective function, a highly debated cornerstone of optimal control. Further, feedforward replay of the control inputs was similarly successful in coping with perturbations as when feedback, or sensory information, was included. These findings suggest that when the control model incorporates a representation of the mechanical properties of the limb, that is, embodies its dynamics, the specific objective function and sensory feedback become less critical, and complex interactions with dynamic objects can be successfully managed.

Research in joint action focuses on the psychological, neurological, and physical mechanisms by which humans collabo-rate with other agents, and overlaps with several domains related to human-robot interaction. The development of artificial systems that can support or emulate the requisite aspects of joint action could lead to improved human-robot team performance as well as improvements in subjective metrics (e.g., trust). This workshop highlights theoretical and technical considerations about human-robot joint action and real-time adaptation, with a particular focus on socio-motor entrainment, showing how the emulation of psychological mechanisms (e.g., emotion, intention signaling, mirroring) can lead to improved performance. We will invite speakers with backgrounds in robotics, neuroscience and psychol-ogy, as well as speakers with a focus in adjacent works, such as in human-robot coordinated dance, alignment, or synchronization. We will call for papers that utilize the theory of joint-action in an interactive human-robot context. We will also call for position papers on the application of the theory of joint action to robotics, with a heavy focus on psychological mechanisms that could potentially be emulated or adapted to a human-robot context. Participants will have the opportunity to brainstorm considerations and techniques that would be applicable to joint action inspired works through breakout sessions with the aim to lead to new and improved collaborations across fields.

Modern computer algorithms easily beat world champions in chess or Go, but state-of-the-art robots are still outperformed by two-year-old’s in manipulating the pieces, let alone interacting with more complex objects. This work studied human behavior when moving an underactuated object, a cup with a ball rolling inside creating internal dynamics like sloshing coffee in a cup. The objective was to develop a control model that could replicate human behavior. Human movement data were collected for transporting this cup-and-ball system, both with and without external perturbations. The existing models in the human control literature, including maximum smoothness, optimal feedback control with minimum effort, and dynamic primitives with impedance were revisited for this challenging task. As these control models were primarily developed for unconstrained reaching movements, they could replicate human trajectories when transporting a rigid object. However, they fell short when the object introduced complex interaction forces due to its internal dynamics. Therefore, this study extended the framework of dynamic primitives and used an optimal controller to generate a maximally smooth zero-force trajectory for the impedance operator when interacting with perturbations from the object or the environment. Given the challenges that robot control still faces when interacting with complex objects, these findings may inform the development of bio-inspired controllers for robotic manipulation.

A prominent challenge in the field of robotics is manipulation of flexible objects. One major factor that makes this task difficult is the complex dynamics emerging from its high-dimensional structure. This argues against the use of popular optimization-based approaches, which scale poorly with system dimension (the "curse of dimensionality"). Nevertheless, almost indifferent to this complexity, humans handle it on a daily basis, without any apparent difficulty.Inspired by human motor control, we propose that encoding movements based on dynamic primitives can simplify the task of manipulating flexible objects and provides a way around the curse of dimensionality. Using an extreme example — manipulating a whip — we tested in simulation whether targets at various locations could be reached with a whip by using a controller based on dynamic primitives. Regardless of the target location, this approach successfully managed the complexity of a 54 degree-of-freedom system (yielding a 108-dimensional state-space representation) and identified an upper-limb movement that achieved the task. This approach did not require a detailed model of the whip, which thereby significantly simplified the computational complexity of the control task. We believe that this approach may facilitate robotic manipulation of flexible materials, and in general afford a simplified way to control dynamically complex objects.

Manipulation of flexible objects is one of the major challenges in robotics as the nonlinear dynamics of the high-dimensional object structure makes it difficult to apply current control methods. A previous simulation study showed that control with few pre-structured joint trajectories coupled with joint impedance (dynamic primitives) could control a 25-dimensional whip to hit a target. This was possible even though the impedance values were constant. This paper explores whether time-varying impedance throughout the movement may further enhance performance. We present an online impedance adaptation (OIA) controller that modulates the joint impedances of a two-joint actuator in real time for the same task. Results showed that the OIA control method increased the speed of optimization and resulted in smaller deviation from the zero-torque joint trajectories compared to the controller with constant joint impedances. This novel way to modulate both motion and impedance of a manipulator may facilitate the control of flexible objects with significant dynamics.

Humans dexterously interact with a variety of objects, including those with complex internal dynamics. Even in the simple action of carrying a cup of coffee, the hand not only applies a force to the cup, but also indirectly to the liquid, which elicits complex reaction forces back on the hand. Due to underactuation and nonlinearity, the object's dynamic response to an action sensitively depends on its initial state and can display unpredictable, even chaotic behavior. With the overarching hypothesis that subjects strive for predictable object-hand interactions, this study examined how subjects explored and prepared the dynamics of an object for subsequent execution of the target task. We specifically hypothesized that subjects find initial conditions that shorten the transients prior to reaching a stable and predictable steady state. Reaching a predictable steady state is desirable as it may reduce the need for online error corrections and facilitate feed forward control. Alternative hypotheses were that subjects seek to reduce effort, increase smoothness, and reduce risk of failure. Motivated by the task of 'carrying a cup of coffee', a simplified cup-and-ball model was implemented in a virtual environment. Human subjects interacted with this virtual object via a robotic manipulandum that provided force feedback. Subjects were encouraged to first explore and prepare the cup-and-ball before initiating a rhythmic movement at a specified frequency between two targets without losing the ball. Consistent with the hypotheses, subjects increased the predictability of interaction forces between hand and object and converged to a set of initial conditions followed by significantly decreased transients. The three alternative hypotheses were not supported. Surprisingly, the subjects' strategy was more effortful and less smooth, unlike the observed behavior in simple reaching movements. Inverse dynamics of the cup-and-ball system and forward simulations with an impedance controller successfully described subjects' behavior. The initial conditions chosen by the subjects in the experiment matched those that produced the most predictable interactions in simulation. These results present first support for the hypothesis that humans prepare the object to minimize transients and increase stability and, overall, the predictability of hand-object interactions.

Relatively little work in human and robot control has examined the control of underactuated objects with internal dynamics, such as transporting a cup of coffee, a task that presents little problems for humans. This study examined how humans move a `cup of coffee' with a view to identify principles that may be useful for robot control. The specific focus was on how humans choose initial conditions to safely reach a steady state. We hypothesized that subjects choose initial conditions that minimized the transient duration to reach the steady state faster, as it presented more predictable dynamics. In the experiment, the cup of coffee was reduced to a 2-D cup with a sliding ball inside which was simulated in a virtual environment. Human subjects interacted with this virtual object via a robotic manipulandum that provided haptic feedback. Participants moved the cup between two targets without losing the ball; they were instructed to explore different initial conditions before initiating the continuous interaction. Results showed that subjects converged to a small set of initial conditions that decreased their transient durations and achieved a predictable steady state faster. Simulations with a simple feedforward controller and inverse dynamics calculations confirmed that these initial conditions indeed led to shorter transients and less complex interaction forces. These results may inform robot control of objects with internal dynamics where the effects of initial conditions need further investigation.

This study examined strategies humans chose to manipulate an object with complex (nonlinear, underactuated) dynamics, such as liquid sloshing in a cup of coffee. The problem was simplified to the well-known cart-and-pendulum system moving on a horizontal line. This model was implemented in a virtual environment and human subjects manipulated the object via a robotic manipulandum. The task was to maneuver the system from rest to arrive at a target position such that no residual oscillations of the pendulum bob remained. Our goal was to test whether humans simplified control by employing dynamic primitives, specifically submovements. Experimental velocity profiles of the human movements were compared to those predicted by three different control models. Two models used continuous optimization-based control, the third control model was based on Input Shaping. Input Shaping is a method for controlling flexible objects by convolving a motion profile with impulses of appropriate amplitude and timing. To evaluate whether humans used Input Shaping, we decomposed the velocity profiles recorded from humans into submovements, as proxies for the convolved impulses. Comparing the motion profiles from the 3 models with the experimentally measured human profiles showed superior performance of the Input Shaping model. These initial results are consistent with our hypothesis that combining dynamic primitives, submovements, is a competent description of human performance and may provide a simpler alternative to computationally complex optimization-based methods of robot control.

This study examines human control of physical interaction with objects that exhibit complex (nonlinear, chaotic, underactuated) dynamics. We hypothesized that humans exploited stability properties of the human-object interaction. Using a simplified 2D model for carrying a "cup of coffee", we developed a virtual implementation to identify human control strategies. Transporting a cup of coffee was modeled as a cart with a suspended pendulum, where humans moved the cart on a horizontal line via a robotic manipulandum. The specific task was to transport the cart-pendulum system to a target, as fast as possible, while accommodating assistive and resistive perturbations. To assess trajectory stability, we applied contraction analysis. We showed that when the perturbation was assistive, humans absorbed the perturbation by controlling cart trajectories into a contraction region prior to the perturbation. When the perturbation was resistive, subjects passed through a contraction region following the perturbation. Entering a contraction region stabilizes performance and makes the dynamics more predictable. This human control strategy could inspire more robust control strategies for physical interaction in robots.

Throwing is a uniquely human skill that requires a high degree of coordination to successfully hit a target. Timing of ball release appears crucial as previous studies report required timing accuracies as short as 1-2ms, which however appear physiologically challenging. This study mathematically and experimentally demonstrates that humans can overcome these seemingly stringent timing requirements by shaping their hand trajectories to create extended timing windows, where ball releases achieve target hits despite temporal imprecision. Subjects practiced four task variations in a virtual environment, each with a distinct geometry of the solution space and different demands for timing. Model-based analyses of arm trajectories revealed that subjects first decreased timing error, followed by lengthening timing windows in their hand trajectories. This pattern was invariant across solution spaces, except for a control case. Hence, the exquisite skill that humans evolved for throwing is achieved by developing strategies that are less sensitive to temporal variability arising from neuromotor noise. This analysis also provides an explanation why coaches emphasize the "follow-through" in many ball sports.

Variability in motor performance results from the interplay of error correction and neuromotor noise. This study examined whether visual amplification of error, previously shown to improve performance, affects not only error correction, but also neuromotor noise, typically regarded as inaccessible to intervention. Seven groups of healthy individuals, with six participants in each group, practiced a virtual throwing task for three days until reaching a performance plateau. Over three more days of practice, six of the groups received different magnitudes of visual error amplification; three of these groups also had noise added. An additional control group was not subjected to any manipulations for all six practice days. The results showed that the control group did not improve further after the first three practice days, but the error amplification groups continued to decrease their error under the manipulations. Analysis of the temporal structure of participants' corrective actions based on stochastic learning models revealed that these performance gains were attained by reducing neuromotor noise and, to a considerably lesser degree, by increasing the size of corrective actions. Based on these results, error amplification presents a promising intervention to improve motor function by decreasing neuromotor noise after performance has reached an asymptote. These results are relevant for patients with neurological disorders and the elderly. More fundamentally, these results suggest that neuromotor noise may be accessible to practice interventions.

The study of object manipulation has been largely confined to discrete tasks, where accuracy, mechanical effort, or smoothness were examined to explain subjects' preferred movements. This study investigated a rhythmic manipulation task, which involved continuous interaction with a nonlinear object that led to unpredictable object behavior. Using a simplified virtual version of the task of carrying a cup of coffee, we studied how this unpredictable object behavior affected the selected strategies. The experiment was conducted in a virtual set-up, where subjects moved a cup with a ball inside, modeled by cart-and-pendulum dynamics. Inverse dynamics calculations of the system showed that performing the task with different amplitudes and relative phases required different force profiles and rendered the object's dynamics with different degrees of predictability (quantified by Mutual Information between the applied force and the cup kinematics and its sensitivity). Subjects (n = 8) oscillated the virtual cup between two targets via a robotic manipulandum, paced by a metronome at 1 Hz for 50 trials, each lasting 45 s. They were free to choose their movement amplitude and relative phase between the ball and cup. Experimental results showed that subjects increased their movement amplitudes, which rendered the interactions with the object more predictable and with lower sensitivity to the execution variables. These solutions were associated with higher average exerted force and lower object smoothness, contradicting common expectations from studies on discrete object manipulation and unrestrained movements. Instead, the findings showed that humans selected strategies with higher predictability of interaction dynamics. This finding expressed that humans seek movement strategies where force and kinematics synchronize to repeatable patterns that may require less sensorimotor information processing.

In motor tasks with redundancy neuromotor noise can lead to variations in execution while achieving relative invariance in the result. The present study examined whether humans find solutions that are tolerant to intrinsic noise. Using a throwing task in a virtual set-up where an infinite set of angle and velocity combinations at ball release yield throwing accuracy, our computational approach permitted quantitative predictions about solution strategies that are tolerant to noise. Based on a mathematical model of the task expected results were computed and provided predictions about error-tolerant strategies (Hypothesis 1). As strategies can take on a large range of velocities, a second hypothesis was that subjects select strategies that minimize velocity at release to avoid costs associated with signal- or velocity-dependent noise or higher energy demands (Hypothesis 2). Two experiments with different target constellations tested these two hypotheses. Results of Experiment 1 showed that subjects chose solutions with high error-tolerance, although these solutions also had relatively low velocity. These two benefits seemed to outweigh that for many subjects these solutions were close to a high-penalty area, i.e. they were risky. Experiment 2 dissociated the two hypotheses. Results showed that individuals were consistent with Hypothesis 1 although their solutions were distributed over a range of velocities. Additional analyses revealed that a velocity-dependent increase in variability was absent, probably due to the presence of a solution manifold that channeled variability in a task-specific manner. Hence, the general acceptance of signal-dependent noise may need some qualification. These findings have significance for the fundamental understanding of how the central nervous system deals with its inherent neuromotor noise.

Analysis of motor performance variability in tasks with redundancy affords insight about synergies underlying central nervous system (CNS) control. Preferential distribution of variability in ways that minimally affect task performance suggests sophisticated neural control. Unfortunately, in the analysis of variability the choice of coordinates used to represent multi-dimensional data may profoundly affect analysis, introducing an arbitrariness which compromises its conclusions. This paper assesses the influence of coordinates. Methods based on analyzing a covariance matrix are fundamentally dependent on an investigator's choices. Two reasons are identified: using anisotropy of a covariance matrix as evidence of preferential distribution of variability; and using orthogonality to quantify relevance of variability to task performance. Both are exquisitely sensitive to coordinates. Unless coordinates are known a priori, these methods do not support unambiguous inferences about CNS control. An alternative method uses a two-level approach where variability in task execution (expressed in one coordinate frame) is mapped by a function to its result (expressed in another coordinate frame). An analysis of variability in execution using this function to quantify performance at the level of results offers substantially less sensitivity to coordinates than analysis of a covariance matrix of execution variables. This is an initial step towards developing coordinate-invariant analysis methods for movement neuroscience.

Current research on discrete and rhythmic movements differs in both experimental procedures and theory, despite the ubiquitous overlap between discrete and rhythmic components in everyday behaviors. Models of rhythmic movements usually use oscillatory systems mimicking central pattern generators (CPGs). In contrast, models of discrete movements often employ optimization principles, thereby reflecting the higher-level cortical resources involved in the generation of such movements. This letter proposes a unified model for the generation of both rhythmic and discrete movements. We show that a physiologically motivated model of a CPG can not only generate simple rhythmic movements with only a small set of parameters, but can also produce discrete movements if the CPG is fed with an exponentially decaying phasic input. We further show that a particular coupling between two of these units can reproduce main findings on in-phase and antiphase stability. Finally, we propose an integrated model of combined rhythmic and discrete movements for the two hands. These movement classes are sequentially addressed in this letter with increasing model complexity. The model variations are discussed in relation to the degree of recruitment of the higher-level cortical resources, necessary for such movements.

The question of how best to model rhythmic movements at self-selected amplitude-frequency combinations, and their variability, is a long-standing issue. This study presents a systematic analysis of a coupled oscillator system that has successfully accounted for the experimental result that humans' preferred oscillation frequencies closely correspond to the linear resonance frequencies of the biomechanical limb systems, a phenomenon known as resonance tuning or frequency scaling. The dynamics of the coupled oscillator model is explored by numerical integration in different areas of its parameter space, where a period doubling route to chaotic dynamics is discovered. It is shown that even in the regions of the parameter space with chaotic solutions, the model still effectively scales to the biomechanical oscillator's natural frequency. Hence, there is a solution providing for frequency scaling in the presence of chaotic variability. The implications of these results for interpreting variability as fundamentally stochastic or chaotic are discussed.