Soft robots can revolutionize several applications with high demands on dexterity and safety. When operating these systems, real-time estimation and control require fast and accurate models. However, prediction with first-principles (FP) models is slow, and learned black-box models have poor generalizability. Physics-informed machine learning offers excellent advantages here, but it is currently limited to simple, often simulated systems without considering changes after training. We propose physics-informed neural networks (PINNs) for articulated soft robots (ASRs) with a focus on data efficiency. The amount of expensive real-world training data is reduced to a minimum — one dataset in one system domain. Two hours of data in different domains are used for a comparison against two gold-standard approaches: In contrast to a recurrent neural network, the PINN provides a high generalizability. The prediction speed of an accurate FP model is exceeded with the PINN by up to a factor of 467 at slightly reduced accuracy. This enables nonlinear model predictive control (MPC) of a pneumatic ASR. Accurate position tracking with the MPC running at 47 Hz is achieved in six dynamic experiments.

Fast and safe motion is crucial for the successful deployment of physically interactive robots. Parallel robots (PRs) offer the potential for higher speeds while maintaining the same energy limits due to their low moving masses. However, they require methods for contact detection and reaction while avoiding singularities and self-collisions. We address this issue and present SafePR - a unified approach for the detection and localization, including the distinction between collision and clamping to perform a reaction that is safe for humans and feasible for PRs. Our approach uses information from the encoders and motor currents to estimate forces via a generalized-momentum observer. Neural networks and particle filters classify and localize the contacts. We introduce reactions with redundancy resolution to avoid self-collisions and type-II singularities. Our approach detected and terminated 72 real-world collision and clamping contacts with end-effector speeds of up to 1.5m/s, each within 25-275ms. The forces were below the thresholds from ISO/TS 15066. By using built-in sensors, SafePR enables safe interaction with already assembled PRs without the need for new hardware components.

This paper investigates a novel approach for continuous gravity compensation in robotic systems using a Lagrangian Neural Network (LNN)-based Computed Torque Controller (CTC). Specifically, we use LNNs to model the system’s dynamics and adaptively improve the performance of the CTC. Unlike traditional methods relying on predefined models or extensive offline training, this approach enables the controller to learn and adapt in real-time, without prior model knowledge, and within seconds during operation. The proposed approach focuses on energy-based representations and uses Lagrangian mechanics to ensure that the learned dynamics are physically interpretable and reliable, reducing the need for extensive training datasets. The LNN-CTC utilizes an online learning mechanism to continuously update the LNN-based on real-time data, ensuring accurate gravity compensation without prior model knowledge. Real-world experiments on a humanoid robotic arm with 4 degrees of freedom show that the LNN-based online learning CTC effectively compensates for gravity and adapts to changes in mass within 25s. We compare the proposed controller to a conventional model-based controller that relies on precise parameter knowledge and demonstrate that the LNN-CTC achieves similar trajectory tracking accuracy with significantly less data, no prior knowledge, and rapid adaptation to dynamic changes, such as added payloads without requiring parameter identification. This work contributes an adaptive real-time control framework that compensates for gravity. It overcomes high modeling efforts and large datasets, showing promise for scalability and generalization in autonomous systems and advanced robotics in uncertain environments.

Eye movements routinely induce motion streaks as they shift visual projections across the retina at high speeds. To investigate the visual consequences of intra-saccadic motion streaks, we co-registered eye tracking and EEG while gaze-contingently shifting target objects during saccades, presenting either continuous, 'streaky' or apparent, step-like motion in four directions. We found significant reductions of secondary saccade latency, as well as improved decoding of the post-saccadic target location from the EEG signal when motion streaks were available. These signals arose as early as 50 ms after saccade offset and had a clear occipital topography. Using a physiologically plausible visual processing model, we provide evidence that the target's motion trajectory is coded in orientation-selective channels and that speed of gaze correction was linked to the visual dynamics arising from the combination of saccadic and target motion, providing a parsimonious explanation of the behavioral benefits of intra-saccadic motion streaks.

This work presents a body velocity control strategy for quadruped robots. Such control typically requires accurate kinematic and dynamic model knowledge, which is very challenging because of the multidimensional input-output system and the ground contact. Based on the inverse kinematics, we propose a Proportional-Derivative controlled robot that uses Iterative Learning Control to learn discrete body velocities, which are then generalized using the Gaussian Process Regression model for each joint separately. This controller design enables onboard control and learning in real-time without any simulation. This study illustrates the effectiveness of the proposed methodology over a range of velocities while emphasizing the minimal computational effort associated with its application in a practical context.

Sophisticated models can accurately describe deformations of continuum robots while being computationally demanding, which limits their application. Especially when considering sampling-based path planning, the model has to be evaluated frequently, which can lead to substantially increased computation times. We present a new approach to compute the entire shape of a tendon-driven continuum robot by a physics-informed neural network (PINN). The underlying physics is modelled with the Cosserat rod theory and incorporated into the PINN’s loss function. The boundary values for the training are obtained from a reference model, solved by the shooting method. Our approach allows for a computation of the learned Cosserat rod model multiple orders of magnitude faster than a publicly available reference model. The median position deviation from the reference model lies below 1mm (0.5% of the simulated robot length) for each of the robot’s 20 disks.

Using compliant materials leads to continuum robots undergoing large deformations. Their nonlinear behavior motivates the use of model-based controllers. They require state estimation as an essential step to be deployed. Available sensors are usually realized by introducing rigid bodies to the soft robot or inserting soft sensors made of materials different from the robot itself. Both approaches result in changes in the system’s dynamics. Optical measurements are problematic, especially in confined spaces. This can be avoided when the sensor is located at the robot's base. This paper studies the state estimation of a pneumatically actuated soft robot using the measured forces and torques at its base. For the first time, this is done using an unscented Kalman filter without restraining the dynamics to a planar or quasi-static motion while applying it to a real system. Real-time capability is achieved with our implementation. The state estimation is tested in a Cosserat rod simulation and on the physical system. The position is estimated with an accuracy of three to five millimeters for a 130 millimeter long pneumatic robot.

Many application domains, e.g., in medicine and manufacturing, can greatly benefit from pneumatic Soft Robots (SRs). However, the accurate control of SRs has remained a significant challenge to date, mainly due to their nonlinear dynamics and viscoelastic material properties. Conventional control design methods often rely on either complex system modeling or time-intensive manual tuning, both of which require significant amounts of human expertise and thus limit their practicality. In recent works, the data-driven method, Automatic Neural ODE Control (ANODEC) has been successfully used to – fully automatically and utilizing only input-output data – design controllers for various nonlinear systems in silico, and without requiring prior model knowledge or extensive manual tuning. In this work, we successfully apply ANODEC to automatically learn to perform agile, non-repetitive reference tracking motion tasks in a real-world SR and within a finite time horizon. To the best of the authors’ knowledge, ANODEC achieves, for the first time, performant control of a SR with hysteresis effects from only 30 s of input-output data and without any prior model knowledge. We show that for multiple, qualitatively different and even out-of-training-distribution reference signals, a single feedback controller designed by ANODEC outperforms a manually tuned PID baseline consistently. Overall, this contribution not only further strengthens the validity of ANODEC, but it marks an important step towards more practical, easy-to-use SRs that can automatically learn to perform agile motions from minimal experimental interaction time.

For the development of advanced driver assistance systems (ADAS) and autonomous driving, a perception of the vehicle’s environment is necessary. This includes, among others, road gradients, cross-slopes, and the road surface condition, with the maximum friction coefficient of the tire-road contact as a safety-relevant parameter. However, these three road parameters cannot be measured directly while driving by sensors installed in modern vehicles. Current estimation methods provide either the maximum friction coefficient or the road gradient and cross-slope but never combined. Since the road angles influence the maximum friction coefficient estimation and vice versa, separate estimation of these parameters, in general, leads to incorrect estimation results. In this paper, a new Unscented Kalman Filter (UKF)-based approach is proposed for simultaneous estimation of all three mentioned road parameters. For this purpose, a dynamic vehicle model considering road gradients and cross-slopes is introduced and integrated into the UKF. It is demonstrated that, in contrast to a state-of-the-art UKF, the proposed algorithm yields improved accuracy and correct maximum friction coefficient estimates even on roads with gradients or cross-slopes.

Gestural interaction with freehands and while grasping an everyday object enables always-available input . To sense such gestures, minimal instrumentation of the user’s hand is desirable. However, the choice of an effective but minimal IMU layout remains challenging, due to the complexity of the multi-factorial space that comprises diverse finger gestures, objects, and grasps. We present SparseIMU , a rapid method for selecting minimal inertial sensor-based layouts for effective gesture recognition. Furthermore, we contribute a computational tool to guide designers with optimal sensor placement. Our approach builds on an extensive microgestures dataset that we collected with a dense network of 17 inertial measurement units (IMUs). We performed a series of analyses, including an evaluation of the entire combinatorial space for freehand and grasping microgestures (393 K layouts), and quantified the performance across different layout choices, revealing new gesture detection opportunities with IMUs. Finally, we demonstrate the versatility of our method with four scenarios.

When robots are used to automate repetitive production tasks, the productivity of the manufacturing system crucially depends on the robot's task execution speed. An out-of-the-box solution is typically slow, whereas achieving shorter cycle times typically requires large efforts with respect to controller design and tuning. This dilemma can be resolved by learning control algorithms that autonomously improve performance without requiring any system-specific tuning. In the present work, we propose a novel learning control scheme that autonomously reduces the execution times of robotic systems that perform repetitive manufacturing tasks. To this end, we combine an Iterative Learning Control (ILC) approach with a trial-varying reference adaptation. The reference trajectory is slowly adapted to ensure that the given task is performed successfully on every single iteration without constraint violations. Therefore, the learning process can be carried out during operation. We validate the practical applicability of the method by real-world experiments on a 6-axis robot that performs a linear motion and a contact-force task. Despite the fundamentally different characteristics of these two tasks, the proposed algorithm achieves a remarkable reduction of cycle times, namely, by a factor of 4 in the linear motion task and a factor of 10 in the contact-force task. These results provide an important step toward robotic manufacturing systems that autonomously optimize their own performance during operation.

Inertial measurement units are commonly used to estimate the attitude of moving objects. Numerous nonlinear filter approaches have been proposed for solving the inherent sensor fusion problem. However, when a large range of different dynamic and static rotational and translational motions is considered, the attainable accuracy is limited by the need for situation-dependent adjustment of accelerometer and gyroscope fusion weights. We investigate to what extent these limitations can be overcome by means of artificial neural networks and how much domain-specific optimization of the neural network model is required to outperform the conventional filter solution. A diverse set of motion recordings with a marker-based optical ground truth is used for performance evaluation and comparison. The proposed neural networks are found to outperform the conventional filter across all motions only if domain-specific optimizations are introduced. We conclude that they are a promising tool for inertial-sensor-based real-time attitude estimation, but both expert knowledge and rich datasets are required to achieve top performance.

Low back pain (LBP) is a leading cause of activity limitation. Objective assessment of the spinal motion plays a key role in diagnosis and treatment of LBP. We propose a method that facilitates clinical assessment of lower back motions by means of a wireless inertial sensor network. The sensor units are attached to the right and left side of the lumbar region, the pelvis and the thighs, respectively. Since magnetometers are known to be unreliable in indoor environments, we use only 3D accelerometer and 3D gyroscope readings. Compensation of integration drift in the horizontal plane is achieved by estimating the gyroscope biases from automatically detected initial rest phases. For the estimation of sensor orientations, both a smoothing algorithm and a filtering algorithm are presented. From these orientations, we determine three-dimensional joint angles between the thighs and the pelvis and between the pelvis and the lumbar region. We compare the orientations and joint angles to measurements of an optical motion tracking system that tracks each skin-mounted sensor by means of reflective markers. Eight subjects perform a neutral initial pose, then flexion/extension, lateral flexion, and rotation of the trunk. The root mean square deviation between inertial and optical angles is about one degree for angles in the frontal and sagittal plane and about two degrees for angles in the transverse plane (both values averaged over all trials). We choose five features that characterize the initial pose and the three motions. Interindividual differences of all features are found to be clearly larger than the observed measurement deviations. These results indicate that the proposed inertial sensor-based method is a promising tool for lower back motion assessment.

Robotics and Functional Electrical Stimulation (FES) are well-established technologies for the rehabilitation of stroke and spinal cord injured (SCI) patients. We propose a hybrid solution that combines feedback-controlled FES of biceps and triceps as well as posterior and anterior deltoid with a cable-driven robotic system to support repetitive arm movements, like “breaststroke swimming” exercises. The robotic system partially compensates the arm weight by controlling the cable tension forces, and the FES promotes motion in the transversal plane. To adjust the FES support to the needs of the individual patients we use an iterative learning vector field (ILVF) which encodes the stimulation intensities that are applied to guide the patient along a pre-specified reference trajectory in the joint angle space. In contrast to previous iterative learning control approaches, the ILVF allows the patient to perform the motion at self-selected cadence. The proposed learning algorithm explicitly takes the dynamics of the artificially activated muscles into account and assures smooth stimulation intensity profiles. The control algorithm is tested in simulations using a complex neuro-musculoskeletal model. For “breaststroke” motions, the initial RMS error of purely volitional movements is reduced from 38° to 10° within 21 cycles by the adaptive FES support. After 50 iterations of the ILVF, the algorithm converges to a steady state RMS error of 4°. Changes in the patient's muscle activity and cadence were well tolerated by the control system and did not cause a noticable increase in the steady state RMS error.

Due to their relative ease of handling and low cost, inertial measurement unit (IMU)-based joint angle measurements are used for a widespread range of applications. These include sports performance, gait analysis, and rehabilitation (e.g., Parkinson's disease monitoring or poststroke assessment). However, a major downside of current algorithms, recomposing human kinematics from IMU data, is that they require calibration motions and/or the careful alignment of the IMUs with respect to the body segments. In this article, we propose a new method, which is alignment-free and self-calibrating using arbitrary movements of the user and an initial zero reference arm pose. The proposed method utilizes real-time optimization to identify the two dominant axes of rotation of the elbow joint. The performance of the algorithm was assessed in an optical motion capture laboratory. The estimated IMU-based angles of a human subject were compared to the ones from a marker-based optical tracking system. The self-calibration converged in under 9.5 s on average and the rms errors with respect to the optical reference system were 2.7° for the flexion/extension and 3.8° for the pronation/supination angle. Our method can be particularly useful in the field of rehabilitation, where precise manual sensor-to-segment alignment as well as precise, predefined calibration movements are impractical.

A noninvasive continuous blood pressure measurement technique that has been developed lately requires precise control of the blood flow through a superficial artery. The flow is measured using ultrasound and influenced via manipulating the pressure inside an inflatable air balloon which is placed over the artery. This contribution is concerned with the design and evaluation of a learning cascaded control structure for such measurement devices. Two feedback control loops are designed in discrete time via pole placement and then combined with an iterative learning control. The latter exploits the repetitive nature of the disturbance that is induced by the oscillating arterial pressure. Experimental results indicate that the proposed controller structure yields considerably smaller set point deviations than previous approaches.