Shared control in assistive robotics blends human autonomy with computer assistance, thus simplifying complex tasks for individuals with physical impairments. This study assesses an adaptive Degrees of Freedom control method specifically tailored for individuals with upper limb impairments. It employs a between-subjects analysis with 24 participants, conducting 81 trials across three distinct input devices in a realistic everyday-task setting. Given the diverse capabilities of the vulnerable target demographic and the known challenges in statistical comparisons due to individual differences, the study focuses primarily on subjective qualitative data. The results reveal consistently high success rates in trial completions, irrespective of the input device used. Participants appreciated their involvement in the research process, displayed a positive outlook, and quick adaptability to the control system. Notably, each participant effectively managed the given task within a short time frame.

With the ongoing efforts to empower people with mobility impairments and the increase in technological acceptance by the general public, assistive technologies, such as collaborative robotic arms, are gaining popularity. Yet, their widespread success is limited by usability issues, specifically the disparity between user input and software control along the autonomy continuum. To address this, shared control concepts provide opportunities to combine the targeted increase of user autonomy with a certain level of computer assistance. This paper presents the free and open-source AdaptiX XR framework for developing and evaluating shared control applications in a high-resolution simulation environment. The initial framework consists of a simulated robotic arm with an example scenario in Virtual Reality (VR), multiple standard control interfaces, and a specialized recording/replay system. AdaptiX can easily be extended for specific research needs, allowing Human-Robot Interaction (HRI) researchers to rapidly design and test novel interaction methods, intervention strategies, and multi-modal feedback techniques, without requiring an actual physical robotic arm during the early phases of ideation, prototyping, and evaluation. Also, a Robot Operating System (ROS) integration enables the controlling of a real robotic arm in a PhysicalTwin approach without any simulation-reality gap. Here, we review the capabilities and limitations of AdaptiX in detail and present three bodies of research based on the framework. AdaptiX can be accessed at https://adaptix.robot-research.de.

This work presents a dataset of human-operated robot motion to be used within the context of assistive robotics and assorted fields, such as learning from demonstrations, machine-learning based robot control, and activity recognition. The data consists of individual sequences of intentional robot motion performing a task in an environment of daily living. There are 2973 sequences generated in a high-resolution simulation and 986 sequences performed in reality, totaling to 1.16 M datapoints. The data includes labels for the robot's pose, motion and activity. This paper also provides data augmentation methods and a detailed dataset analysis as well as simple models trained on the dataset as a baseline for future research. The dataset can be downloaded free-of-charge at https://www.kaggle.com/f371xx/dormadl.

Robotic solutions, in particular robotic arms, are becoming more frequently deployed for close collaboration with humans, for example in manufacturing or domestic care environments. These robotic arms require the user to control several Degrees-of-Freedom (DoFs) to perform tasks, primarily involving grasping and manipulating objects. Standard input devices predominantly have two DoFs, requiring time-consuming and cognitively demanding mode switches to select individual DoFs. Contemporary Adaptive DoF Mapping Controls (ADMCs) have shown to decrease the necessary number of mode switches but were up to now not able to significantly reduce the perceived workload. Users still bear the mental workload of incorporating abstract mode switching into their workflow. We address this by providing feed-forward multimodal feedback using updated recommendations of ADMC, allowing users to visually compare the current and the suggested mapping in real-time. We contrast the effectiveness of two new approaches that a) continuously recommend updated DoF combinations or b) use discrete thresholds between current robot movements and new recommendations. Both are compared in a Virtual Reality (VR) in-person study against a classic control method. Significant results for lowered task completion time, fewer mode switches, and reduced perceived workload conclusively establish that in combination with feedforward, ADMC methods can indeed outperform classic mode switching. A lack of apparent quantitative differences between Continuous and Threshold reveals the importance of user-centered customization options. Including these implications in the development process will improve usability, which is essential for successfully implementing robotic technologies with high user acceptance.

Ice cores of polar regions (ice sheets) are one of the most prominent natural archives that can reveal essential historical information from the past environment of our planet. The ice core microstructure is a key feature in determining the principal properties of ice such as pore close-off, albedo, and melt events. Micro-CT scans can provide valuable information about the microstructure of materials, although achieving a high-quality automated segmentation of porous materials, especially with phase/density changes is still a challenge. This work proposes a new method for improving the segmentation of porous microstructures where a weak segmentation (Gaussian Mixture Model) on high-resolution (30 μm) data is used as ground truth to train a deep learning model (U-net) for segmentation of low-resolution (60 μm) data. This approach has reached high segmentation accuracy in terms of quantitative metrics having the F1-score of 92.5% and an intersection over the union of 91%, with a considerable improvement compared to thresholding and unsupervised methods. Also, the segmentation results of U-net are closer to the real weight, density, and specific surface area of the specimen.

In this paper, we investigate the problems that Convolutional Neural Networks (CNN)-based pose estimators have with symmetric objects. We considered the value of the CNN’s output representation when continuously rotating the object and found that it has to form a closed loop after each step of symmetry. Otherwise, the CNN (which is itself a continuous function) has to replicate an uncontinuous function. On a 1-DOF toy example we show that commonly used representations do not fulfill this demand and analyze the problems caused thereby. In particular, we find that the popular min-over-symmetries approach for creating a symmetry-aware loss tends not to work well with gradient-based optimization, i.e. deep learning.We propose a representation called "closed symmetry loop" (csl) from these insights, where the angle of relevant vectors is multiplied by the symmetry order and then generalize it to 6-DOF. The representation extends our algorithm from [1] including a method to disambiguate symmetric equivalents during the final pose estimation. The algorithm handles continuous rotational symmetry (e.g. a bottle) and discrete rotational symmetry (e.g. a 4-fold symmetric box). It is evaluated on the T-LESS dataset, where it reaches state-of-the-art for unrefining RGB-based methods.

Control, tracking, and obstacle detection algorithms for mobile robots, including autonomous cars, rely on a jump-free estimate of the vehicle's pose. While one cannot completely avoid jumps in global solutions like INS/GNSS and SLAM, relative localization (i.e., odometry) does not suffer from this problem. Methods based on graph optimization are popular in that field, but they do not scale very well with high-frequency measurements. Kalman filters (KFs) are able to cope with those measurements, but they face the issue of a continuously growing covariance. This results in instabilities and eventually jumps in the state estimate. We present an approach to handle this problem by periodically moving the reference state forward in time, which is realized using two filters. The equations for implementing this in both the extended Kalman filter (EKF) and the unscented Kalman filter (UKF) are derived. The algorithm is evaluated using real-world datasets covering different scenarios of autonomous driving. We show that our method provides a smooth and stable estimate even over long time periods and that it achieves a better localization performance than the standard approach.

Inertial Navigation Systems suffer from unbounded errors on the position and orientation estimate. Exteroceptive sensors may not always be available to correct the error. Applications in the literature overcome this problem by fusing IMU data with prior knowledge in an ad-hoc fashion. In different applications, various knowledge is available, which allows to correct the erroneous state estimate. In this position paper, we argue that the fusion of knowledge and inertial sensor data should be viewed as a paradigm and that the observability of systems with prior knowledge should be analysed theoretically. With a theoretical foundation, application design will be simplified and verifiable. We show methods to start the analysis and give a first proof with practical insight.

Inertial Navigation Systems suffer from unbounded errors on the position and orientation estimate. This drift can be corrected by applying prior knowledge, instead of using exteroceptive sensors. Analysing the state observability induced by prior knowledge motivates us to track bikers in track cycling races. In this paper, we show that the pose of the bikers can be estimated with an IMU as the only sensor by using a heightmap of the track and the knowledge that the biker drives forward. We present a dataset with three 60-round trials and evaluate the state estimate. We show that the influences of the priors match the expectation derived from state observability analysis.

We present a new dataset with the goal of advancing the state-of-the-art in object pose estimation especially for stored porcelain and glass crockery in kitchen scenes. Specifically the ESKO 6d (EASE Stored Kitchen Objects with 6d poses) dataset features texture-less, glossy or glassy ordinary used objects which were naturally stored in a cupboard, drawer or dishwasher. There is a large degree of occlusion being the specific challenge in these scenes. Each scene was recorded in video sequences by two cameras (RGB-D (Kinect) and binocular) within multiple setup stages. The dataset contains an RGB-D image or binocular RGB image plus stereo-matched depth image as well as 6d pose ground truth and instance segmentation. Our dataset contains twelve stored object scenes with a combined amount of 47 video sequences captured by each camera, resulting in over 17k annotated Kinect images and more than 42k annotated stereo images showing around 50 different objects. The ground truth annotation is precise to 3. 5mm ADD (details see paper). The dataset can be accessed under http://www.informatik.uni-bremen.de/esko6d-dataset. Besides the concrete dataset we propose a method of ground truth pose measurement based on an external 3d tracking system that allows on the one hand to precisely measure the object's pose inside a tight packed storage and on the other hand to obtain the object pose in several images with just one manual measurement.

We introduce a task oriented optimal controller based on the finite horizon quadratic regulator (LQR). The task of being at a specified position with a specified velocity in specified time is formalized by cost functions. Moreover, we include soft constraints which are part of the cost function, such that the optimal control can be computed in fixed time. We show how soft constraints help to use redundancy and point out the limitations of our soft constraint approach in simulations.

This paper presents an estimator to obtain the posture of a moving skeleton, i.e. all relative orientations of the skeleton's bodies that are connected via a common joint. The estimator uses inertial sensor data only, the relative orientation around all axes is obtained without magnetometers. Instead, the accelerations of the skeleton's motion provide information about the otherwise missing degree of freedom. The method also yields the orientation of the skeleton as a whole except for the heading. In addition, a technique to decouple the estimation rate from the sensor sampling rate is introduced. The estimator both with and without rate decoupling is evaluated against ground truth data and yields relative orientations about 5 degrees off ground truth while the skeleton moves.

In this paper we propose a physically motivated sensor model of Time-of-Flight cameras. We provide methods to calibrate our proposed model and compensate all modeled effects. This enables us to reliably detect and filter out inconsistent measurements and to record high dynamic range (HDR) images. We believe that HDR images have a significant benefit especially for mapping narrow-spaced environments as in urban search and rescue. We provide methods to invert our model in real-time and gain significantly higher precision than using the vendor-provided sensor driver. In contrast to previously published purely phenomenological calibration methods our model is physically motivated and thus better captures the structure of the different effects involved.

We developed a task-oriented controller based on optimal finite horizon control. We demonstrate this on a flexible ball playing robot with redundant degrees of freedom. The task is to reach a specified Cartesian position and velocity of the bat at a specified time, in order to rebound the ball. The controller must maintain high accuracy and react to disturbances and changing conditions. Therefore, we formulate this as an optimal control problem giving the controller the possibility to autonomously distribute motor torques amongst the redundant degrees of freedom. In simulations, we show the accuracy of the controller, the intelligent distribution of motor torques, as well as robustness against disturbances and adaptation to changing conditions.

For such common tasks as motion planning or object recognition robots need to perceive their environment and create a dense 3D map of it. A recent breakthrough in this area was the KinectFusion algorithm [16], which relies on step by step matching a depth image to the map via ICP to recover the sensor pose and updating the map based on that pose. In so far it ignores techniques developed in the graph-SLAM area such as fusion with odometry, modeling of uncertainty and distributing an observed inconsistency over the map. This paper presents a method to integrate a dense geometric truncated signed distance function (TSDF) representation as KinectFusion uses with a sparse parametric representation as common in graph SLAM. The key idea is to have local TSDF sub-maps attached to reference nodes in the SLAM graph and derive graph-SLAM links via ICP by matching a map to a depth image. By moving these reference nodes according to the graph-SLAM estimate, the overall map can be deformed without touching individual sub-maps so that re-building of sub-maps is only needed in case of significant deformation within a sub-map. Also, further information can be added to the graph as common in graph SLAM. Examples are odometry or the fact that the ground is roughly but not exactly planar. Additionally, the paper proposes a modification of the KinectFusion algorithm to improve handling of long range data by taking the range dependent uncertainty into account.

We present a method to simultaneously track multiple objects which are subject to physical motion and can be evaluated through raw detector responses in video. Due to their two-staged design, popular tracking-by-detection approaches lack precision in the estimated trajectories due to detector inaccuracies, e.g., lighting, deformation or background clutter. Instead of separating the tasks of detection and tracking, we propose to integrate both in a single probabilistic objective function for determining the object states in a sequence. Both support each other accounting for detection inaccuracies and leading to a robust and precise single target tracker. Based on this, we extend it to multiple targets by solving the problem of determining trajectory limits and sorting out any multiple target ambiguities probabilistically. We apply our method to the task of tracking thrown balls with the goal of accurate trajectory prediction for the purpose of ball catching with a humanoid robot. Our results show improved tracking accuracy with respect to ground truth on average by around 17 %, which is dominated by increased accuracy at the beginning of the trajectory.

This paper studies different criteria for selecting configurations for the task of calibrating a robotic system. Given an automatic and self-contained procedure which allows the robot to calibrate itself without the need of external tools, we are interested in how to select the set of configurations that maximize calibration accuracy while minimizing calibration time. We experiment with the active calibration of a multi-sensorial humanoid's upper body and report that determinant-based criteria should be preferred when a greedy selection is used. In addition to criteria comparison, we further propose a new criterion for configuration selection. Its novelty stems from a direct treatment of the robot's end-effector tool variance. This is contrary to previous approaches which target the variance indirectly via calibration parameters. Our proposed objective function is derived as a compact formulation from the mean error of the robot's end-effector tool from which its variance can be computed using traditional criteria known from the theory of optimal experimental design (e.g. A-optimality).

The Kinect sensor and KinectFusion algorithm have revolutionized environment modeling. We bring these advances to optimization-based motion planning by computing the obstacle and self-collision avoidance objective functions and their gradients directly from the KinectFusion model on the GPU without ever transferring any model to the CPU. Based on this, we implement a proof-of-concept motion planner which we validate in an experiment with a 19-DOF humanoid robot using real data from a tabletop work space. The summed-up time from taking the first look at the scene until the planned path avoiding an obstacle on the table is executed is only three seconds.

Without a precise and up-to-date model of its environment a humanoid robot cannot move safely or act usefully. Ideally, the robot should create a dense 3D environment model in real-time, all the time, and respect obstacle information from it in every move it makes as well as obtain the information it needs for fine manipulation with its fingers from the same map. We propose to use a multi-scale truncated signed distance function (TSDF) map consisting of concentric, nested cubes with exponentially decreasing resolution for this purpose. We show how to extend the KinectFusion real-time SLAM algorithm to the multi-scale case as well as how to compute a multi-scale Euclidean distance transform (EDT) thereby establishing the link to optimization-based planning. We overcome the inability of KinectFusion's localization to handle scenes without enough constraining geometry by switching to mapping-with-known-poses based on forward kinematics. The latter is always available and we know when it is precise. The resulting map has the desired properties: It is computed in real-time (7.5 ms per depth frame for a (8 m)3multi-scale TSDF volume), covers the entire laboratory, does not depend on scene properties (geometry, texture, etc.) and is precise enough to facilitate grasp planning for fine manipulation tasks - all in a single map.

Complex manipulation tasks require an accurate interplay of actuation and sensing. This accuracy can only be achieved by calibrating the relevant components beforehand. Typically calibration procedures are time-consuming and often include subsequent calibration steps, involve multiple people and require external tools. In this paper we alleviate these issues by auto-calibrating the different sensors of DLR's humanoid Rollin' Justin in a single, completely automatic and self-contained procedure, i.e. without calibration plate. By observing a single point feature on each wrist while moving the robot's head, the stereo cameras' intrinsic and extrinsic parameters are calibrated together with the arm joint elasticities and joint angle offsets. Additionally, we use the head motion to calibrate an Inertial Measurement Unit (IMU) extrinsically. Parameters are obtained by formulating the calibration problem as a batch-optimization problem that estimates all parameters jointly. A rough initial guess, as is, e.g., available when re-calibrating, is needed for the estimation and to facilitate marker detection. The procedure is validated on real hardware and reduces the effort considerably allowing rapid (5 min movement time), automatic, and accurate calibration by simply “pushing a button”.

Map joining is an efficient strategy for solving feature based SLAM problems. This paper demonstrates that joining of two 2D local maps, formulated as a nonlinear least squares problem has at most two local minima, when the associated uncertainties can be described using spherical covariance matrices. Necessary and sufficient condition for the existence of two minima is derived and it is shown that more than one minimum exists only when the quality of the local maps used for map joining is extremely poor. The analysis explains to some extent why a number of optimization based SLAM algorithms proposed in the recent literature that rely on local search strategies are successful in converging to the globally optimal solution from poor initial conditions, particularly when covariance matrices are spherical. It also demonstrates that the map joining problem has special properties that may be exploited to reliably obtain globally optimal solutions to the SLAM problem.

The mobile humanoid Rollin'Justin is a versatile experimental platform for research in manipulation tasks. Previously, different state of the art control methods and first autonomous task execution scenarios have been demonstrated. In this video two new applications with challenging task requirements are presented. One is the catching of one or even two flying balls using all of Justin's degrees of freedom. The other is the autonomous preparation of coffee. Both applications need adequate sensors to support local referencing. The required precision in position and timing is realized in software, using the sensor information, taking the varying precision of Justin's kinematic sub-chains into account and handling all timings in sub-millisecond range.

This paper presents a realtime perception system for catching flying balls with DLR's humanoid Rollin' Justin. We use a two-staged bottom up approach in which we first detect balls as circles and feed these measurements into a multiple hypothesis tracker (MHT). The novel circle detection scheme works in realistic scenes without tuning parameters or background assumptions. We extend the classical multi-hypothesis tracking with prior information about the expected trajectories, therefore limiting the number of hypotheses in the first place. Since the robot starts moving while the ball is still tracked, the cameras shake heavily. A 6-DOF inertial measurements unit (IMU) is integrated to compensate this motion. Using ground-truth from a marker based tracking system we evaluate the metrical accuracy of the motion compensation as well as the tracker's prediction accuracy while in motion.

This paper shows that the field of visual SLAM has matured enough to build a visual SLAM system from open source components. The system consists of feature detection, data association, and sparse bundle adjustment. For all three modules we evaluate different libraries w.r.t. ground truth.

We describe a method for estimating position and velocity of multiple flying balls for the purpose of robotic ball catching. For this a multi-target recursive Bayes filter, the Gaussian Mixture Probability Hypothesis Density filter (GMPHD), fed by a circle detector is used. This recently developed filter avoids the need to enumerate all possible data association decisions, making them computationally efficient. Over time, a mixture of Gaussians is propagated as tracks, predicted into the future and then sent to the robot. By learning a prior from training data we are focusing on detections that are likely to lead to a catchable trajectory which increases robustness. We evaluate the tracker's performance by comparing it with ground truth data, assessing tracking performance as well as the prediction precision of single tracks. Reasonable prediction performance is acquired right from the start, leading to a good overall catching rate.

We present a real-time self collision detection algorithm applicable for industrial and humanoid robots. The algorithm is based on computing the swept volumes of all bodies and checking them pairwise for collisions. The algorithm operates on joint angle intervals. Such, it does not only test a single or N intermediate configurations but assures safety of a whole movement. Key idea of the new swept volume computation is representing volumes as convex hulls extended by a buffer radius, so called sphere swept convex hulls (SSCH). This leads to tight and compact bounding volumes. The operation set available to model the different joints is strictly conservative and allows for a trade-off between accuracy and computation time. During a configurable timespan the algorithm updates a table of pairwise distances and thus can guarantee hard real-time. It is applied on DLR's humanoid Justin in a sports robotic scenario, where also accuracy and computational performance is evaluated (0.4ms, INTEL T2500@2GHz).

Non-linear optimization on constraint graphs has recently been applied very successfully in a variety of SLAM backends. We combine this technique with a principled way of handling non-Euclidean spaces, 3D orientations in particular, based on manifolds to build a generic and very flexible framework, the Manifold Toolkit for Matlab (MTKM). We show that MTKM makes it particularly easy to solve non-trivial multi-sensor calibration problems while remaining generic enough to handle a very different class of problems, namely SLAM, as well: After an introductory example on single camera calibration we apply MTKM to calibration of stereo vision and IMU w.r.t. the kinematic chain of a service robot, RGB-D and accelerometer calibration of a Microsoft Kinect, stereo calibration on a Nao soccer robot, and several SLAM benchmark data sets illustrating MTKM's versatility. MTKM and all presented examples are available as open source from http://openslam.org/MTK.html.

In this paper, we present a new hierarchical optimization solution to the graph-based simultaneous localization and mapping (SLAM) problem. During online mapping, the approach corrects only the coarse structure of the scene and not the overall map. In this way, only updates for the parts of the map that need to be considered for making data associations are carried out. The hierarchical approach provides accurate non-linear map estimates while being highly efficient. Our error minimization approach exploits the manifold structure of the underlying space. In this way, it avoids singularities in the state space parameterization. The overall approach is accurate, efficient, designed for online operation, overcomes singularities, provides a hierarchical representation, and outperforms a series of state-of-the-art methods.

Most people believe SLAM is a complex nonlinear estimation/optimization problem. However, recent research shows that some simple iterative methods based on linearization can sometimes provide surprisingly good solutions to SLAM without being trapped into a local minimum. This demonstrates that hidden structure exists in the SLAM problem that is yet to be understood. In this paper, we first analyze how far SLAM is from a convex optimization problem. Then we show that by properly choosing the state vector, SLAM problem can be formulated as a nonlinear least squares problem with many quadratic terms in the objective function, thus it is clearer how far SLAM is from a linear least squares problem. Furthermore, we explain that how the map joining approaches reduce the nonlinearity/nonconvexity of the SLAM problem.

Treemap is a generic SLAM algorithm that has been successfully used to estimate extremely large 2D maps closing a loop over a million landmarks in 442ms. We are currently working on an open-source implementation that can handle most variants of SLAM. In this paper we discuss the generic part of the algorithm constituting the treemap backend and the variant specific parts acting as a driver. We present their interplay from a software-engineering point of view and show results for the case of 6-DOF feature based SLAM, closing a simulated loop over 106657 3D features in 209ms.

We present an improved version of the treemap SLAM algorithm which uses Cholesky factors for representing Gaussians and a hierarchical tree partitioning algorithm derived from the established Kernighan-Lin heuristic for graph bisection. We demonstrate the algorithm's efficiency by mapping a simulated building with 1032271 landmarks. In the end, we close a million-landmarks loop in 21 ms, providing an estimate for ap10000 selected landmarks close to the robot, or in 442 ms for computing a full estimate

This paper describes the use of natural language route descriptions in the mobile robot navigation domain. Guided by corpus analysis and earlier work on coarse qualitative route descriptions, we decompose instructions given by humans into sequences of imprecise route segment descriptions. By applying fuzzy rules for the involved spatial relations and actions, we construct a search tree that can be searched in a depth-first branch-and-bound manner for the most probable goal configuration w.r.t. the global workspace knowledge of the robot. The applicability of our approach is shown by a real-world experiment where an operator instructs his automated wheelchair to navigate in an office-like environment

For the Simultaneous Localization and Mapping problem several efficient algorithms have been proposed that make use of a sparse information matrix representation (e.g. SEIF, TJTF, treemap). Since the exact SLAM information matrix is dense, these algorithm have to approximate it (sparsification). It has been empirically observed that this approximation is adequate because entries in the matrix corresponding to distant landmarks are extremely small. This paper provides a theoretical proof for this observation, specifically showing that the off-diagonal entries corresponding to two landmarks decay exponentially with the distance traveled between observation of first and second landmark.

This paper addresses the problem of simultaneous localization and mapping (SLAM) by a mobile robot. An incremental SLAM algorithm is introduced that is derived from multigrid methods used for solving partial differential equations. The approach improves on the performance of previous relaxation methods for robot mapping, because it optimizes the map at multiple levels of resolution. The resulting algorithm has an update time that is linear in the number of estimated features for typical indoor environments, even when closing very large loops, and offers advantages in handling nonlinearities compared with other SLAM algorithms. Experimental comparisons with alternative algorithms using two well-known data sets and mapping results on a real robot are also presented.

This paper addresses the problem of impedance control for flexible joint robots based on a singular perturbation approach. Some aspects of the impedance controller, which turned out to be of high practical relevance during applications are then addressed, such as the implementation of nullspace stiffness for redundant manipulators, the avoiding of mass matrix decoupling and the related design of the desired damping matrix. Finally, the proposed methods are validated through measurements on the DLR robot.

We present a system for catching a flying ball with a robot arm using off-the-shelf components (PC based system) for visual tracking. The ball is observed by a large baseline stereo camera, comparing each image to a slowly adapting reference image. We track and predict the target position using an extended Kalman filter, also taking into account the air drag. The calibration is achieved by simply performing a few throws and observing their trajectories, as well as moving the robot to some predefined positions.