Industrial manufacturing of machine parts typically relies on executing operations planned by Computer-Aided-Manufacturing (CAM). Sometimes, there is no CAM program to the factory worker on the shopfloor available or an existing program must be altered. Integrating path planning capabilities into the machine can empower the factory worker to generate and adapt machine programs directly on the shop floor. In this paper, we sketched a novel approach. The human-centric machine uses large language model-powered agentic frameworks to generate and adapt manufacturing programs and their execution. We explored the capabilities and limitations of this approach using the case study of object reconstruction via laser scanning. Our findings demonstrated that agent-based interactions can effectively bridge the gap between high-level user intentions and precise machine operations. Furthermore, we have distilled our learnings into a general concept for a human-centric machine to seamlessly integrate operation planning into industrial robots and machines, aiming to further empower factory workers and enhance adaptability in manufacturing environments.

Accurate temporal segmentation of human actions is critical for intelligent robots in collaborative settings, where a precise understanding of sub-activity labels and their temporal structure is essential. However, the inherent noise in both human pose estimation and object detection often leads to over-segmentation errors, disrupting the coherence of action sequences. To address this, we propose a Multi-Modal Graph Convolutional Network (MMGCN) that integrates low-frame-rate (e.g., 1 fps) visual data with high-frame-rate (e.g., 30 fps) motion data (skeleton and object detections) to mitigate fragmentation. Our framework introduces three key contributions. First, a sinusoidal encoding strategy that maps 3D skeleton coordinates into a continuous sin-cos space to enhance spatial representation robustness. Second, a temporal graph fusion module that aligns multi-modal inputs with differing resolutions via hierarchical feature aggregation, Third, inspired by the smooth transitions inherent to human actions, we design SmoothLabelMix, a data augmentation technique that mixes input sequences and labels to generate synthetic training examples with gradual action transitions, enhancing temporal consistency in predictions and reducing over-segmentation artifacts.Extensive experiments on the Bimanual Actions Dataset, a public benchmark for human-object interaction understanding, demonstrate that our approach outperforms state-of-the-art methods, especially in action segmentation accuracy, achieving F1@10: 94.5% and F1@25: 92.8%.

Motion detection in 3D LiDAR is crucial for autonomous systems. While deep learning dominates Moving Object Segmentation (MOS), the potential of learning-free approaches remains underexplored. Unlike problems like semantic segmentation, motion can be explicitly modeled, potentially enabling efficient, interpretable, and computationally lightweight solutions. Motivated by this, we introduce a novel real-time, online, learning-free MOS method. We propose the novel Join Count Feature to extract motion cues from a local window of range images, and long-term filtering with efficient two-step association to enhance accuracy. Compared to learning-based models, we achieve superior precision and competitive IoU for saliently moving objects on SemanticKITTI. Further evaluation on HeLiMOS demonstrate stronger generalization by the proposed method across different LiDAR sensors. These results highlight the potential of learning-free methods for motion detection in 3D LiDAR data.

Reorienting diverse objects with a multi-fingered hand is a challenging task. Current methods in robotic in-hand manipulation are either object-specific or require permanent supervision of the object state from visual sensors. This is far from human capabilities and from what is needed in real-world applications. In this work, we address this gap by training shape-conditioned agents to reorient diverse objects in hand, relying purely on tactile feedback (via torque and position measurements of the fingers’ joints). To achieve this, we propose a learning framework that exploits shape information in a reinforcement learning policy and a learned state estimator. We find that representing 3D shapes by vectors from a fixed set of basis points to the shape’s surface, transformed by its predicted 3D pose, is especially helpful for learning dexterous in-hand manipulation. In simulation and real-world experiments, we show the reorientation of many objects with high success rates, on par with state-of-the-art results obtained with specialized single-object agents. Moreover, we show generalization to novel objects, achieving success rates of ~90% even for non-convex shapes.Website: https://aidx-lab.org/manipulation/iros24

The ability to follow people can benefit the human-robot interaction of mobile robots. This work proposes a hybrid human tracking system for human following robots, integrating sensor fusion of Ultra-Wideband (UWB) and monocular visual positioning to enhance tracking accuracy and precision. At the same time, UWB and the visual positioning system can operate independently, thereby creating a redundancy in the system. Based on our previous study of UWB-positioning, this article elaborates on a visual positioning system that employs human detection using a pre-trained Convolutional Neural Network (CNN), coupled with data fusion process based on experimental assessments. The hybrid human tracking system achieves a 2D Euclidean accuracy RMS of 7.4 cm, demonstrating sufficient accuracy for human following and improving the following performance in real-world experiments compared to our previous study.

We propose a framework for the measurement of static and dynamic physical properties of manipulation objects using both robotic tactile and kinesthetic sensing - in particular data from fingertip force/torque (F/T) and robot joint torque sensors. It completes the manipulation-relevant information about new objects that cannot be estimated from a passive camera observation. The system allows to balance the accuracy and complexity of the estimation system against the costs and complexity of the approach. We evaluate methods that allow improving robustness against noise and model errors in the manipulation system used for the estimation. The approach is validated on experimental results using data from a torque-controlled robot manipulator and precision F/T sensors.

In skeleton-based action recognition, Graph Convolutional Networks model human skeletal joints as vertices and connect them through an adjacency matrix, which can be seen as a local attention mask. However, in most existing Graph Convolutional Networks, the local attention mask is defined based on natural connections of human skeleton joints and ignores the dynamic relations for example between head, hands and feet joints. In addition, the attention mechanism has been proven effective in Natural Language Processing and image description, which is rarely investigated in existing methods. In this work, we proposed a new adaptive spatial attention layer that extends local attention map to global based on relative distance and relative angle information. Moreover, we design a new initial graph adjacency matrix that connects head, hands and feet, which shows visible improvement in terms of action recognition accuracy. The proposed model is evaluated on two large-scale and challenging datasets in the field of human activities in daily life: NTU-RGB+D and Kinetics skeleton. The results demonstrate that our model has strong performance on both dataset.

Vital signs such as heart rate, oxygen saturation, and blood pressure are crucial information for healthcare workers to identify clinical deterioration of ward patients. Currently, medical devices monitor these vital signs and trigger alarms when the vital signs are not in the normal ranges based on predefined thresholds, which suggests the presence of clinical deterioration. However, such threshold-based approach is not robust for patient monitoring. This is because vital signs differ among patients due to human physiology and change across time based on the action performed by a patient. In this work, we want to tackle these problems by building adaptable action-aware vital models. These models can understand the changes in vital signs caused by patient's actions and can be adapted to the normal vital sign ranges of individual patients. Our experimental results show that general vital sign patterns for different actions exist and can be personalized to new patients. Additionally, we investigate the possibility of estimating the initial vital model for an unobserved action using models of observed actions for model personalization. The resulting adaptable action-aware vital models have the potential to improve patient monitoring by reducing false clinical alarms.

Planning collision-free motions for robots with many degrees of freedom is challenging in environments with complex obstacle geometries. Recent work introduced the idea of speeding up the planning by encoding prior experience of successful motion plans in a neural network. However, this “neural motion planning” did not scale to complex robots in unseen 3D environments as needed for real-world applications. Here, we introduce “basis point set”, well-known in computer vision, to neural motion planning as a modern compact environment encoding enabling efficient supervised training networks that generalize well over diverse 3D worlds. Combined with a new elaborate training scheme, we reach a planning success rate of 100 %. We use the network to predict an educated initial guess for an optimization-based planner (OMP), which quickly converges to a feasible solution, massively outperforming random multi-starts when tested on previously unseen environments. For the DLR humanoid Agile Justin with 19 DoF and in challenging obstacle environments, optimal paths can be generated in 200 ms using only a single CPU core. We also show a first successful real-world experiment based on a high-resolution world model from an integrated 3D sensor.

Human activities recognition is an important task for an intelligent robot, especially in the field of human-robot collaboration, it requires not only the label of sub-activities but also the temporal structure of the activity. In order to automatically recognize both the label and the temporal structure in sequence of human-object interaction, we propose a novel Pyramid Graph Convolutional Network (PGCN), which employs a pyramidal encoder-decoder architecture consisting of an attention based graph convolution network and a temporal pyramid pooling module for downsampling and upsampling interaction sequence on the temporal axis, respectively. The system represents the 2D or 3D spatial relation of human and objects from the detection results in video data as a graph. To learn the human-object relations, a new attention graph convolutional network is trained to extract condensed information from the graph representation. To segment action into sub-actions, a novel temporal pyramid pooling module is proposed, which upsamples compressed features back to the original time scale and classifies actions per frame. We explore various attention layers, namely spatial attention, temporal attention and channel attention, and combine different upsampling decoders to test the performance on action recognition and segmentation. We evaluate our model on two challenging datasets in the field of human-object interaction recognition, i.e. Bimanual Actions and IKEA Assembly datasets. We demonstrate that our classifier significantly improves both framewise action recognition and segmentation, e.g., F1 micro and F1@50 scores on Bimanual Actions dataset are improved by 4.3% and 8.5% respectively.

The need for comprehensive telemedicine solutions is becoming increasingly relevant due to challenges associated with the ageing population, the increasing shortage of health-care providers, and, more recently, the global pandemic. Existing solutions primarily focus on, e.g., electronic medical records, audiovisual connections, and, in some cases, robotic systems with very basic capabilities. Here we present a fundamentally new, holistic approach to a remote doctor visit, which enables transparent remote examination, anomaly detection, diagnosis, and rehabilitation. Our dual doctor-patient twin paradigm involves two robotic systems: one representing the doctor to the patient ("GARMI") and one representing the patient to the doctor ("MUCKI"). Through bidirectional telepresence control, this system enables transparent, natural, remote haptic interaction between doctor and patient. The control, interaction, and knowledge transfer to the doctor is enhanced by AI-based visual motion and facial expression analysis as well as a digital twin of the patient. Thus, each stage of a doctor visit can be replicated in the context of telemedicine and shared autonomy: from first assessment to observation-based and remote physical examination, to a better-informed doctor diagnosis and robot-assisted telerehabilitation.

This paper proposes a novel dictionary learning approach to detect event anomalities using skeletal information extracted from RGBD video. The event action is represented as several latent action atoms and composed of latent spatial and temporal attributes. We aim to construct a network able to learn from few examples and also rules defined by the user. The skeleton frames are clustered by an initial K-means method. Each skeleton frame is assigned with a varying weight parameter and fed into our Gradual Online Dictionary Learning (GODL) algorithm. During the training process, outlier frames will be gradually filtered by reducing the weight that is inversely proportional to a cost. To strictly distinguish the event action from similar actions and robustly acquire its action units, we build a latent unit temporal structure for each sub-action.We validate the method at the example of fall event detection on NTU RGB+D dataset, because it provides a benchmark available for comparison. We present the experimental validation of the achieved accuracy, recall, and precision. Our approach achieves the best performance in precision and accuracy of human fall event detection, compared with other existing dictionary learning methods. Our method remains the highest accuracy and the lowest variance, with increasing noise ratio.

Autonomous vehicles need to be able to perceive both the presence and motion of objects in the surrounding environment to navigate in the real world. In this work, we propose to solve the tasks of identifying objects and estimating the corresponding motion by viewing them as a single unified task known as instance flow. Instance flow provides the pixel-wise instance mask of an object and the dense optical flow within it. To achieve this, we extended the state of the art object detection model to include a dense optical flow estimator. The estimator is used to estimate the optical flow for each region of interest only, instead of the entire image. We tested the approach by carrying out experiments on publicly available datasets for autonomous driving research, VKITTI, KITTI and HD1K. Furthermore, we also introduced a new instance flow quality metric to evaluate the instance flow estimation.

We present a novel representation of dynamic scenes perceived by moving agents. This type of environments require constant updates of the map information that uses conventional geometric representations due to the changing relative position of dynamic object to the static scene. We show that this changed representation of the environment increases the robustness and accuracy in the perception of the scene and simplifies significantly the processing and complexity of the perception module. At the same time, the changed representation allows also a better prioritization of attention to the moving object around the robot that takes into account not the Euclidean distance but the time-to-interaction (TTI), which is the core contribution of this approach. We present the mathematical framework behind the pro-posed representation and show examples, how this framework simplifies and robustifies the processing in the perception modules of a moving agent.

Human drivers have complex and individual behavior characteristics which describe how they act in a specific situation. Accurate behavior models are essential for many applications in the field of autonomous driving, ranging from microscopic traffic simulation, intention estimation and trajectory prediction, to interactive and cooperative motion planning. Designing such models by hand is cumbersome and inaccurate, especially in urban environments, with their high variety of situations and the corresponding diversity in human behavior. Learning how humans act from recorded scenarios is a promising way to overcome these problems. However, predicting complete trajectories at once is challenging, as one needs to account for multiple hypotheses and long-term interactions between multiple agents. In contrast, we propose to learn Markovian action models with deep neural networks that are conditioned on a driver's route intention (such as turning left or right) and the situational context. Step-wise forward simulation of these models for the different possible routes of all agents allows for multi-modal and interaction-aware scene predictions at arbitrary road layouts. Learning to predict only one time step ahead given a specific route reduces learning complexity, such that simpler and faster models are obtained. This enables the integration into particle-based algorithms such as Monte Carlo tree search or particle filtering. We evaluate the learned model both on its own and integrated into our previously presented dynamic Bayesian network for intention estimation and show that it outperforms our previous hand-tuned rule-based model.

We propose an object-centric framework to label and predict human manipulation actions from observations of the object trajectories in 3D space. The goal is to lift the low-level sensor observation to a context specific human vocabulary. The low-level visual sensory input from a depth camera is processed into high-level descriptive action labels using a directed action graph representation. It is built based on the concepts of pre-computed Location Areas (LA), regions within a scene where an action typically occur, and Sector-Maps (SM), reference trajectories between the LAs. The framework consists of two stages, an offline teaching phase for graph generation, and an online action recognition phase that maps the current observations to the generated graph. This graph representation allows the framework to predict the most probable action from the observed motion in real-time and to adapt its structure whenever a new LA appears. Furthermore, the descriptive action labels enable not only a better exchange of information between a human and a robot but they allow also the robots to perform high-level reasoning. We present experimental results on real human manipulation actions using a system designed with this framework to show the performance of prediction and labeling that can be achieved.

Planning for autonomous driving in complex, urban scenarios requires accurate prediction of the trajectories of surrounding traffic participants. Their future behavior depends on their route intentions, the road-geometry, traffic rules and mutual interaction, resulting in interdependencies between their trajectories. We present a probabilistic prediction framework based on a dynamic Bayesian network, which represents the state of the complete scene including all agents and respects the aforementioned dependencies. We propose Markovian, context-dependent motion models to define the interaction-aware behavior of drivers. At first, the state of the dynamic Bayesian network is estimated over time by tracking the single agents via sequential Monte Carlo inference. Secondly, we perform a probabilistic forward simulation of the network's estimated belief state to generate the different combinatorial scene developments. This provides the corresponding trajectories for the set of possible, future scenes. Our framework can handle various road layouts and number of traffic participants. We evaluate the approach in online simulations and real-world scenarios. It is shown that our interaction-aware prediction outperforms interaction-unaware physics- and map-based approaches.

Isotropic three-dimensional (3D) acquisition is a challenging task in magnetic resonance imaging (MRI). Particularly in cardiac MRI, due to hardware and time limitations, current 3D acquisitions are limited by low-resolution, especially in the through-plane direction, leading to poor image quality in that dimension. To overcome this problem, super-resolution (SR) techniques have been proposed to reconstruct a single isotropic 3D volume from multiple anisotropic acquisitions. Previously, local regularization techniques such as total variation have been applied to limit noise amplification while preserving sharp edges and small features in the images. In this paper, inspired by the recent progress in patch-based reconstruction, we propose a novel isotropic 3D reconstruction scheme that integrates non-local and self-similarity information from 3D patch neighborhoods. By grouping 3D patches with similar structures, we enforce the natural sparsity of MR images, which can be expressed by a low-rank structure, leading to robust image reconstruction with high signal-to-noise ratio efficiency. An Augmented Lagrangian formulation of the problem is proposed to efficiently decompose the optimization into a low-rank volume denoising and a SR reconstruction. Experimental results in simulations, brain imaging and clinical cardiac MRI, demonstrate that the proposed joint SR and self-similarity learning framework outperforms current state-of-the-art methods. The proposed reconstruction of isotropic 3D volumes may be particularly useful for cardiac applications, such as myocardial infarction scar assessment by late gadolinium enhancement MRI.

In order to determine a cooperative driving strategy, it is beneficial for an autonomous vehicle to incorporate the intended motion of surrounding vehicles within its own motion planning. However, as intentions cannot be measured directly and the motion of multiple vehicles often are highly interdependent, this incorporation has proven challenging. In this paper, the problem of maneuver estimation is addressed, focusing on situations with close interaction between traffic participants. Therefore, we define collective maneuvers based on trajectory homotopy, describing the relative motion of multiple vehicles in a scene. Representing maneuvers by sample trajectories, maneuver-dependent prediction models of the vehicle states can be defined. This allows for a Bayesian estimation of maneuver probabilities given observations of the real motion. The approach is evaluated by simulation in overtaking scenarios with oncoming traffic and merging scenarios at an intersection.

This paper proposes a reactive obstacle avoidance approach based solely on image data from a monocular camera stream. By clustering and analyzing the optical flow, this approach is able to identify potential collisions with dynamic obstacles. Epipolar geometry is exploited to derive velocity commands that ensure a collision-free path for a highly maneuverable autonomous vehicle via a real-time optimizer. First, the underlying image processing and optimization principles are explained in detail, before simulation results show the general feasibility of the approach. Finally, real-world tests with the ROboMObil, the German Aerospace Center's robotic electric vehicle, are provided to demonstrate its applicability.

This paper introduces a vision-based controller for automatic vehicle following, also known as 2-vehicle platooning. A direct homography controller is applied to calculate the motion demand for an autonomous vehicle from only the data of a monocular camera. The direct control without an intermediate step to a Cartesian representation increases the robustness of the scheme. A robustness analysis of the closed loop controller is provided using the parameter space approach. Furthermore, the direct homography controller is extended by an estimation of the absolute angular difference to the goal position, which then enables the estimation of the position error. The proposed homography-based position estimation is tested on rendered camera images for better evaluation of the underlying error and the platooning controller is verified in simulation. Finally, the results of both are presented.

We present a novel hierarchical MRFs optimization method for dense and deformable motion extraction in dynamic scenes. In particular, this hierarchical MRFs structure consists of two layers, the segmentation and the correspondence layer. Firstly, dynamic RGB-D foreground data is segmented through a pixel-level MRF in the segmentation layer. Subsequently, the extracted foreground data is transformed into a 3D point-level MRF in the correspondence layer. A new surface descriptor named deformable color and shape histogram is proposed. It is combined with photometric and geometric features to represent a deformable surface. Finally, the dynamic scene motion is retrieved from correspondences established in the image sequence. Discrete optimization schemes are used for the binary classification and multi-labeling problems. We provide an RGB-D dataset of dynamic scenes, which involves different motion patterns and surface properties of foreground objects. The effectiveness and efficiency of our proposed approach for high accurate foreground segmentation and motion extraction is validated in experiments.

In this paper we present a framework for dynamic substitution of different sensory modalities with existing physical sensors. Our system is capable of finding the most optimal set of mathematical and physical transformations between two modalities of physical and virtual sensors. It allows a creation of new virtual sensors from given set of physical sensors. The virtual sensing may extend to new sensing modalities for which no direct physical sensors exist. The framework optimizes for a minimal error and optimal observation in the resulting fusion. It is processing the chain for a given spatial measurement and measurement range. The framework is capable of increasing the reliability of acquired data in multi-sensor systems by being able to asses the amount of accumulated errors. We give two examples of real-world applications of this framework in robotic environments.

In this paper we present an approach to extending the capabilities of telemanipulation systems by intelligently augmenting a human operator's motion commands based on quantitative three-dimensional scene perception at the remote telemanipulation site. This framework is the first prototype of the Augmented Shared-Control for Efficient, Natural Telemanipulation (ASCENT) System. ASCENT aims to enable new robotic applications in environments where task complexity precludes autonomous execution or where low-bandwidth and/or high-latency communication channels exist between the nearest human operator and the application site. These constraints can constrain the domain of telemanipulation to simple or static environments, reduce the effectiveness of telemanipulation, and even preclude remote intervention entirely. ASCENT is a semi-autonomous framework that increases the speed and accuracy of a human operator's actions via seamless transitions between one-to-one teleoperation and autonomous interventions. We report the promising results of a pilot study validating ASCENT in a transatlantic telemanipulation experiment between The Johns Hopkins University in Baltimore, MD, USA and the German Aerospace Center (DLR) in Oberpfaffenhofen, Germany. In these experiments, we observed average telemetry delays of 200ms, and average video delays of 2s with peaks of up to 6s for all data. We also observed 75% frame loss for video streams due to bandwidth limits, giving 4fps video.

In order to be able to replace a human operator, a robotic manipulation system needs to deal with a variety of possible actions. These actions may be more or less constrained in their motion profile and in the accuracy of the transport goals. The robotic system can make use of some of this variation to simplify the control to improve the efficiency of the generated motion. Nevertheless, the human's intention behind the manipulation may not change. We introduce the Elastic Power Path to optimize paths with respect to efficient control in the context of abstractly represented tasks. Our experiments show, that the proposed Elastic Power Path is an efficient method to achieve this aim. The magnitude and the number of turnarounds of the accelerations along the path are significantly reduced.

Robots are meanwhile able to perform several tasks. But what happens, if one or multiple of the robot's joints fail? Is the robot still able to perform the required tasks? Which capabilities of the robot get limited and which ones are lost? We propose an analysis of manipulator structures for the comparison of a robot's capabilities with respect to efficient control. The comparison is processed (1) within a robot in the case of joint failures and (2) between robots with or without joint failures. It is important, that the analysis can be processed independently of the structure of the manipulator. The results have to be comparable between different manipulator structures. Therefore, an abstract representation of the robot's dynamic capabilities is necessary. We introduce the Maneuverability Volume and the Spinning Pencil for this purpose. The Maneuverability Volume shows, how efficiently the end-effector can be moved to any other position. The Spinning Pencil reflects the robot's capability to change its end-effector orientation efficiently. Our experiments show not only the different capabilities of two manipulator structures, but also the change of the capabilities if one or multiple joints fail.

The main contribution of this paper is a probabilistic method for predicting human manipulation intention from image sequences of human-object interaction. Predicting intention amounts to inferring the imminent manipulation task when human hand is observed to have stably grasped the object. Inference is performed by means of a probabilistic graphical model that encodes object grasping tasks over the 3D state of the observed scene. The 3D state is extracted from RGB-D image sequences by a novel vision-based, markerless hand-object 3D tracking framework. To deal with the high-dimensional state-space and mixed data types (discrete and continuous) involved in grasping tasks, we introduce a generative vector quantization method using mixture models and self-organizing maps. This yields a compact model for encoding of grasping actions, able of handling uncertain and partial sensory data. Experimentation showed that the model trained on simulated data can provide a potent basis for accurate goal-inference with partial and noisy observations of actual real-world demonstrations. We also show a grasp selection process, guided by the inferred human intention, to illustrate the use of the system for goal-directed grasp imitation.

Efficient visual pose estimation plays an important role for a variety of applications. To improve the quality, the measurements from different sensors can be fused. However, a reliable fusion requires the knowledge of the uncertainty of each estimate. In this work, we provide an error analysis for the Z∞algorithm. Furthermore, we extend the existing first-order error propagation for the 8-point algorithm to allow for feature normalization, as proposed by Hartley or Mühlich, and the rotation matrix based decomposition. Both methods are efficient visual odometry techniques which allow high frame-rates and, thus, dynamic motions in unbounded workspaces. Finally, we provide experiments which validate the accuracy of the error propagation and which enable a brief comparison, showing that the Z∞significantly outperforms the 8-point algorithm. We also discuss the influence of the number of features, the aperture angle, and the image resolution on the accuracy of the pose estimation.

RGB-D sensors are becoming more and more vital to robotics. Sensors such as the Microsoft Kinect and time of flight cameras provide 3D colored point-clouds in real time can play a crucial role in Robot Vision. However these sensors suffer from precision deficiencies, and often the density of the point-clouds they provide is insufficient. In this paper, we present a multi-camera system for correction and enhancement of the data acquired from an RGB-D sensor. Our system consists of two sensors, the RGB-D sensor (main sensor) and a regular RGB camera (auxiliary sensor). We perform the correction and the enhancement of the data acquired from the RGB-D sensor by placing the auxiliary sensor in a close proximity to the target object and taking advantage of the established epipolar geometry. We have managed to reduce the relative error of the raw point-cloud from a Microsoft Kinect RGB-D sensor by 74.5 % and increase its density up to 2.5 times.

This paper presents a novel approach for reactive obstacle avoidance for static and dynamic objects using monocular image sequences. A sparse motion field is calculated by tracking point features using the Kanade-Lucas-Tomasi method. The rotational component of this sparse optical flow due to ego motion of the camera is compensated using motion parameters estimated directly from the images. A robust method for detection of static and dynamic objects in the scene is applied to identify collision candidates. The approach operates entirely in the image space of a monocular camera and does not require any extrinsic information about the configuration of the sensor or speed of the camera. The system prioritizes the detected collision candidates by their time to collision. Additionally, the spatial distribution of the candidates is calculated for non-degenerated conditions. We present the mathematical framework and the experimental validation of the suggested approach on simulated and real-world data.

System state estimation is an essential part for robot navigation and control. A combination of Inertial Navigation Systems (INS) and further exteroceptive sensors such as cameras or laser scanners is widely used. On small robotic systems with limitations in payload, power consumption and computational resources the processing of exteroceptive sensor data often introduces time delays which have to be considered in the sensor data fusion process. These time delays are especially critical in the estimation of system velocity. In this paper we present a state estimation framework fusing an INS with time delayed, relative exteroceptive sensor measurements. We evaluate its performance for a highly dynamic flight system trajectory including a flip. The evolution of velocity and position errors for varying measurement frequencies from 15Hz to 1Hz and time delays up to 1s is shown in Monte Carlo simulations. The filter algorithm with key frame based odometry permits an optimal, local drift free navigation while still being computationally tractable on small onboard computers. Finally, we present the results of the algorithm applied to a real quadrotor by flying from inside a house out through the window.

Inertia-visual sensor fusion has become popular due to the complementary characteristics of cameras and IMUs. Once the spatial and temporal alignment between the sensors is known, the fusion of measurements of these devices is straightforward. Determining the alignment, however, is a challenging problem. Especially the spatial translation estimation has turned out to be difficult, mainly due to limitations of camera dynamics and noisy accelerometer measurements. Up to now, filtering-based approaches for this calibration problem are largely prevalent. However, we are not convinced that calibration, as an offline step, is necessarily a filtering issue, and we explore the benefits of interpreting it as a batch-optimization problem. To this end, we show how to model the IMU-camera calibration problem in a nonlinear optimization framework by modeling the sensors' trajectory, and we present experiments comparing this approach to filtering and system identification techniques. The results are based both on simulated and real data, showing that our approach compares favorably to conventional methods.

We propose a framework for the sensor-based estimation of manipulation-relevant object properties and the abstraction of known actions in a learning setup from the observation of humans. The descriptors consists of an object-centric representation of manipulation constraints and a scene-specific action graph. The graph spans between the typical places, where objects are placed. This framework allows to abstract the strongly varying actions of a human operator and to monitor unexpected new actions, that require a modification of the knowledge stored in the system. The usage of an abstract, object-centric structure enables not only the application of knowledge in the same situation, but also the transfer to similar environments. Furthermore, the information can be derived from different sensing modalities. The proposed system builds up the representation of manipulation-relevant properties and actions. The properties, which are directly related to the object, are stored in the Object Container. The Functionality Map links the actions with the typical action areas in the environment. We present experimental results on real human actions, showing the quality of the results, that can be obtained with our system.

The knowledge about lanes and the exact position on the road is fundamental for many advanced driver assistance systems. In this paper, a novel iterative histogram based approach with occupancy grids for the detection of multiple lanes is proposed. In highway scenarios, our approach is highly suitable to determine the correct number of all existing lanes on the road. Additionally, the output of the laserscannner based lane detection is fused with a production-available vision based system. It is shown that both sensor systems perfectly complement each other to increase the robustness of a lane tracking system. The achieved accuracy of the fusion system, the laserscannner and video based system is evaluated with a highly accurate DGPS to investigate the performance with respect to lateral vehicle control applications.

Abstract In this paper, a new method for deformable 3D shape registration is proposed. The algorithm computes shape transitions based on local similarity transforms which allows to model not only as‐rigid‐as‐possible deformations but also local and global scale. We formulate an ordinary differential equation (ODE) which describes the transition of a source shape towards a target shape. We assume that both shapes are roughly pre‐aligned (e.g., frames of a motion sequence). The ODE consists of two terms. The first one causes the deformation by pulling the source shape points towards corresponding points on the target shape. Initial correspondences are estimated by closest‐point search and then refined by an efficient smoothing scheme. The second term regularizes the deformation by drawing the points towards locally defined rest positions. These are given by the optimal similarity transform which matches the initial (undeformed) neighborhood of a source point to its current (deformed) neighborhood. The proposed ODE allows for a very efficient explicit numerical integration. This avoids the repeated solution of large linear systems usually done when solving the registration problem within general‐purpose non‐linear optimization frameworks. We experimentally validate the proposed method on a variety of real data and perform a comparison with several state‐of‐the‐art approaches.

We present a novel method for vision-based recovery of three-dimensional structures through simultaneous model reconstruction and camera position tracking from monocular images. Our approach does not rely on robust feature detecting schemes (such as SIFT, Good Features to Track etc.), but works directly on intensity values in the captured images. Thus, it is well-suited for reconstruction of surfaces that exhibit only little texture due to partial homogeneity of the surfaces.

We present a general framework for tracking image regions in two views simultaneously based on sum-of-squared differences (SSD) minimization. Our method allows for motion models up to affine transformations. Contrary to earlier approaches, we incorporate the well-known epipolar constraints directly into the SSD optimization process. Since the epipolar geometry can be computed from the image directly, no prior calibration is necessary. Our algorithm has been tested in different applications including camera localization, wide-baseline stereo, object tracking and medical imaging. We show experimental results on robustness and accuracy compared to the known ground truth given by a conventional tracking device.

Image-based scene representations enable a mobile robot to make a realistic prediction of its environment. Hence, it is able to rapidly detect changes in its surroundings by comparing a virtual image generated from previously acquired reference images and its current observation. This facilitates attentional control to novel events. However, illumination effects can impair attentional control if the robot does not take them into account. To address this issue, we present in this paper an approach for the acquisition of illumination-invariant scene representations. Using multiple spatial image sequences which are captured under varying illumination conditions the robot computes an illumination-invariant image-based environment model. With this representation and statistical models about the illumination behavior, the robot is able to robustly detect texture changes in its environment under different lighting. Experimental results show high-quality images which are free of illumination effects as well as more robust novelty detection compared to state-of-the-art methods.

We propose a system for vision-based estimation of manipulation-relevant properties of objects in natural scenes based on observation of human actions. The system consists of an a-priori (Atlas) knowledge about known generic objects in the scene and classifies the scene into mission relevant objects and background geometry that is important only for collision avoidance. We present the object-centric structure of our system consisting of an Atlas representation and a Working Memory storing the current knowledge about the scene, the manipulated objects and actions applied to them in the local environment. We present experimental results how the system maintains the information in the database and we show the quality of the results that can be obtained with our system.

This paper describes a novel method for motion generation and reactive collision avoidance. The algorithm performs arbitrary desired velocity profiles in absence of external disturbances and reacts if virtual or physical contact is made in a unified fashion with a clear physically interpretable behavior. The method uses physical analogies for defining attractor dynamics in order to generate smooth paths even in presence of virtual and physical objects. The proposed algorithm can, due to its low complexity, run in the inner most control loop of the robot, which is absolutely crucial for safe Human Robot Interaction. The method is thought as the locally reactive real-time motion generator connecting control, collision detection and reaction, and global path planning.

We present a generalization of the Koenderink-van Doorn (KvD) algorithm that allows robust monocular localization with large motion between the camera frames for a wide range of optical systems including omnidirectional systems and standard perspective cameras. The KvD algorithm estimates simultaneously ego-motion parameters, i.e. rotation, translation, and object distances in an iterative way. However due to the linearization of the rotational component of optic flow, the original algorithm fails for larger rotations. We present a generalization of the algorithm to arbitrary rotations that is especially suited for omnidirectional cameras where features can be tracked for long sequences. This reduces the need for vector summation of several individual motion estimates that leads to accumulation of odometry errors. The significant improvement in the performance of the proposed generalized algorithm compared to the original KvD implementation is validated using simulated data. The algorithm is also tested in a real-world experiment with ground-truth data obtained from an external tracking system. The experiment was carried out using a novel compact omnidirectional camera that is designed for small aerial vehicles. It consists of an off-the-shelf webcam that is combined with a reflective surface machined into acrylic glass.

Accurate maps of the static environment are essential for many advanced driver-assistance systems. In this paper a new method for the fast computation of occupancy grid maps with laser range-finders and radar sensors is proposed. The approach utilizes the Graphics Processing Unit to overcome the limitations of classical occupancy grid computation in automotive environments. It is possible to generate highly accurate grid maps in just a few milliseconds without the loss of sensor precision. Moreover, in the case of a lower resolution radar sensor it is shown that it is suitable to apply super-resolution algorithms to achieve the accuracy of a higher resolution laser-scanner. Finally, a novel histogram based approach for road boundary detection with lidar and radar sensors is presented.

One important feature of a cognitive system is to perceive and understand its environment and to adapt its actions to changes and unforeseen situations. In this paper, we propose a scheme for visual surprise detection in cognitive mobile robots. With the robot's observation and a set of reference images previously acquired near its current viewpoint, a pixel-wise surprise trigger is computed using Bayesian probabilistic inference techniques. With appropriate mathematical approximations this algorithm can be implemented on modern graphics hardware which nearly allows for real-time surprise detection. In order to refer to prior observations, a mobile robot has to be able to re-localize itself with respect to its environment. Thus, we also present two online image-based homing algorithms which both facilitate the computation of location-independent surprise triggers. Experiments show acceptable results in terms of robust and fast detection of unexpected changes in the environment.

Accurate online localization is crucial for mobile robotics. In this paper, we describe a real-time image-based localization technique, which is based on a single calibrated camera. This can be supported by a second camera to improve accuracy and to provide the correct translational scale. Our goal is a robust and unbiased pose estimation in highly dynamic scenes on resource-limited systems. The presented approach is characterized through significantly improved robustness of the pose estimation, a novel approach for stereo subpixel accurate landmark initialization, and the speed-up of conventional tracking routines to achieve online capability. Although the algorithm is designed for accurate, online short-range egomotion estimation in hand-held scanning devices, it can be used for any mobile robot application as shown in this paper. Various tests and experimental results with a mobile platform and a hand-held 3D modeler are presented and discussed.

In the context of 3-D scene modeling, this work aims at the accurate estimation of the pose of a close-range 3-D modeling device, in real-time and passively from its own images. This novel development makes it possible to abandon using inconvenient, expensive external positioning systems. The approach comprises an ego-motion algorithm tracking natural, distinctive features, concurrently with customary 3-D modeling of the scene. The use of stereo vision, an inertial measurement unit, and robust cost functions for pose estimation further increases performance. Demonstrations and abundant video material validate the approach.

The term scaffolding, with respect to human education, was first coined in the 1970ies, although the basic concept originates back to the 1930ies. The main idea is to formalize the superior knowledge of a teacher in a certain way to generate support for a trainee. In practice, this concept can be implemented as concrete as a cloze, which assists pupils in learning a foreign language, or it might be as abstract as a social environment, which facilitates learning of specific tasks. This paper introduces a novel approach towards robotic learning by means of such a scaffolding framework. In this case, the scaffolding is constituted by abstract patterns, which facilitate the structuring and segmentation of information during "Learning by Demonstration". The methodology was applied to a real-world scenario of robot-assisted surgery.

For the 3D reconstruction inside historic buildings, we need a marker-free automatic registration approach to align different views together, because GPS does not work indoors and markers are not allowed to paste on the walls. This paper presents an automatic matching process, which employs a novel algorithm, Dynamic Matching Tree technique, for a fast and stable coarse-matching to achieve the automatic pre-alignment of two point clouds and uses modified ICP to do a fine matching efficiently. The whole process can be divided in the following stages: preprocessing, 2-View matching and N-View matching. To validate our method, various experiments has been done on reconstruction of historic sites and industrial objects.

With the rapid development of information and communication technology and widely spreading of internet, digital landscape becomes a high topic recently. This paper presents a landscape modeling- and visualizationssystem. The first part of this paper focuses on the real time 3D-model- lingsprocess out of large point-clouds. Diverse experiments have been done on the reconstruction of various famous regions of Bavaria with tourist features. And the second part concentrates on the interactive online visualizationssystem for massive meshes. To enable an efficient interactive online visualization of these large meshes, we convert them firstly with a multi-resolution hierarchy, and then display them progressively. As a preprocessing for the inter-active rendering, we generate a hierarchical tree of bounding spheres from triangle meshes and write it to disk. It will be used for view frustum culling, backface culling and level-of-detail control. We use a recursive algorithm for display. And the traverse-depth in the tree is decided by the projected size of the current node on the screen. By an interactive rendering, once the user stops moving the mouse, the scene will be redrawn with successively smaller thresholds until a size of one pixel is reached.

We propose a vision-based inertial system that overcomes the problems associated with slow update rates in navigation systems based on high-resolution cameras. Due to bandwidth limitations in current camera interfaces, like Firewire or USB, an increase in the camera resolution results in a drop of the effective frame-rate from the sensor. This increases the correspondence problem for point features in consecutive images due to significant motion of the features. We solve the correspondence problem for very significant motion of point features between consecutive images by analyzing the motion blur to estimate the current motion parameters. The proposed algorithm can be used to track the position of point features in video images originating from slow frame-rate cameras. The system is validated on real images from a camera moved by a mobile system, but it can be used for any type of motion ranging from flying systems to mobile robots operating in outdoor or indoor environments

We present a vision-based system for traffic sign detection and ego-motion estimation in road scenarios. The system is capable of autonomous scene reconstruction and classification. It is used to pre-select candidate surfaces in the vicinity of the road that should be inspected more closely by a sign recognition system. We compare two approaches based on a binocular and a monocular camera system, respectively. We discuss their advantages and disadvantages for applications in driver assistance systems.

We present a novel vision-based approach to simultaneous localization and mapping (SLAM). We discuss it in the context of estimating the 6 DoF pose of a mobile robot from the perception of a monocular camera using a minimum set of three natural landmarks. In contrast to our previously presented V-GPS system, which navigates based on a set of known landmarks, the current approach allows to estimate the required information about the landmarks on-the-fly during the exploration of an unknown environment The method is applicable to indoor and outdoor environments. The calculation is done from the image position of a set of natural landmarks that are tracked in a continuous video stream at frame-rate. An automatic hand-off process allows an update of the set to compensate for occlusions and decreasing reconstruction accuracies with the distance to an imaged landmark. A generic sensor model allows a system configuration with a variety of physical sensors including: monocular perspective cameras, omni-directional cameras and laser range finders.

We present an approach for scale recovery from monocular stereo images of an endoscopic camera with simultaneous registration to dense 3D surface models. We assume the camera motion to be unknown or at least uncertain. An example application is the registration of endoscope images to pre-operative CT scans that allows instrument navigation during surgical procedures. The application field is not restricted to the medical field. It can be extended to registration of monocular video images to laser-based surface reconstructions in, e.g., mobile navigation area or to autonomous aircraft navigation from topological surveys. A novel way for depth estimation from arbitrary camera motion is presented. In this paper, we focus on the robust initialization of the system and on the scale recovery for the reconstructed 3D point clouds with accurate registration to the candidate surfaces extracted from the CT data. We provide experimental validation of the algorithm with data obtained from our experiments with a phantom skull.

We analyze the problem of finding the optimal placement of tracked primitives for robust vision-based control of a mobile robot. The analysis evaluates the properties of the Image Jacobian matrix, used for direct generation of the control signals from the error signal in the image, and the accuracy of the underlying sensor system. The analysis is then used to select optimal tracking primitives that ensure good observability and controllability of the mobile system for a variety of sensor system configurations. The theoretical results are validated with our mobile robot for system configurations that use standard video cameras mounted on a pan-tilt head and catadioptric systems.

We present a novel plane tracking algorithm based on the direct update of surface parameters from two stereo images. The plane tracking algorithm is posed as an optimization problem, and maintains an iteratively re-weighted least squares approximation of the plane's orientation using direct pixel measurements. To facilitate autonomous operation, we include an algorithm for robust detection of significant planes in the environment. The algorithms have been implemented in a robot navigation system.

We present our approach for pose verification with monocular cameras in 3-dimensional space based on the image-based control paradigm. We describe the extensions to our previous control system for mobile navigation that allow us to estimate the complete set of those parameters in space. The major contribution of this approach is a sensor-independent formulation that allows a flexible configuration with a variety of sensor systems including standard cameras, omnidirectional cameras and laser systems. Our second contribution is a way to re-initialize the tracked landmarks during a multi-segment navigation in applications with significant derivations from the pre-taught trajectory as it is the case for handheld systems and flying robots. The presented system can be used as a guidance system for visitors. The localization is based on known landmarks that are in our case natural landmarks in the environment. These landmarks correspond to the satellites of a GPS system. We call it V-GPS (vision-based GPS) because of this similarity in the concept. A camera carried by a person allows to navigate along pre-specified paths through environments, like galleries, hospitals, parks, and other public places.

A long-standing goal in image-based modeling and rendering is to capture a scene from camera images and construct a sufficient model to allow photo-realistic rendering of new views. With the confluence of computer graphics and vision, the combination of research on recovering geometric structure from un-calibrated cameras with modeling and rendering has yielded numerous new methods. Yet, many challenging issues remain to be addressed before a sufficiently general and robust system could be built to (for instance) allow an average user to model their home and garden from camcorder video. This tutorial aims to give researchers and students in computer graphics a working knowledge of relevant theory and techniques covering the steps from real-time vision for tracking and the capture of scene geometry and appearance, to the efficient representation and real-time rendering of image-based models. It also includes hands-on demos of real-time visual tracking, modeling and rendering systems.

Extraction of three-dimensional structure of a scene from stereo images is a problem that has been studied by the computer vision community for decades. Early work focused on the fundamentals of image correspondence and stereo geometry. Stereo research has matured significantly throughout the years and many advances in computational stereo continue to be made, allowing stereo to be applied to new and more demanding problems. We review recent advances in computational stereo, focusing primarily on three important topics: correspondence methods, methods for occlusion, and real-time implementations. Throughout, we present tables that summarize and draw distinctions among key ideas and approaches. Where available, we provide comparative analyses and we make suggestions for analyses yet to be done.

We present a stereo-based obstacle avoidance system for mobile vehicles. The system operates in three steps. First, it models the surface geometry of the supporting surface and removes the supporting surface from the scene. Next, it segments the remaining stereo disparities into connected components in image and disparity space. Finally, it projects the resulting connected components onto the supporting surface and plans a path around them. One interesting aspect of this system is that it can detect both positive and "negative" obstacles (e.g. stairways) in its path. The algorithms we have developed have been implemented on a mobile robot equipped with a real-time stereo system. We present experimental results on indoor environments with planar supporting surfaces that show the algorithms to be both fast and robust.

This paper presents an approach for direct control of a mobile robot to keep it on a pre-taught path based solely on the perception from a monocular CCD camera. In particular, we present a novel vision-based control algorithm for mobile systems equipped with a conventional camera and a pan-tilt head or with an omnidirection camera. This algorithm avoids numerical instabilities of previously reported approaches. The experimental performance of the method as well as its practical limitations are discussed.

This paper proposes an approach for motion planning in indoor environments based on incomplete and uncertain information from a line-based binocular stereo system. The primary goal of the planning process is to plan an optimal path through an unknown or partially known environment, depending on the information gained from exploration and the current mission goal. This paper presents an adaptable motion planner that supports sensor-based map construction, object recognition and navigation in an unknown environment while carrying out a mission. Also presented are some preliminary experimental results that demonstrate the utility of the approach.

This paper presents our approach to retrieve a dependable three-dimensional description of a partially known indoor environment. We describe the way the sensor data from a video camera is preprocessed by contour tracing to extract the boundary lines of the objects and how this information is transformed into a three-dimensional environmental model of the world. We introduce a dynamic map that operates in a closed loop with various sensor systems improving their performance by filtering and contributing certain knowledge. The filtering relies on the capability of a mobile robot to gather sensor readings from different positions. An important part of our approach is the interaction between the dynamic map, storing and filtering the incoming information, and a module predicting missing sensor features based on structures and reference objects. This interaction helps to generate a more accurate model containing also poor detectable features, that are impossible to extract from a single sensor view.