Quest2ROS2 is an open-source ROS2 framework for bi-manual teleoperation designed to scale robot data collection. Extending Quest2ROS, it overcomes workspace limitations via relative motion-based control, calculating robot movement from VR controller pose changes to enable intuitive, pose-independent operation. The framework integrates essential usability and safety features, including real-time RViz visualization, streamlined gripper control, and a pause-and-reset function for smooth transitions. We detail a modular architecture that supports ”Side-by-Side” and ”Mirror” control modes to optimize operator experience across diverse platforms. Code is available at: https://github.com/Taokt/Quest2ROS2.

Abstract In order for any learning-based model to be considered reliable, it needs a well-behaved uncertainty or confidence estimate. Most modern neural networks do produce a confidence estimate in the form of their softmax output probability. However, the softmax probability is invalid for out-of-distribution data. Gaussian processes are known to produce a well-behaved confidence estimate that is aware of out-of-distribution samples. Inspired by Gaussian processes, we propose GPify, which combines the softmax probability with a Normalizing Flow in order to add out-of-distribution awareness to the confidence estimate from a neural network. The resulting confidence from GPify is an uncertainty measure that is interpretable and intuitive, while also being probabilistically sound. We evaluate GPify in a selective classification framework, and conclude that it achieves comparable performance to state-of-the-art methods. In addition, we show that GPify has capabilities for detecting adversarial examples, which is a direct improvement over softmax confidence.

Robotic systems for manipulation tasks are increasingly expected to be easy to configure for new tasks. While in the past, robot programs were often written statically and tuned manually, the current, faster transition times call for robust, modular and interpretable solutions that also allow a robotic system to learn how to perform a task. We propose the method Behavior-based Bayesian Optimization and Planning (BeBOP) that combines two approaches for generating behavior trees: we build the structure using a reactive planner and learn specific parameters with Bayesian optimization. The method is evaluated on a set of robotic manipulation benchmarks and is shown to outperform state-of-the-art reinforcement learning algorithms by being up to 46 times faster while simultaneously being less dependent on reward shaping. We also propose a modification to the uncertainty estimate for the random forest surrogate models that drastically improves the results.

Modern robotic perception is highly dependent on neural networks. It is well known that neural network-based perception can be unreliable in real-world deployment, especially in difficult imaging conditions. Out-of-distribution detection is commonly proposed as a solution for ensuring reliability in real-world deployment. Previous work has shown that normalizing flow models can be used for out-of-distribution detection to improve reliability of robotic perception tasks. Specifically, camera parameters can be optimized with respect to the likelihood output from a normalizing flow, which allows a perception system to adapt to difficult vision scenarios. With this work we propose to use the absolute gradient values from a normalizing flow, which allows the perception system to optimize local regions rather than the whole image. By setting up a table top picking experiment with exceptionally difficult lighting conditions, we show that our method achieves a 60% higher success rate for an object detection task compared to previous methods.

When deploying deep neural networks on robots or other physical systems, the learned model should reliably quantify predictive uncertainty. A reliable uncertainty allows downstream modules to reason about the safety of its actions. In this work, we address metrics for uncertainty quantification. Specifically, we focus on regression tasks, and investigate Area Under Sparsification Error (AUSE), Calibration Error (CE), Spearman’s Rank Correlation, and Negative Log-Likelihood (NLL). Using multiple datasets, we look into how those metrics behave under four typical types of uncertainty, their stability regarding the size of the test set, and reveal their strengths and weaknesses. Our results indicate that Calibration Error is the most stable and interpretable metric, but AUSE and NLL also have their respective use cases. We discourage the usage of Spearman’s Rank Correlation for evaluating uncertainties and recommend replacing it with AUSE. • We explore evaluation metrics for uncertainty quantification. • We create toy datasets that highlight different sources of uncertainty. • Using our toy datasets, we compare and contrast metrics for uncertainty. • We evaluate: AUSE, Spearman Correlation, Calibration Error, and NLL. • Results: AUSE, NLL, Calibration error are good metrics with different strengths.

The assignment of products to storage locations significantly impacts the efficiency of warehouse operations. We propose a multi-phase optimizer for a Storage Location Assignment Problem (SLAP) where solution quality is based on a distance estimate of future-forecasted order picking. Candidate assignments are first sampled using a Markov Chain accept/reject method. Future-forecasted pick-rounds are then modified according to the candidate assignments and solved as Traveling Salesman Problems (TSP). The model is graph-based and generalizes to any obstacle layout in 2D. Due to the intractability of the SLAP, methods are proposed to speed up search for strong solution candidates. These include usage of fast function approximation to find potentially strong samples, as well as restarts from local minima. Results show that these methods improve performance and that total travel distance can be reduced by as much as 30% within 8 hours of CPU-time. We share a public repository with SLAP instances and corresponding benchmark results on the generalizable TSPLIB format.

The ability to learn new tasks and quickly adapt to different variations or dimensions is an important attribute in agile robotics. In our previous work, we have explored Behavior Trees and Motion Generators (BTMGs) as a robot arm policy representation to facilitate the learning and execution of assembly tasks. The current implementation of the BTMGs for a specific task may not be robust to the changes in the environment and may not generalize well to different variations of tasks. We propose to extend the BTMG policy representation with a module that predicts BTMG parameters for a new task variation. To achieve this, we propose a model that combines a Gaussian process and a weighted support vector machine classifier. This model predicts the performance measure and the feasibility of the predicted policy with BTMG parameters and task variations as inputs. Using the outputs of the model, we then construct a surrogate reward function that is utilized within an optimizer to maximize the performance of a task over BTMG parameters for a fixed task variation. To demonstrate the effectiveness of our proposed approach, we conducted experimental evaluations on push and obstacle avoidance tasks in simulation and with a real KUKA iiwa robot. Furthermore, we compared the performance of our approach with four baseline methods.

The need for autonomous robot systems in both the service and the industrial domain is larger than ever. In the latter, the transition to small batches or even “batch size 1” in production created a need for robot control system architectures that can provide the required flexibility. Such architectures must not only have a sufficient knowledge integration framework. It must also support autonomous mission execution and allow for interchangeability and interoperability between different tasks and robot systems. We introduce SkiROS2, a skill-based robot control platform on top of ROS. SkiROS2 proposes a layered, hybrid control structure for automated task planning, and reactive execution, supported by a knowledge base for reasoning about the world state and entities. The scheduling formulation builds on the extended behavior tree model that merges task-level planning and execution. This allows for a high degree of modularity and a fast reaction to changes in the environment. The skill formulation based on pre-, hold-and post-conditions allows to organize robot programs and to compose diverse skills reaching from perception to low-level control and the incorporation of external tools. We relate SkiROS2 to the field and outline three example use cases that cover task planning, reasoning, multisensory input, integration in a manufacturing execution system and reinforcement learning.

Order Picking in warehouses is often optimized with a method known as Order Batching, which means thatone vehicle can be assigned to pick a batch of several orders at a time. Although there exists a rich body ofresearch on Order Batching Problem (OBP) optimization, one area which demands more attention is that ofcomputational efficiency, especially for optimization scenarios where warehouses have unconventionallayouts and vehicle capacity configurations. Due to the NP-hard nature of the OBP, computational cost foroptimally solving large instances is often prohibitive. In this paper we compare the performance of twoapproximate optimizers designed for maximum computational efficiency. The first optimizer, Single BatchIterated (SBI), is based on a Seed Algorithm, and the second, Metropolis Batch Sampling (MBS), is based ona Metropolis algorithm. Trade-offs in memory and CPU-usage and generalizability of both algorithms isanalysed and discussed. Existing benchmark datasets are used to evaluate the optimizers on various scenarios.On smaller instances we find that both optimizers come within a few percentage points of optimality atminimal CPU-time. For larger instances we find that solution improvement continues throughout the allottedtime but at a rate which is difficult to justify in many operational scenarios. SBI generally outperforms MBSand this is mainly attributed to the large search space and the latter’s failure to efficiently cover it. Therelevance of the results within Industry 4.0 era warehouse operations is discussed.

The application of robotic solutions to small-batch production is challenging: economical constraints tend to dramatically limit the time for setting up new batches. Organizing robot tasks into modular software components, called skills, and allowing the assignment of multiple concurrent tasks to a single robot is potentially game-changing. However, due to cycle time constraints, it may be necessary for a skill to take over without waiting on another to terminate, and the available literature lacks a systematic approach in this case. In the present article, we fill the gap by (a) establishing the specifications of skills that can be sequenced with partial executions, (b) proposing an implementation based on the combination of finite-state machines and behavior trees, and (c) demonstrating the benefits of such skills through extensive trials in the environment of ARIAC (Agile Robotics for Industrial Automation Competition).

We propose a novel multi-pose loss function to train a neural network for 6D pose estimation, using synthetic data and evaluating it on real images. Our loss is inspired by the VSD (Visible Surface Discrepancy) metric and relies on a differentiable renderer and CAD models. This novel multi-pose approach produces multiple weighted pose estimates to avoid getting stuck in local minima. Our method resolves pose ambiguities without using predefined symmetries. It is trained only on synthetic data. We test on real-world RGB images from the T-LESS dataset, containing highly symmetric objects common in industrial settings. We show that our solution can be used to replace the codebook in a state-of-the-art approach. So far, the codebook approach has had the shortest inference time in the field. Our approach reduces inference time further while a) avoiding discretization, b) requiring a much smaller memory footprint and c) improving pose recall.3

Reinforcement Learning (RL) is a powerful mathematical framework that allows robots to learn complex skills by trial-and-error. Despite numerous successes in many applications, RL algorithms still require thousands of trials to converge to high-performing policies, can produce dangerous behaviors while learning, and the optimized policies (usually modeled as neural networks) give almost zero explanation when they fail to perform the task. For these reasons, the adoption of RL in industrial settings is not common. Behavior Trees (BTs), on the other hand, can provide a policy representation that a) supports modular and composable skills, b) allows for easy interpretation of the robot actions, and c) provides an advantageous low-dimensional parameter space. In this paper, we present a novel algorithm that can learn the parameters of a BT policy in simulation and then generalize to the physical robot without any additional training. We leverage a physical simulator with a digital twin of our workstation, and optimize the relevant parameters with a black-box optimizer. We showcase the efficacy of our method with a 7-DOF KUKAiiwa manipulator in a task that includes obstacle avoidance and a contact-rich insertion (peg-in-hole), in which our method outperforms the baselines.

In the context of future manufacturing lines, removing fixtures will be a fundamental step to increase the flexibility of autonomous systems in assembly and logistic operations. Vision-based 3D pose estimation is a necessity to accurately handle objects that might not be placed at fixed positions during the robot task execution. Industrial tasks bring multiple challenges for the robust pose estimation of objects such as difficult object properties, tight cycle times and constraints on camera views. In particular, when interacting with objects, we have to work with close-range partial views of objects that pose a new challenge for typical view-based pose estimation methods.In this paper, we present a 3D pose estimation method based on a gradient-ascend particle filter that integrates new observations on-the-fly to improve the pose estimate. Thereby, we can apply this method online during task execution to save valuable cycle time. In contrast to other view-based pose estimation methods, we model potential views in full 6dimensional space that allows us to cope with close-range partial objects views. We demonstrate the approach on a real assembly task, in which the algorithm usually converges to the correct pose within 10-15 iterations with an average accuracy of less than 8mm.

Task level programming based on skills has often been proposed as a mean to decrease programming complexity of industrial robots. Several models are based on encapsulating complex motions into self-contained primitive blocks. A semantic skill is then defined as a deterministic sequence of these primitives. A major limitation is that existing frameworks do not support the coordination of concurrent motion primitives with possible interference. This decreases their reusability and scalability in unstructured environments where a dynamic and reactive adaptation of motions is often required. This paper presents a novel framework that generates adaptive behaviors by modeling skills as concurrent motion primitives activated dynamically when conditions trigger. The approach exploits the additive property of motion generators to superpose multiple contributions. We demonstrate the applicability on a real assembly use-case and discuss the gained benefits.

The recent development of calibration algorithms has been driven into two major directions: (1) an increasing accuracy of mathematical approaches and (2) an increasing flexibility in usage by reducing the dependency on calibration objects. These two trends, however, seem to be contradictory since the overall accuracy is directly related to the accuracy of the pose estimation of the calibration object and therefore demanding large objects, while an increased flexibility leads to smaller objects or noisier estimation methods. The method presented in this paper aims to resolves this problem in two steps: First, we derive a simple closed-form solution with a shifted focus towards the equation of translation that only solves for the necessary hand-eye transformation. We show that it is superior in accuracy and robustness compared to traditional approaches. Second, we decrease the dependency on the calibration object to a single 3D-point by using a similar formulation based on the equation of translation which is much less affected by the estimation error of the calibration object's orientation. Moreover, it makes the estimation of the orientation obsolete while taking advantage of the higher accuracy and robustness from the first solution, resulting in a versatile method for continuous hand-eye calibration.

The registration of an observed point set to a known model to estimate its 3D pose is a common task for the autonomous manipulation of objects. Especially in industrial environments, robotic systems need to accurately estimate the pose of objects in order to successfully perform picking, placing or assembly tasks. However, the characteristics of industrial objects often cause difficulties for classical pose estimation algorithms, especially when using IR depth sensors. In this work, we propose to solve ambiguities of the pose estimate by representing the it as a virtual view on a reference model within an adapted particle filter system. Therefore, a simple but fast method to cast views from the reference model is presented, making a training phase obsolete while increasing the accuracy of the estimate. The view-based approach increases the robustness of the registration process and reformulates the pose estimation as a problem of determining the most likely view using a particle filter. By incorporating a local optimizer (ICP) into the dynamics model of the particle filter, the proposed method directs the search in the 6-dimensional pose space, thereby reducing the amount of needed particles to about 50 while decreasing the convergence time to a minimum and therefore making it viable for real-time pose estimation. In contrast to other pose estimation methods, this approach explores the possibilities of sequential pose estimation by only using plain point clouds without additional features.

The requirement of flexibility in the modern industries demands robots that can be efficiently and quickly adapted to different tasks. A way to achieve such a flexible programming paradigm is to instruct robots with task goals and leave planning algorithms to deduct the correct sequence of actions to use in the specific context. A common approach is to connect the skills that realize a semantically defined operation in the planning domain - such as picking or placing an object - to specific executable functions. As a result the skills are treated as independent components, which results into suboptimal execution. In this paper we present an approach where the execution procedures and the planning domain are specified at the same time using solely extended Behavior Trees (eBT), a model formalized and discussed in this paper. At run-time, the robot can use the more abstract skills to plan a sequence using a PDDL planner, expand the sequence into a hierarchical tree, and re-organize it to optimize the time of execution and the use of resources. The optimization is demonstrated on a kitting operation in both simulation and lab environment, showing up to 20% save in the final execution time.

Cognitive robots, able to adapt their actions based on sensory information and the management of uncertainty, have begun to find their way into manufacturing settings. However, the full potential of these robots has not been fully exploited, largely due to the lack of vertical integration with existing IT infrastructures, such as the manufacturing execution system (MES), as part of a large-scale cyber-physical entity. This paper reports on considerations and findings from the research project STAMINA that is developing such a cognitive cyber-physical system and applying it to a concrete and well-known use case from the automotive industry. Our approach allows manufacturing tasks to be performed without human intervention, even if the available description of the environment-the world model-suffers from large uncertainties. Thus, the robot becomes an integral part of the MES, resulting in a highly flexible overall system.

The automation of kitting tasks-collecting a set of parts for one particular car into a kit-has a huge impact in the automotive industry. It considerably increases the automation levels of tasks typically conducted by human workers. Collecting the parts involves picking up objects from pallets and bins as well as placing them in the respective compartments of the kitting box. In this paper, we present a complete system for automated kitting with a mobile manipulator thereby focusing on depalletizing tasks and placing. In order to allow for low cycle times, we present particularly efficient solutions to object perception as well as motion planning and execution. For easy portability to different platforms, all components are integrated into a skill-based framework that is tightly coupled with a task planning component. We present results of experiments at both a research laboratory environment and at the industrial site of PSA Peugeot Citroën serving as a proof of concept for the overall system design and implementation.

In certain cases analytical derivation of physics-based models of robots is difficult or even impossible. A potential workaround is the approximation of robot models from sensor data-streams employing machine learning approaches. In this paper, the inverse dynamics models are learned by employing a novel real-time deep learning algorithm. The algorithm exploits the methods of self-organized learning, reservoir computing and Bayesian inference. It is evaluated and compared to other state of the art algorithms in terms of generalization ability, convergence and adaptability using five datasets gathered from four robots. Results show that the proposed algorithm can adapt to real-time changes of the inverse dynamics model significantly better than the other state of the art algorithms.

In order for manufacturing companies to remain competitive while also offering a high degree of customization for the customers, flexible robots that can be rapidly reprogrammed to new tasks need to be applied in the factories. In this paper we propose a method for the intuitive programming of an industrial mobile robot by combining robot skills, a graphical user interface and human gesture recognition. We give a brief introduction to robot skills as we envision them for intuitive programming, and how they are used in the robot system. We then describe the tracking and gesture recognition, and how the instructor uses the method for programming. We have verified our approach through experiments on several subjects, showing that the system is generally easy to use even for inexperienced users. Furthermore, the programming time required to program a new task is very short, especially keeping traditional industrial robot programming methods in mind.

In this paper we discuss the use of the infinite Gaussian mixture model and Dirichlet processes for learning robot movements from demonstrations. Starting point of this work is an earlier paper where the authors learn a non-linear dynamic robot movement model from a small number of observations. The model in that work is learned using a classical finite Gaussian mixture model (FGMM) where the Gaussian mixtures are appropriately constrained. The problem with this approach is that one needs to make a good guess for how many mixtures the FGMM should use. In this work, we generalize this approach to use an infinite Gaussian mixture model (IGMM) which does not have this limitation. Instead, the IGMM automatically finds the number of mixtures that are necessary to reflect the data complexity. For use in the context of a non-linear dynamic model, we develop a Constrained IGMM (CIGMM). We validate our algorithm on the same data that was used in [5], where the authors use motion capture devices to record the demonstrations. As further validation we test our approach on novel data acquired on our iCub in a different demonstration scenario in which the robot is physically driven by the human demonstrator.

In this paper we address the problem of unsupervised discovery of action classes in video data. Different from all existing methods thus far proposed for this task, we present a space-time link analysis approach which matches the performance of traditional unsupervised action categorization methods in a standard dataset. Our method is inspired by the recent success of link analysis techniques in the image domain. By applying these techniques in the space-time domain, we are able to naturally take into account the spatio-temporal relationships between the video features, while leveraging the power of graph matching for action classification. We present an experiment to demonstrate that our approach is capable of handling cluttered backgrounds, activities with subtle movements, and video data from moving cameras.

A common problem in human movement recognition is the recognition of movements of a particular type (semantic). E.g., grasping movements have a particular semantic (grasping) but the actual movements usually have very different appearances due to, e.g., different grasping directions. In this paper, we develop an exemplar-based parametric hidden Markov model (PHMM) that allows to represent, e.g., movements of a particular type and that compensates for the different appearances and parameterizations of that movement. The PHMM is based on exemplar movements that have to be ”demonstrated ” to the system. Recognition and synthesis are carried out through locally linear interpolation of the exemplar movements. For a meaningful interpolation, the exemplars have to be in sync, what exhibits certain problems that are resolved in this paper. In our experiments we combine our PHMM approach with our 3D body tracker. Experiments are performed with pointing and grasping movements. Synthesis for grasping is parameterized by the positions of the objects to be grasped. In case of recognition, our approach is able to recover the position of an object at which a human volunteer is pointing. Our experiments show the flexibility of the PHMMs in terms of the amount of training data and its robustness in terms of noisy observation data. In addition, we

We propose a view-based approach to recognize humans from their gait. Two different image features have been considered: the width of the outer contour of the binarized silhouette of the walking person and the entire binary silhouette itself. To obtain the observation vector from the image features, we employ two different methods. In the first method, referred to as the indirect approach, the high-dimensional image feature is transformed to a lower dimensional space by generating what we call the frame to exemplar (FED) distance. The FED vector captures both structural and dynamic traits of each individual. For compact and effective gait representation and recognition, the gait information in the FED vector sequences is captured in a hidden Markov model (HMM). In the second method, referred to as the direct approach, we work with the feature vector directly (as opposed to computing the FED) and train an HMM. We estimate the HMM parameters (specifically the observation probability B) based on the distance between the exemplars and the image features. In this way, we avoid learning high-dimensional probability density functions. The statistical nature of the HMM lends overall robustness to representation and recognition. The performance of the methods is illustrated using several databases.

Most present face recognition approaches recognize faces based on still images. We present a novel approach to recognize faces in video. In that scenario, the face gallery may consist of still images or may be derived from a videos. For evidence integration we use classical Bayesian propagation over time and compute the posterior distribution using sequential importance sampling. The probabilistic approach allows us to handle uncertainties in a systematic manner. Experimental results using videos collected by NIST/USF and CMU illustrate the effectiveness of this approach in both still-to-video and video-to-video scenarios with appropriate model choices.

In this article we want to introduce first the Gabor wavelet network as a model based approach for an effective and efficient object representation. The Gabor wavelet network has several advantages such as invariance to some degree with respect to translation, rotation and dilation. Furthermore, the use of Gabor filters ensured that geometrical and textural object features are encoded. The feasibility of the Gabor filters as a model for local object features ensures a considerable data reduction while at the same time allowing any desired precision of the object representation ranging from a sparse to a photo-realistic representation. In the second part of the paper we will present an approach for the estimation of a head pose that is based on the Gabor wavelet networks. 1 Introduction Recently, model-based approaches for the recognition and the interpretation of images of variable objects, like the bunch graph approach, PCA, eigenfaces and active appearance models, have re...

We present a method for visual face tracking that is based on a wavelet representation of a face template. The wavelet representation allows: arbitrary affine deformations of the facial image; generalization from an individual face template to a rather general face template; and adapting the computational needs of the tracking algorithm to the computational resources available. The method presented was implemented on a Linux Pentium 450 MHz and runs off-line with 25 Hz, and online using an active camera mount at 22 Hz. We present experimental results on the off-line tests on several common image sequences including the salesman-sequence as well as on the online tests.