Tactile hand gesture recognition is a crucial task for user control in the automotive sector, where Human-Machine Interactions (HMI) demand low latency and high energy efficiency. This study addresses the challenges of power-constrained edge training and inference by utilizing analog Resistive Random Access Memory (ReRAM) technology in conjunction with a real tactile hand gesture dataset. By optimizing the input space through a feature engineering strategy, we avoid relying on large-scale crossbar arrays, making the system more suitable for edge deployment. Through realistic hardware-aware simulations that account for device non-idealities derived from experimental data, we demonstrate the functionalities of our analog ReRAM-based analog in-memory computing for on-chip training, utilizing the state-of-the-art Tiki-Taka algorithm. Furthermore, we validate the classification accuracy of approximately 91.4% for post-deployment inference of hand gestures. The results highlight the potential of analog ReRAM technology and crossbar architecture with fully parallelized matrix computations for real-time HMI systems at the Edge.

Interactive exploration of unknown objects' properties, such as stiffness, mass, center of mass, friction coefficient, and shape, is crucial for autonomous robotic systems operating in unstructured environments. Precise identification of these properties is essential for stable and controlled object manipulation and for anticipating the outcomes of (prehensile or nonprehensile) manipulation actions, such as pushing, pulling, and lifting. Our study focuses on autonomously inferring the physical properties of a diverse set of homogeneous, heterogeneous, and articulated objects using a robotic system equipped with vision and tactile sensors. We propose a novel predictive perception framework to identify object properties by leveraging versatile exploratory actions: nonprehensile pushing and prehensile pulling. A key component of our framework is a novel active shape perception mechanism that seamlessly initiates exploration. In addition, our dual differentiable filtering with graph neural networks learns the object–robot interaction and enables consistent inference of indirectly observable, time-invariant object properties. Finally, we develop a N-step information gain approach to select the most informative actions for efficient learning and inference. Extensive real-robot experiments with planar objects show that our predictive perception framework outperforms state-of-the-art baselines and showcases it in three major applications for object tracking, goal-driven task, and environmental change detection.

Point cloud registration is a fundamental and challenging problem for autonomous robots interacting in unstructured environments for applications such as object pose estimation, simultaneous localization and mapping, robot-sensor calibration, and so on. In global correspondence-based point cloud registration, data association is a highly brittle task and commonly produces high amounts of outliers. Failure to reject outliers can lead to errors propagating to downstream perception tasks. Maximum Consensus (MC) is a widely used technique for robust estimation, which is however known to be NP-hard. Exact methods struggle to scale to realistic problem instances, whereas high outlier rates are challenging for approximate methods. To this end, we propose Graph-based Maximum Consensus Registration (GMCR), which is highly robust to outliers and scales to realistic problem instances. We propose novel consensus functions to map the decoupled MC-objective to the graph domain, wherein we find a tight approximation to the maximum consensus set as the maximum clique. The final pose estimate is given in closed-form. We extensively evaluated our proposed GMCR on a synthetic registration benchmark, robotic object localization task, and additionally on a scan matching benchmark. Our proposed method shows high accuracy and time efficiency compared to other state-of-the-art MC methods and compares favorably to other robust registration methods.

For robotic systems to interact with objects in dynamic environments, it is essential to perceive the physical properties of the objects such as shape, friction coefficient, mass, center of mass, and inertia. This not only eases selecting manipulation action but also ensures the task is performed as desired. However, estimating the physical properties of especially novel objects is a challenging problem, using either vision or tactile sensing. In this work, we propose a novel framework to estimate key object parameters using non-prehensile manipulation using vision and tactile sensing. Our proposed active dual differentiable filtering (ADDF) approach as part of our framework learns the object-robot interaction during non-prehensile object push to infer the object's parameters. Our proposed method enables the robotic system to employ vision and tactile information to interactively explore a novel object via non-prehensile object push. The novel proposed$N$-step active formulation within the differentiable filtering facilitates efficient learning of the object-robot interaction model and during inference by selecting the next best exploratory push actions (where to push? and how to push?). We extensively evaluated our framework in simulation and real-robotic scenarios, yielding superior performance to the state-of-the-art baseline.

Accurate shape reconstruction of transparent ob-jects is a challenging task due to their non-Lambertian surfaces and yet necessary for robots for accurate pose perception and safe manipulation. As vision-based sensing can produce erroneous measurements for transparent objects, the tactile modality is not sensitive to object transparency and can be used for reconstructing the object's shape. We propose AC-TOR, a novel framework for ACtive tactile-based category-level Transparent Object Reconstruction. ACTOR leverages large datasets of synthetic object with our proposed self-supervised learning approach for object shape reconstruction as the collection of real-world tactile data is prohibitively expensive. ACTOR can be used during inference with tactile data from category-level unknown transparent objects for reconstruction. Furthermore, we propose an active-tactile object exploration strategy as probing every part of the object surface can be sample inefficient. We also demonstrate tactile-based category-level object pose estimation task using ACTOR. We perform an extensive evaluation of our proposed methodology with real-world robotic experiments with comprehensive comparison studies with state-of-the-art approaches. Our proposed method outperforms these approaches in terms of tactile-based object reconstruction and object pose estimation.

This paper proposes a novel active visuo-tactile based methodology wherein the accurate estimation of the time-invariant SE(3) pose of objects is considered for autonomous robotic manipulators. The robot equipped with tactile sensors on the gripper is guided by a vision estimate to actively explore and localize the objects in the unknown workspace. The robot is capable of reasoning over multiple potential actions, and execute the action to maximize information gain to update the current belief of the object. We formulate the pose estimation process as a linear translation invariant quaternion filter (TIQF) by decoupling the estimation of translation and rotation and formulating the update and measurement model in linear form. We perform pose estimation sequentially on acquired measurements using very sparse point cloud (≤ 15 points) as acquiring each measurement using tactile sensing is time consuming. Furthermore, our proposed method is computationally efficient to perform an exhaustive uncertainty-based active touch selection strategy in real-time without the need for trading information gain with execution time. We evaluated the performance of our approach extensively in simulation and by a robotic system.

Car manufacturers are facing the challenge of defining suitable sensor setups that cover all requirements for the particular SAE level of automated driving. Besides the sensors' performance and surround-view coverage, other factors like vehicle integration, costs and design aspects need to be taken into account. Additionally, a redundant sensor arrangement and the sensors' sensitivity to environmental influences are of crucial importance for safety. By increasing the degree of automation, vehicles require more external sensors to observe their surrounding environment sufficiently, which raises the variety of setup configurations and the difficulty to identify the optimal one. Concerning the vehicle development process, concepts for sensor setups need to be defined at a very early stage. In this concept stage, it is not feasible to explore every possible sensor arrangement with test drives or to simulate the setup performance with tools used for vehicle validation. Thus, we propose a new simulation-based evaluation method, which allows the configuration of arbitrary sensor setups and enables virtual test drives within specific scenarios to evaluate the setup performance in this early development phase with metrics and key performance indicators. Two different setups are analyzed to demonstrate the results of this evaluation method.

Generating realistic motion in a motion-based (dynamic) driving simulator is challenging due to the limited workspace of the motion system of the simulator compared to the motion range of the simulated vehicle. Motion Cueing Algorithms (MCAs) render accelerations by controlling the motion system of the simulators to provide the driver with a realistic driving experience. Commonly used methods such as Classical Washout-based MCA (CW-MCA) typically achieves suboptimal results due to scaling and filtering, which results in an inefficient usage of the workspace. The Model Predictive Control-based MCA (MPC-MCA) has been shown to achieve superior results and more efficient workspace use. However, it's performance is in practice constrained due to the computationally expensive operations and the requirement of an accurate prediction of future vehicle states. Finally, the Optimal Control (OC) has been shown to provide optimal cueing in an open-loop setup wherein the precalculated control signals are re-played to the driver. However, OC cannot be used in real-time with the driver-in-the-loop. Our work introduces a novel Neural Network-based MCA (NN-MCA), which is trained to imitate the behavior of the OC. After training, the NN-MCA provides an approximated model of the OC, which can run in real-time with the driver in-the-loop, while achieving similar quality. The experiments demonstrate the potential of this approach through objective evaluations of the generated motion-cues on the simulator model and the real simulator. A demonstration video for the performance comparison of the CW-MCA, Optimal-Control-based MCA (OC-MCA) and our proposed method is available at http://go.tum.de/708350.

Tactile sensing is a key sensor modality for robots interacting with their surroundings. These sensors provide a rich and diverse set of data signals that contain detailed information collected from contacts between the robot and its environment. The data are however not limited to individual contacts and can be used to extract a wide range of information about the objects in the environment as well as the actions of the robot during the interactions. In this article, we provide an overview of tactile information and its applications in robotics. We present a hierarchy consisting of raw, contact, object, and action levels to structure the tactile information, with higher-level information often building upon lower-level information. We discuss different types of information that can be extracted at each level of the hierarchy. The article also includes an overview of different types of robot applications and the types of tactile information that they employ. Finally we end the article with a discussion for future tactile applications which are still beyond the current capabilities of robots.

In this paper, we propose a set of novel tactile descriptors to enable robotic systems to extract robust tactile information during tactile object explorations, regardless of the number of the tactile sensors, sensing technologies, type of exploratory movements, and duration of the objects' surface exploration. The performance and robustness of the tactile descriptors are verified by testing on four different sensing technologies (dynamic pressure sensors, accelerometers, capacitive sensors, and impedance electrode arrays) with two robotic platforms (one anthropomorphic hand and one humanoid), and with a large set of objects and materials. Using our proposed tactile descriptors, the Shadow Hand, which has multimodal robotic skin on its fingertips, successfully classified 120 materials (100% accuracy) and 30 in-hand objects (98% accuracy) with regular and irregular textural structure by executing human-like active exploratory movements on their surface. The robustness of the proposed descriptors was assessed further during the large object discrimination with a humanoid. With a large sensing area on its upper body, the humanoid classified 120 large objects with multiple weights and various textures while the objects slid between its sensitive hands, arms, and chest. The achieved 90% recognition rate shows that the proposed tactile descriptors provided robust tactile information from the large number of tactile signals for identifying large objects via their surface texture regardless of their weight.

This paper proposes an online tactile transfer learning strategy for discriminating objects through the surface texture properties via a robotic hand and an artificial robotic skin. The proposed method has the ability to autonomously select and exploit the previously learned multiple texture models while discriminating new textures with a very few available training samples or even one. The experimental results show that employing the proposed method and 10 prior texture models, the robotic hand could discriminate 12 objects via their surface textures with 97% and 100% recognition accuracy with only one and ten training samples respectively. Moreover, the experimental outcomes illustrate that our proposed algorithm is robust against of any negative tactile knowledge transfer.

Open ended learning is a dynamic process based on the continuous analysis of new data, guided by past experience. On one side it is helpful to take advantage of prior knowledge when only few information on a new task is available (transfer learning). On the other, it is important to continuously update an existing model so to exploit the new incoming data, especially if their informative content is very different from what is already known (online learning). Until today these two aspects of the learning process have been tackled separately. In this paper we propose an algorithm that takes the best of both worlds: we consider a sequential learning setting, and we exploit the potentiality of knowledge transfer with a computationally cheap solution. At the same time, by relying on past experience we boost online learning to predict reliably on future problems. A theoretical analysis, coupled with extensive experiments, show that our approach performs well in terms of the online number of training mistakes, as well as in terms of performance on separate test sets.