Robotic assembly tasks remain an open challenge due to their long horizon nature and complex part relations. Behavior trees (BTs) are increasingly used in robot task planning for their modularity and flexibility, but creating them manually can be effort-intensive. Large language models (LLMs) have recently been applied to robotic task planning for generating action sequences, yet their ability to generate BTs has not been fully investigated. To this end, we propose LLM-as-BT-Planner, a novel framework that leverages LLMs for BT generation in robotic assembly task planning. Four in-context learning methods are introduced to utilize the natural language processing and inference capabilities of LLMs for producing task plans in BT format, reducing manual effort while ensuring robustness and comprehensibility. Additionally, we evaluate the performance of fine-tuned smaller LLMs on the same tasks. Experiments in both simulated and real-world settings demonstrate that our framework enhances LLMs' ability to generate BTs, improving success rate through in-context learning and supervised fine-tuning.

Large Language Models (LLMs) have gained popularity in task planning for long-horizon manipulation tasks. To enhance the validity of LLM-generated plans, visual demonstrations and online videos have been widely employed to guide the planning process. However, for manipulation tasks involving subtle movements but rich contact interactions, visual perception alone may be insufficient for the LLM to fully interpret the demonstration. Additionally, visual data provides limited information on force-related parameters and conditions, which are crucial for effective execution on real robots. In this paper, we introduce LEMMo-Plan, an in-context learning framework that incorporates tactile and force-torque information from human demonstrations to enhance LLMs' ability to generate plans for new task scenarios. We propose a bootstrapped reasoning pipeline that sequentially integrates each modality into a comprehensive task plan. This task plan is then used as a reference for planning in new task configurations. Real-world experiments on two different sequential manipulation tasks demonstrate the effectiveness of our framework in improving LLMs' understanding of multi-modal demonstrations and enhancing the overall planning performance. More materials are available on our project website: lemmo-plan.github.io/LEMMo-Plan/.

Assembly is a crucial skill for robots in both modern manufacturing and service robotics. However, mastering transferable insertion skills that can handle a variety of high-precision assembly tasks remains a significant challenge. This paper presents a novel framework that utilizes diffusion models to generate 6D wrench for high-precision tactile robotic insertion tasks. It learns from demonstrations performed on a single task and achieves a zero-shot transfer success rate of 95.7% across various novel high-precision tasks. Our method effectively inherits the self-adaptability demonstrated by our previous work. In this framework, we address the frequency misalignment between the diffusion policy and the real-time control loop with a dynamic system-based filter, significantly improving the task success rate by 9.15%. Furthermore, we provide a practical guideline regarding the trade-off between diffusion models' inference ability and speed.

Optimal path planning requires finding a series of feasible states from the starting point to the goal to optimize objectives. Popular path planning algorithms, such as Effort Informed Trees (EIT*), employ effort heuristics to guide the search. Effective heuristics are accurate and computationally efficient, but achieving both can be challenging due to their conflicting nature. This paper proposes Direction Informed Trees (DIT*), a sampling-based planner that focuses on optimizing the search direction for each edge, resulting in goal bias during exploration. We define edges as generalized vectors and integrate similarity indexes to establish a directional filter that selects the nearest neighbors and estimates direction costs. The estimated direction cost heuristics are utilized in edge evaluation. This strategy allows the exploration to share directional information efficiently. DIT* convergence faster than existing single-query, sampling-based planners on tested problems in$\mathbb{R}^{4}$to$\mathbb{R}^{16}$and has been demonstrated in real-world environments with various planning tasks. A video showcasing our experimental results is available at: https://youtu.be/2SX6QT2NOek.

This paper introduces Context Informed Trees (CIT*), a sampling-based motion planning algorithm that enhances exploration efficiency by biasing sampling based on uncertainty estimation from local samples and connectivity information obtained during the search process. CIT* is based on Flexible Informed Trees (FIT*) and incorporates three key components: region-based sampling, uncertainty-driven weighting, and connection-greedy prioritization (CGP). It generates regions from sampled states based on local obstacle proximity, assigning weights to these regions using probability uncertainty estimation via kernel density estimation (KDE) classification. To further refine the sampling focus, CGP prioritizes regions that exhibit strong connectivity in previous searches, ensuring that exploration is directed toward unknown and critical areas that have a higher likelihood of contributing to feasible and efficient paths. The sampling process is then guided by a mixture of Gaussian distributions centered on weighted regions, where the weighting biases sampling toward more critical regions, thereby improving search efficiency and accelerating convergence. Benchmark evaluations demonstrate that CIT* improves efficiency by reducing reliance on random sampling, which often leads to slower solution discovery and higher path costs. With biased sampling, CIT* maintains strong performance in solving complex motion planning problems in ${\mathbb{R}^4}$ to ${\mathbb{R}^{16}}$ and has been demonstrated on a real-world manipulation task. A video showcasing our method and experimental results is available at: https://youtu.be/SG2cy9WmjD0.

Path planning in robotics often involves solving continuously valued, high-dimensional problems. Popular informed approaches include graph-based searches, such as A*, and sampling-based methods, such as Informed RRT*, which utilize informed set and anytime strategies to expedite path optimization incrementally. Informed sampling-based planners define informed sets as subsets of the problem domain based on the current best solution cost. However, when no solution is found, these planners re-sample and explore the entire configuration space, which is time-consuming and computationally expensive. This article introduces Multi-Informed Trees (MIT*), a novel planner that constructs estimated informed sets based on prior admissible solution costs before finding the initial solution, thereby accelerating the initial convergence rate. Moreover, MIT* employs an adaptive sampler that dynamically adjusts the sampling strategy based on the exploration process. Furthermore, MIT* utilizes length-related adaptive sparse collision checks to guide lazy reverse search. These features enhance path cost efficiency and computation times while ensuring high success rates in confined scenarios. Through a series of simulations and real-world experiments, it is confirmed that MIT* outperforms existing single-query, sampling-based planners for problems in$\mathbb {R}^{4}$to$\mathbb {R}^{16}$and has been successfully applied to real-world robot manipulation tasks. A video showcasing our experimental results is available at:https://youtu.be/30RsBIdexTUNote to Practitioners—The motivation for this work stems from the challenges faced by existing informed path planners in high-dimensional, continuously valued environments, particularly when an initial feasible solution is difficult to find. Traditional asymmetric bidirectional planners rely on the best current solution to define problem subsets. When a lazy path has been found through lazy reverse search, these planners tend to re-sample and explore the entire problem space, which could hinder the path planning process. Our proposed MIT* algorithm addresses this issue by constructing an estimated informed set based on prior admissible solution costs before finding the initial solution. This estimated set helps to narrow the search area, thereby accelerating the initial convergence rate. MIT* also integrates an adaptive sampling strategy that dynamically adjusts based on the ongoing exploration process, enhancing the planner’s ability to efficiently navigate through challenging spaces. Furthermore, MIT* employs adaptive sparse collision checks, which guide the lazy reverse search that balances computational efficiency with accuracy in pathfinding. The proposed algorithm can be applied to industrial robots, humanoid robots, or service robots to achieve efficient path planning.

Controlling the shape of deformable linear objects using robots and constraints provided by environmental fixtures has diverse industrial applications. In order to establish robust contacts with these fixtures, accurate estimation of the contact state is essential for preventing and rectifying potential anomalies. However, this task is challenging due to the small sizes of fixtures, the requirement for real-time performances, and the infinite degrees of freedom of the deformable linear objects. In this paper, we propose a real-time approach for estimating both contact establishment and subsequent changes by leveraging the dependency between the applied and detected contact force on the deformable linear objects. We seamlessly integrate this method into the robot control loop and achieve an adaptive shape control framework which avoids, detects and corrects anomalies automatically. Real-world experiments validate the robustness and effectiveness of our contact estimation approach across various scenarios, significantly increasing the success rate of shape control processes.

Flexible manufacturing lines are required to meet the demand for customized and small batch-size products. Even though state-of-the-art tactile robots may provide the versatility for increased adaptability and flexibility, their potential is yet to be fully exploited. To support robotics deployment in manufacturing, we propose a task-based tactile robot programming paradigm that uses an object-centric tactile skill definition that directly links identified object constraints of the task to the definition of constraint-based unified force-impedance control. In this study, we first explain the basic concept of abstracting the task constraints experienced by the object and transferring them to the robot’s operational space frame. Second, using the object-centric tactile skill definition, we synthesize unified force-impedance control and formalized holonomic constraints to enable flexible task execution. Later, we propose the quantified analysis metrics for the process by analyzing them as a typical example of flexible manipulation disassembly skills, e.g., levering and unscrew-driving regarding their object requirements. Supported by realistic experimental evaluation using a Franka Emika robot, our tactile robot programming approach for the direct translation between task-level constraints and robot control parameter design is shown to be a viable solution for increased robotic deployment in flexible manufacturing lines.

Insertion is an essential skill for robots in both modern manufacturing and services robotics. In our previous study, we proposed an insertion skill framework based on forcedomain wiggle motion. The main limitation of this method lies in the robot’s inability to adjust its behavior according to changing contact state during interaction. In this paper, we extend the skill formalism by incorporating a behavior tree-based primitive switching mechanism that leverages highfrequency tactile data for the estimation of contact state. The efficacy of our proposed framework is validated with a series of experiments that involve the execution of tightly constrained peg-in-hole tasks. The experiment results demonstrate a significant improvement in performance, characterized by reduced execution time, heightened robustness, and superior adaptability when confronted with unknown tasks. Moreover, in the context of transfer learning, our paper provides empirical evidence indicating that the proposed skill framework contributes to enhanced transferability across distinct operational contexts and tasks.

Despite recent advancements in torque-controlled tactile robots, integrating them into manufacturing settings remains challenging, particularly in complex environments. Simplifying robotic skill programming for non-experts is crucial for increasing robot deployment in manufacturing. This work proposes an innovative approach, Vision-Augmented Unified Force-Impedance Control (VA-UFIC), aimed at intuitive visuo-tactile exploration of unknown 3D curvatures. VA-UFIC stands out by seamlessly integrating vision and tactile data, enabling the exploration of diverse contact shapes in three dimensions, including point contacts, flat contacts with concave and convex curvatures, and scenarios involving contact loss. A pivotal component of our method is a robust online contact alignment monitoring system that considers tactile error, local surface curvature, and orientation, facilitating adaptive adjustments of robot stiffness and force regulation during exploration. We introduce virtual energy tanks within the control framework to ensure safety and stability, effectively addressing inherent safety concerns in visuo-tactile exploration. Evaluation using a Franka Emika research robot demonstrates the efficacy of VA-UFIC in exploring unknown 3D curvatures while adhering to arbitrarily defined force-motion policies. By seamlessly integrating vision and tactile sensing, VA-UFIC offers a promising avenue for intuitive exploration of complex environments, with potential applications spanning manufacturing, inspection, and beyond.

The intersection of robotics and artificial intelligence led to a profound paradigm shift in Robot Learning. Robots have the capacity to replicate human actions and also dynamically adapt, innovate, and excel across a spectrum of tasks. However, the heterogeneity in the deployment of robot platforms and software frameworks poses considerable challenges in terms of systematic testing and comparative analyses. Additionally, the data scarcity of especially force controlled robot manipulation is still restraining the development of advanced foundation models. A reference platform with default software stack can help to increase comparability, reducing development time and collect a large amount of tactile robot manipulation data. To address on this problem, we developed a Parallel and Distributed Robot AI (PD.RAI) framework, comprising a scalable ensemble of Robot Learning Units (RLUs), a global database, and the Robot Cluster Intelligence (RoCI). Each RLU is endowed with robot arms, cameras, and local computational units to autonomously engage in planning, control, and local machine learning of tactile manipulation skills. The RoCI system oversees the learning process and schedules the RLUs tasks. To show the functionality of the system, two black-box optimization algorithms are compared within the robot skill learning domain. An experiment with 24 different optimization tasks is conducted in parallel. The algorithms are incorporated into the same existing default modules acting as a reference environment. This allows for a realistic comparison without sacrificing diversity of possible configurations and testing environments.

In path planning, anytime almost-surely asymptotically optimal planners dominate the benchmark of sampling-based planners. A notable example is Batch Informed Trees (BIT*), where planners iteratively determine paths to batches of vertices within the exploration area. However, utilizing a consistent batch size is inefficient for initial pathfinding and optimal performance, it relies on effective task allocation. This paper introduces Flexible Informed Trees (FIT*), a sampling-based planner that integrates an adaptive batch-size method to enhance the initial path convergence rate. FIT* employs a flexible approach in adjusting batch sizes dynamically based on the inherent dimension of the configuration spaces and the hypervolume of the n-dimensional hyperellipsoid. By applying dense and sparse sampling strategy, FIT* improves convergence rate while finding successful solutions faster with lower initial solution cost. This method enhances the planner’s ability to handle confined, narrow spaces in the initial finding phase and increases batch vertices sampling frequency in the optimization phase. FIT* outperforms existing single-query, sampling-based planners on the tested problems in R2to R8, and was demonstrated on a real-world mobile manipulation task.

Path planning has long been an important and active research area in robotics. To address challenges in high-dimensional motion planning, this study introduces the Force Direction Informed Trees (FDIT*), a sampling-based planner designed to enhance speed and cost-effectiveness in pathfinding. FDIT* builds upon the state-of-the-art informed sampling planner, the Effort Informed Trees (EIT*), by capitalizing on often-overlooked information in invalid vertices. It incorporates principles of physical force, particularly Coulomb’s law. This approach proposes the elliptical k-nearest neighbors search method, enabling fast convergence navigation and avoiding high solution cost or infeasible paths by exploring more problem-specific search-worthy areas. It demonstrates benefits in search efficiency and cost reduction, particularly in confined, high-dimensional environments. It can be viewed as an extension of nearest neighbors search techniques. Fusing invalid vertex data with physical dynamics facilitates force-direction-based search regions, resulting in an improved convergence rate to the optimum. FDIT* outperforms existing single-query, sampling-based planners on the tested problems in ℝ4to ℝ16and has been demonstrated on a real-world mobile manipulation task.

The rising demand for customized products necessitates the integration of multiple robotic systems, underscoring the need for advanced production planning and scheduling. This paper introduces an ontology-based, artificial intelligence-enhanced method for dynamic task planning and scheduling, aimed at improving the efficiency of production process, reducing machine downtime, and consequently increasing throughput in assembly operations. Designed to generate and execute feasible production plans dynamically, this method minimizes manual planning and scheduling efforts. We evaluate its effectiveness using two gear assembly use cases with various robot skills, highlighting its flexibility in planning and scheduling and its contributions to the evolution of smart manufacturing. The method’s adaptability suggests its applicability across diverse smart factory environments.

Studying the manipulation of deformable linear objects has significant practical applications in industry, including car manufacturing, textile production, and electronics automation. However, deformable linear object manipulation poses a significant challenge in developing planning and control algorithms, due to the precise and continuous control required to effectively manipulate the deformable nature of these objects. In this paper, we propose a new framework to control and maintain the shape of deformable linear objects with two robot manipulators utilizing environmental contacts. The framework is composed of a shape planning algorithm which automatically generates appropriate positions to place fixtures, and an object-centered skill engine which includes task and motion planning to control the motion and force of both robots based on the object status. The status of the deformable linear object is estimated online utilizing visual as well as force information. The framework manages to handle a cable routing task in real-world experiments with two Panda robots and especially achieves contact-aware and flexible clip fixing with challenging fixtures.

Teleoperated robots are believed to play an important role for future applications in industry, medicine and other domains. Examples for this are remote assembly and maintenance, surgery, diagnosis or deep-sea and space exploration. Such applications are made possible by state-of-the-art tactile manipulators, well-researched control schemes and novel communication technologies such as the fifth generation of mobile communication (5G). The achievable performance is highly dependent on the communication delay and thus on the distance between leader and follower station, as well as the potentially used wireless protocol. Specially in this regard, 5G is a promising technology compared to the other communication protocols for transferring tactile information. In this paper, we introduce our telepresence reference platform, which can be used for empirical evaluation of different algorithms and communications. Comparative analysis are conducted to capture the influence of wireless communication protocols on telepresence systems consisting of complex robotic arms. The experiment compares the influence of 5G, LTE and WiFi communication protocols with regard to the motion and force tracking performance of the system.

Compliance in actuation has been exploited to generate highly dynamic maneuvers such as throwing that take advantage of the potential energy stored in joint springs. However, the energy storage and release could not be well-timed yet. On the contrary, for multi-link systems, the natural system dynamics might even work against the actual goal. With the introduction of variable stiffness actuators, this problem has been partially addressed. With a suitable optimal control strategy, the approximate decoupling of the motor from the link can be achieved to maximize the energy transfer into the distal link prior to launch. However, such continuous stiffness variation is complex and typically leads to oscillatory swing-up motions instead of clear launch sequences. To circumvent this issue, we investigate decoupling for speed maximization with a dedicated novel actuator concept denoted Bi-Stiffness Actuation. With this, it is possible to fully decouple the link from the joint mechanism by a switch-and-hold clutch and simultaneously keep the elastic energy stored. We show that with this novel paradigm, it is not only possible to reach the same optimal performance as with power-equivalent variable stiffness actuation, but even directly control the energy transfer timing. This is a major step forward compared to previous optimal control approaches, which rely on optimizing the full time-series control input.

Energy efficiency is a crucial issue towards long-term deployment of compliant robots in the real world. In the context of variable impedance actuators (VIAs), one of the main focuses has been on improving energy efficiency through reduction of energy consumption. However, the harvesting of dissipated energy in such systems remains under-explored. This study proposes a variable damping module design enabling energy regeneration in VIAs by exploiting the regenerative braking effect of dc motors. The proposed damping module uses four switches to combine regenerative and dynamic braking, in a hybrid approach that enables energy regeneration without a reduction in the range of damping achievable. A physical implementation on a simple VIA mechanism is presented in which the regenerative properties of the proposed module are characterized and compared against theoretical predictions. To investigate the role of variable regenerative damping in terms of energy efficiency of long-term operation, experiments are reported in which the VIA, equipped with the proposed damping module, performs sequential reaching to a series of stochastic targets. The results indicate that the combination of variable stiffness and variable regenerative damping results in a 25% performance improvement on metrics incorporating reaching accuracy, settling time, energy consumption and regeneration over comparable schemes, where either stiffness or damping are fixed.

The low-cost manufacturing and maintenance of prostheses is of vital importance to their successful deployment in developing countries. Low-cost prosthesis actuation is generally achieved by combining pre-programmed control strategies, with surface-electromyographic measurements taken from the residual limb. In a standard setting, these signals are measured with disposable gel electrodes. However, this limit on electrode reuse requires that prosthesis users have a stable supply of electrodes. Alternatively, the textile electrodes sewn from conductive thread are studied in the context of hand gesture recognition to consider their future use with low-cost prostheses. In this paper, it is demonstrated that textile electrodes can be applied for gesture recognition. To do so, surface electromyography (sEMG) experiments are run in South Africa on three amputees where they were asked to perform gestures with their phantom limb (i.e., the missing limb segment). A gesture recognition method is implemented, and the classification accuracy with data recorded from textile electrodes is compared to that from gel electrodes. Further analysis examining the relationship between classifier performance and physiological parameters are performed. Results show that textile electrodes can be used to perform accurate gesture recognition, and are comparable to disposable gel electrodes. This demonstrates that low-cost sensory systems are not barrier to myoelectric control in developing countries.

Increasing research efforts have been made to improve the energy efficiency of variable impedance actuators (VIAs) through reduction of energy consumption. However, the harvesting of dissipated energy in such systems remains under-explored. This study proposes a novel variable damping module design enabling energy regeneration in VIAs by exploiting the regenerative braking effect of DC motors. The proposed damping module uses four switches to combine regenerative and dynamic braking, in a hybrid approach that enables energy regeneration without reduction in the range of damping achievable. Numerical simulations and a physical experiment are presented in which the proposed module shows an optimal trade-off between task-performance and energy efficiency.

This paper presents a programming by demonstration framework for teaching impedance modulation using human demonstrations. Physiologically, human stiffness and damping are coupled at the muscle level, restricting the ability to modulate impedance according to task demands. Robotic systems often do not have this restriction (stiffness and damping can be varied independently), but the challenge is to devise an appropriate variable impedance profile for a given task. In this paper, the task critical component is first learned for imitation and a robot-specific controller is then blended into the control using the null space. In doing so, the control cheme takes advantage of both human and robot `best practice'. Experimental results on a physical robot suggest an order of magnitude better mean performance, with lower variance, can be achieved using the blended scheme.