Detecting and segmenting damaged wires remains challenging due to varying lighting conditions and the constraints imposed by limited sample datasets, both of which severely degrade detection and segmentation accuracy. To address these challenges, this paper proposes a novel SECA (Squeeze-and-Excitation Channel Attention) mechanism integrated with YOLOv8n, aimed at enhancing key feature extraction under varying lighting conditions. The incorporation of transfer learning strategies further mitigates the challenges posed by insufficient sample datasets. We built an RGB dataset comprising 3,760 images representing diverse wire damage morphologies and expanded the lighting test sets to span six environments, with illuminance levels ranging from 50 to 200,000 lux, to comprehensively evaluate the model's robustness. Experimental results demonstrate that the proposed model markedly improves detection and segmentation precision as well as robustness across varying lighting conditions.

The existing language-driven grasping methods struggle to fully handle ambiguous instructions containing implicit intents. To tackle this challenge, we propose LangGrasp, a novel language-interactive robotic grasping framework. The framework integrates fine-tuned large language models (LLMs) to leverage their robust commonsense understanding and environmental perception capabilities, thereby deducing implicit intents from linguistic instructions and clarifying task requirements along with target manipulation objects. Furthermore, our designed point cloud localization module, guided by 2D part segmentation, enables partial point cloud localization in scenes, thereby extending grasping operations from coarse-grained object-level to fine-grained part-level manipulation. Experimental results show that the LangGrasp framework accurately resolves implicit intents in ambiguous instructions, identifying critical operations and target information that are unstated yet essential for task completion. Additionally, it dynamically selects optimal grasping poses by integrating environmental information. This enables high-precision grasping from object-level to part-level manipulation, significantly enhancing the adaptability and task execution efficiency of robots in unstructured environments. More information and code are available here: https://github.com/wu467/LangGrasp.

Large Language Models (LLMs) are now transforming the way robots learn to work in unpredictable environments, such as homes or small enterprises. A growing number of approaches are combining LLMs with Behavior Trees (BTs). Not only do user commands need to be interpreted into BTs that contain the task’s goal, but external disturbances also need to be handled during the process when BT planners dynamically expand BTs based on action databases. However, in these approaches, the action database is manually pre-built and requires the capability for incremental learning and expansion. To address this issue, we propose a human-in-the-loop learning mechanism. First, we design a context for the LLM and then use it to generate action knowledge through in-context learning. In addition, we introduce human-in-the-loop. User feedback is utilized to guide the LLM to correct and refine the action knowledge, ensuring its accuracy and safety. Finally, the generated action knowledge can be directly used for adaptive manipulation without the need for knowledge transfer effort, enabling the robot to complete tasks and handle external disturbances. Experiments across various tasks are conducted and the experimental results validate our method.

Large Language Models (LLMs) have been widely utilized to perform complex robotic tasks. However, handling external disturbances during tasks is still an open challenge. This paper proposes a novel method to achieve robotic adaptive tasks based on LLMs and Behavior Trees (BTs). It utilizes ChatGPT to reason the descriptive steps of tasks. In order to enable ChatGPT to understand the environment, semantic maps are constructed by an object recognition algorithm. Then, we design a Parser module based on Bidirectional Encoder Representations from Transformers (BERT) to parse these steps into initial BTs. Subsequently, a BTs Update algorithm is proposed to expand the initial BTs dynamically to control robots to perform adaptive tasks. Different from other LLM-based methods for complex robotic tasks, our method outputs variable BTs that can add and execute new actions according to environmental changes, which is robust to external disturbances. Our method is validated with simulation in different practical scenarios.

In this paper, Prioritized Experience Replay (PER) strategy and Long Short Term Memory (LSTM) neural network are introduced to the path planning process of mobile robots, which solves the problems of slow convergence and inaccurate perception of dynamic obstacles with the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. We dubbed this new method as PL-TD3. Firstly, we improve the convergence speed of the algorithm by introducing PER strategy. Secondly, we use LSTM neural network to achieve the improvement of the algorithm for dynamic obstacle perception. In order to verify the method of this paper, we design static environment, dynamic environment and adaptability to dynamic experiments to compare and analyze the methods before and after improvement. The experimental results show that PL-TD3 outperforms TD3 in terms of execution time and execution path length in all environments.

Reinforcement Learning (RL), a method of learning skills through trial-and-error, has been successfully used in many robotics applications in recent years. We combine manipulation primitives (MPs), behavior trees (BTs), and reinforcement learning to propose an algorithm for peg-in-hole tasks, which speeds up the convergence of the RL model and enhance the adaptability of the dynamic environment. Manipulation primitives are used as actions for RL, which can reduce the gap between control instruction and robotic actions and speed up the convergence of the RL model. Behavior trees are used as robot behavior control, which makes robots can actively adapt to the changes in the environment. In experiments, RL-BT, from the combination of RL and BT, is designed for the peg-in-hole task in the Gazebo simulation environment by using UR5 as the actuator. The experiments are conducted on a simple peg and a complex multi-hole peg by three aspects, which include convergence speed verification experiment, adaptability of dynamic environment experiment, and algorithm robustness experiment. The experiment result proves that our RL-BT can speed up the convergence and adapt to the changes in the environment.