Recent advances in Vision Language Models (VLMs) have enhanced their application in robotics, encompassing both high-level task planning and low-level action control. Despite their strong performance across various robotic tasks, even for zero-shot scenarios, most VLM applications remain open-loop, adhering to a plan-and-execute paradigm without mechanisms to assess task completion. To address this limitation, we propose GenCo, a Generate-Correct framework designed to automate a peg-in-hole task using a UR5e robot. This framework integrates an VLM-based motion generator and motion expert, working collaboratively to refine and correct actions during robotic task execution. Both VLM agents are fine-tuned using the pre-trained LLaVA, enhancing adaptability and scaling efficiently to diverse tasks. Our experiments demonstrate the adaptiveness of the framework, improving the success rate for the peg-in-hole task by 12.75% compared to a single VLM open-loop method. Notably, in unseen scenarios, the success rate for a triangular peg was increased by 15%, and for a random-shaped peg by 17%, underscoring the system's effectiveness in handling novel tasks. Adaptive testing under varied camera positions demonstrated robust performance, affirming reliability despite shifts in the visual input. The framework is also designed to be lightweight and efficient, facilitating broader adoption and practical deployment. Access to our code and model is provided here: https://github.com/Zhengxuez/generate_correct

This paper presents a novel robot arm that is capable of switching between a rigid robot arm and a continuum robot arm. Therefore, the novel robot arm can perform adaptive physical interaction and manipulation against complex working environments and tasks. The switch-ability of the robot arm is achieved with two types of joints: knee-like flexible joints and continuum flexible joints, with which the continuum segment of the robot arm is capable of locking and losing, hence the degree of freedom of the robot arm is capable to be switched. In this work, kinematics is established for specifying the relationship between joints space and global coordinates in both rigid and continuum configurations. Then, the posture and workspace in rigid and continuum configurations are analyzed and illustrated with numerical simulations, and compared based on the established kinematic model. Finally, a series of preliminary experimental testing toward the joint motion and stiffness has been carried out to validate the design, the kinematic model, and the motion performance of the proposed robot arm. Both the numerical and experimental results show that the knee-like joints can guarantee favorable motion accuracy, and the motion of continuum segment from the testing is well aligned with the motion calculated from the theoretical model. Moreover, the stiffness of rigid configuration is larger than the continuum configuration based on the stiffness experiment results. Therefore, the proposed novel robot arm is capable to handle adaptive interaction and manipulation in a diverse environment through the switching between the rigid and continuum configurations.

Collaborative robots are gradually taking over the leading position in automating the production and manufacturing of the SMEs, where the human-robot collaboration is highly emphasized. Therefore, estimating the force and simulating the performance of robots are of great importance. As a newly introduced technology, digital twin, has gained more attentions for simulation, process evaluation, real-time monitoring, etc. However, the current state-of-the-art of digital twin for robots still remains on the kinematic level, and the integrated robot system dynamics is too complex to be incorporated into the digital twin. Therefore, this research starts with the perspective of harmonic drive based robot joint, and proposes a dynamic model of robot joint by analyzing the composition, transmission principle, and internal interactions. Then the experimental parameter identification is performed to obtain the inherent parameters, which can reflect the system performance characteristics. Finally, a preliminary digital twin of robot joint integrated with dynamic model is established with Gazebo and MATLAB. The proposed approach could be used to simulate the dynamic behavior of robot joint in real time and make contributions to the state of the art for digital twin.

To achieve real-time dynamic simulation analysis and optimization design, a dynamic digital twin of a nonholonomic mobile manipulator (one UR5e mounted on an industrial mobile robot MIR 200) has been developed in this paper. First, the digital twin integrated with dynamics of a mobile manipulator is established. The framework of the dynamic digital twin is presented in detail. Then, the dynamic model of the system has been established with the consideration of the physical interaction between the robot and humans/environments using Lagrange formulation. Finally, the experimental testing has been conducted to validate the dynamic model and evaluate the performances (such as real-time property, accuracy, etc.) of the dynamic digital twin that is integrated with the physical human/environment-robot interaction.

Increasing research attention has been attracted to automatic plug-in charging in an unmanned and dangerous environment. In this work, we develop an object detection solution based on deep learning on 3D point clouds using a mobile robot manipulator to provide mobility and manipulation. In this solution, the 3D point cloud technology is adopted to measure the shapes and depth information for plugin charging. Then the deep learning is employed to deal with the uncertainty in 3D detection, such as inconsistent light conditions, irregular distribution, and structural ambiguity of point clouds. We utilize a mobile robot manipulator carrying a 3D camera and a gripper to detect the targeted objects and automate plug-in charging operations. The proposed 3D object detection principle and procedure for the automatic plug-in charging are presented in detail. The automatic plug-in charging testing is conducted to validate the developed 3D object detection algorithm using a mobile robot manipulator.