A paradigm shift towards human-centric robotics is driving the development of immersive human-robot interaction, especially in manipulation tasks. This paper describes a human-robot co-manipulation strategy based on Mixed reality-enhanced Active constraint Generation and Impedance Control known as MAGIC. The proposed approach consists of an immersive mixed-reality interface to facilitate visual augmentation and impedance control based active constraints to realize haptic guidance. In our experimental study that involves the analysis of user subjects’ experiences, it can be observed that the improved task performance is over 20% and enhanced path-following accuracy is more than 30%. The usability of MAGIC is further analyzed. In terms of the human cognitive workload based on NASA-TLX, the mode of operation associated with MAGIC outperformed other conventional modes of co-manipulation. In conclusion, the results demonstrated the feasibility and usability of our proposed system paving the path for the future development of smart human-robot interactive functionalities like the auto-generation of an appropriate geometry through scene recognition and spatial understanding for context-specific active constraint.

Unknown dynamic load carrying is one important practical application for quadruped robots. Such a problem is non-trivial, posing three major challenges in quadruped locomotion control. First, how to model or represent the dynamics of the load in a generic manner. Second, how to make the robot capture the dynamics without any external sensing. Third, how to enable the robot to interact with load handling the mutual effect and stabilizing the load. In this work, we propose a general load modeling approach called load characteristics modeling to capture the dynamics of the load. We integrate this proposed modeling technique and leverage recent advances in Reinforcement Learning (RL) based locomotion control to enable the robot to infer the dynamics of load movement and interact with the load indirectly to stabilize it and realize the sim-to-real deployment to verify its effectiveness in real scenarios. We conduct extensive comparative simulation experiments to validate the effectiveness and superiority of our proposed method. Results show that our method outperforms other methods in sudden load resistance, load stabilizing and locomotion with heavy load on rough terrain. Project Page.

Unlike bitmap images, scalable vector graphics (SVG) maintain quality when scaled, frequently employed in computer vision and artistic design in the representation of SVG code. In this era of proliferating AI-powered systems, enabling AI to understand and generate SVG has become increasingly urgent. However, AI-driven SVG understanding and generation (U&G) remain significant challenges. SVG code, equivalent to a set of curves and lines controlled by floating-point parameters, demands high precision in SVG U&G. Besides, SVG generation operates under diverse conditional constraints, including textual prompts and visual references, which requires powerful multi-modal processing for condition-to-SVG transformation. Recently, the rapid growth of Multi-modal Large Language Models (MLLMs) have demonstrated capabilities to process multi-modal inputs and generate complex vector controlling parameters, suggesting the potential to address SVG U&G tasks within a unified model. To unlock MLLM's capabilities in the SVG area, we propose an SVG-centric dataset called UniSVG, comprising 525k data items, tailored for MLLM training and evaluation. To our best knowledge, it is the first comprehensive dataset designed for unified SVG generation (from textual prompts and images) and SVG understanding (color, category, usage, etc.). As expected, learning on the proposed dataset boosts open-source MLLMs' performance on various SVG U&G tasks, surpassing SOTA close-source MLLMs like GPT-4V. We release dataset, benchmark, weights, codes and experiment details on https://ryanlijinke.github.io/.

Robotic micromanipulation systems (RMS) enable precise and repeatable operations under a microscope. Traditional RMS rely solely on microscopic visual feedback, necessitating time-consuming manual positioning to bring the tool tip within the microscope field-of-view (Micro-FOV), which limits efficiency and heavily depends on operator skill. This paper proposes an innovative RMS that integrates macro and micro vision to automate the aforementioned tool tip positioning and facilitate resilient control. The system utilizes an external camera to obtain the macro field-of-view (Macro-FOV), containing the tool and fiducial markers, and estimates the tool tip’s 3D position by triangulation. Visual servoing is then used to guide the tool tip towards the Micro-FOV. Under the Micro-FOV, a tool-sweep detector based on partitioned difference images is used to sequentially locate the tool’s shaft and tip. After auto-focusing, the system executes tool tip and Petri dish resilient control based on our developed self-calibration and self-recalibration mechanisms. During the operation, the system provides an intuitive user interface that includes both macro and micro information, improving the visualization and productivity of micromanipulation. Experiments show that the self-initialization scheme can be implemented across different macro camera viewpoints, reducing the average tip positioning time from 65.70 to 50.08 seconds compared to manual operation, thereby decreasing manual labor intensity and improving efficiency. The self-recalibration mechanism achieves precision and resilient control, with an average error of$0.95~\mu $m over 25 continuous trials. Additionally, the system exhibits robustness against vibration and visual interference, underscoring its potential for diverse biomedical applications.Note to Practitioners—In the biomedical field, robotic micromanipulation systems (RMS) are valued for their high precision and repeatability. Existing research on RMS typically focuses on achieving varying degrees of automation using visual feedback from the microscope, with the assumption that the tool tip is already within the Micro-FOV prior to the task. However, the initial step of moving the tool tip from the Macro-FOV to the Micro-FOV and eventually bringing it to focus usually requires time-consuming manual operation that highly depends on the skill of individual operators. To achieve automatic positioning and resilient control of the tool tip, this paper presents an RMS that integrates macro and micro vision. The system combines visual feedback from a macro camera and a microscope to complete the self-initialization of the system in three steps: macro tip positioning, micro tip positioning, and self-calibration. After self-initialization, the system provides the operator with an intuitive cursor-based user interface. During operation, the system uses visual feedback to detect the tip position and ensures the control accuracy through a self-recalibration mechanism. The proposed self-initialization method has the potential for a wide range of applications. It can be extended to micromanipulation systems with different types of end-effectors, thereby improving the efficiency and precision of micromanipulation.

This article proposes a confidence-based approach for combining two visual tracking techniques to minimize the influence of unforeseen visual tracking failures to achieve uninterrupted vision-based control. Despite research efforts in vision-guided micromanipulation, existing systems are not designed to overcome visual tracking failures, such as inconsistent illumination condition, regional occlusion, unknown structures, and nonhomogenous background scene. There remains a gap in expanding current procedures beyond the laboratory environment for practical deployment of vision-guided micromanipulation system. A hybrid tracking method, which combines motion-cue feature detection and score-based template matching, is incorporated in an uncalibrated vision-guided workflow capable of self-initializing and recovery during the micromanipulation. Weighted average, based on the respective confidence indices of the motion-cue feature localization and template-based trackers, is inferred from the statistical accuracy of feature locations and the similarity score-based template matches. Results suggest improvement of the tracking performance using hybrid tracking under the conditions. The mean errors of hybrid tracking are maintained at subpixel level under adverse experimental conditions while the original template matching approach has mean errors of 1.53, 1.73, and 2.08 pixels. The method is also demonstrated to be robust in the nonhomogeneous scene with an array of plant cells. By proposing a self-contained fusion method that overcomes unforeseen visual tracking failures using pure vision approach, we demonstrated the robustness in our developed low-cost micromanipulation platform.

Automatic targeting of plant cells to perform tasks like extraction of chloroplast is often desired in the study of plant biology. Hence, this paper proposes an improved cell segmentation method combined with a robust tracking algorithm for vision-guided micromanipulation in plant cells. The objective of this work is to develop an automatic plant cell detection and localization technique to complete the automated workflow for plant cell manipulation. The complex structural properties of plant cells make both segmentation of cells and visual tracking of the microneedle immensely challenging, unlike single animal cell applications. Thus, an improved version of watershed segmentation with adaptive thresholding is proposed to detect the plant cells without the need for staining of the cells or additional tedious preparations. To manipulate the needle to reach the identified centroid of the cells, tracking of the needle tip is required. Visual and motion information from two data sources namely, template tracking and projected manipulator trajectory are combined using score-based normalized weighted averaging to continuously track the microneedle. The selection of trackers is influenced by their complementary nature as the former and latter are individually robust against physical and visual uncertainties, respectively. Experimental results validate the effectiveness of the proposed method by detecting plant cell centroids accurately, tracking the microneedle constantly and reaching the plant cell of interest despite the presence of visual disturbances.

Percutaneous Nephrolithotomy is a recommended treatment method for large kidney stone removal. However, the first and most important step, i.e., getting the percutaneous access to create the tract between the targeted calyx and the flank skin, is challenging as the surgeon is often occupied by several tasks at a given time. Therefore, in this paper, we propose a robotic assisted system that collaborates with the surgeon and provides assistance in order for the surgeons to focus on more critical jobs resulting in better surgical performance. A procedure for this robot including three working stages is described. This procedure allows the surgeon to choose a suitable percutaneous target using an ultrasound probe based on his or her experience and the robot will track the respiratory motion of the target kidney stone and insert the needle automatically after the surgeon releases the probe. Experiments are conducted to demonstrate the procedure with the proposed assisted robot for PCNL.

In this paper, an automatic vision-guided micromanipulation approach to facilitate versatile deployment and portable setup is proposed. This paper is motivated by the importance of micromanipulation and the limitations in existing automation technology in micromanipulation. Despite significant advancements in micromanipulation techniques, there remain bottlenecks in integrating and adopting automation for this application. An underlying reason for the gaps is the difficulty in deploying and setting up such systems. To address this, we identified two important design requirements, namely, portability and versatility of the micromanipulation platform. A self-contained vision-guided approach requiring no complicated preparation or setup is proposed. This is achieved through an uncalibrated self-initializing workflow algorithm also capable of assisted targeting. The feasibility of the solution is demonstrated on a low-cost portable microscope camera and compact actuated microstages. Results suggest subpixel accuracy in localizing the tool tip during initialization steps. The self-focus mechanism could recover intentional blurring of the tip by autonomously manipulating it 95.3% closer to the focal plane. The average error in visual servo is less than a pixel with our depth compensation mechanism showing better maintaining of similarity score in tracking. Cell detection rate in a 1637-frame video stream is 97.7% with subpixels localization uncertainty. Our work addresses the gaps in existing automation technology in the application of robotic vision-guided micromanipulation and potentially contributes to the way cell manipulation is performed.

Robotic image-guided micromanipulation contributes towards the ease of operation, speed, accuracy, and repeatability in cell manipulation. However, such technology is not fully exploited because of the challenges in the integration of robotic modules with existing microscope systems, and the difficulty in incorporating robot assistance seamlessly into the workflow. In this paper, we propose a vision-based workflow algorithm termed Detect-Focus-Track-Servo (DFTS). It facilitates easy integration of robotic modules. It also supports user interactions while minimizing the need for manual intervention and disruption to workflow through automatic detection, focusing, tracking and servoing. Experimental results suggest satisfactory detection accuracy of 99.0 % at 70 μm tolerance. The robustness test suggests no difference in the accuracy under blurred and cluttered images. The self-focus algorithm is also demonstrated to bring the tip into focus consistently. The track-servo algorithm achieves low sub-pixel uncertainty. By proposing the DFTS workflow algorithm, we hope that the level of autonomy and ease of deployment in robot and vision modules for micromanipulation can be improved so as to open up new possibilities in the development of robotic image-guided cell manipulation.

In this paper, we propose a workflow algorithm for timely tracking of the tool tip during cell manipulation using a template-based approach augmented with low level feature detection. Doing so addresses the problem of adverse influences on template-based tracking during tool-cell interaction while maintaining an efficient track-servo framework. This consideration is important in developing autonomous robotic vision-guided micromanipulators. Our method facilitates vision-guided micromanipulation autonomously without manual interventions even during tool-cell interaction. This is done by decomposing the process to four scenarios that operate on their respective mode. The self-initializing mode is first used to localize and focus a region of interest (ROI) which the tip lies in. Once in focus, the tip is manipulated using a unified visual track-servo template-based approach. A reinitialization mechanism will be triggered to prevent tracking from being interrupted by partial cell occlusion of the tracking ROI. This mechanism uses the self-initializing concept combining motion cue and low-level feature detection to localize the needle tip. Following the reinitialization, we further recover tracking of the needle tip using a mechanism that updates the base template. This adaptive approach ensures uninterrupted tracking even when the cell is interacting with the tool and under deformation. Results demonstrated that with the newly incorporated mechanisms, the localized position improved from an error of more than 50% to less than 10% of the specimen size. When there is no specimen in the scene the new workflow shows no adverse effect on the localization through 270 tracked frames. By incorporating reinitialization and recovery to this workflow algorithm, we hope to initiate the first step towards uncalibrated autonomous vision-guided micromanipulation process.

Micromanipulation during live microscopic imaging relies heavily on good manual controls, dexterity, and hand-eye coordination. However, unassisted manual operations in these procedures greatly limit the speed, repeatability, and ease of operation. This is especially challenging in the case of microinjection where the insertion path needs to be in precise alignment with the imaging plane to avoid damage to cells. In this paper, we proposed an assistive robotic system that facilitates micromanipulation under microscopy. This comes in the form of intelligent robotic vision and guided manipulation. Using user-selected patch similarity, the system registers target templates and provides online coordinated depth compensation that ensures in-plane microinjection without the need for any prior calibration. This vision-based auto-registration approach readily integrates to any existing microscope system uncalibrated. It can also work as a standalone imaging solution with any general digital microscope camera. Experiments show that the similarity-score based depth compensation performed better than the uncompensated method. The method was shown to self-recover from an unfocused position. By robotizing conventional microscopy and micromanipulation procedures, we hope to address traditional latent needs and open up new possibilities in the ways experimental biology is performed.