Magnetic microrobots (MMs) have emerged as promising tools for targeted therapies, including non-invasive in vivo treatments and precise drug delivery, owing to their untethered controllability and biocompatibility. Current actuation strategies for MMs primarily rely on two magnetic field (MF) generation approaches: gradient-based and rotational methods. Unlike the gradient method, rotational actuation enables efficient manipulation of MMs under significantly weaker magnetic fields. To fully leverage the potential of rotationally driven MMs, a comprehensive understanding of their fundamental spin motility is essential. Achieving accurate characterization of these MMs necessitates the development of an MF generation system equipped with rapid motion-tracking and broad-range measurement capabilities. This study proposes a high-speed rotating states observation scheme by developing a tracking-based optimal local imaging and estimation scheme, simultaneously meeting the broad-range observation capability and the high imaging speed requirement. Specifically, the CSR-DCF tracking method is adopted to detect the MM’s location, and based on this, the observation system adjusts the imaging region optimally. An estimation scheme based on the asynchronous rectification method is derived to measure the MM rotating states consistently using measured MF data and local optical images of the target. Experimental studies are carried out to validate the effectiveness of the proposed scheme.

Efficient motion planning with the error tolerance is crucial for dynamic robotic tasks, particularly robotic table tennis. This task demands simultaneous high efficiency and error tolerance. First, the incoming ball’s high speed allows only tens of milliseconds for motion planning. Second, two types of errors, namely, ball-paddle motion uncertainty and joint limit violation, must be tolerated to ensure a high success rate of planning (SRP) and striking. This article proposes an advanced joint planning framework designed for high efficiency and error tolerance. To tolerate the error of ball-paddle motion uncertainty, this work introduces a joint classification criterion according to the joint motion characteristics. To tolerate the error of joint limit violation, based on the classification criterion, this study also develops a robust reference trajectory generation scheme, named error-tolerant-variable-sigmoid-based motion template (ETVSMT), to fully consider the motion capabilities of different joints. The ETVSMT approach utilizes the variable-sigmoid-based motion template (VSMT) as the backbone and designs its basic and remedial portions to tolerate the error of hard joint limit violation. The implementation of the ETVSMT scheme results in an average SRP of 98% and a ball-striking success rate of up to 90%, with execution time on an Intel Xeon CPU as low as 15 ms. Furthermore, the proposed method ensures the outgoing ball lands on the opposite side of the table with an average landing error of 19.62 cm and a net-passing height error of 12.68 cm. The proposed method can also benefit other dynamic tasks like human–robot interaction to improve the planning efficiency.

Abstract Deep reinforcement learning has significantly advanced robot manipulations by providing an alternative solution for designing control strategies using raw images as direct inputs. While images offer additional environmental information, the end-to-end policy training manner (from image to action) requires simultaneous representation and task learning by the agent. This often necessitates a substantial number of interaction samples to achieve satisfactory policy performance. Previous works has attempted to address this challenge by learning a visual representation model that encodes the entire image into a low-dimensional vector before the policy training. However, since this vector contains both robot and object information, it inevitably introduces coupling within the state, which can mislead the policy training process. In this study, a novel method called Reinforcement Learning with Decoupled State Representation is proposed to effectively decouple robot and object information within the state representation. Experimental results demonstrate that the proposed method exhibits faster learning speed and achieves superior performance compared to previous methods across various robot manipulation tasks. Moreover, with only 3096 offline images, the proposed method successfully applies to real-world robot pushing tasks, which demonstrates its high practicability.

To date, untethered micro/nanorobots have attracted considerable attention in various aspects due to their unique potential for in-vivo applications such as the targeted therapy. One of the most promising types of micro/nanorobots is the class of ferromagnetic microrobots which can be efficiently actuated via gradient/rotational magnetic field generated by less costly electromagnetic coil systems. For performing successful operations, locomotion control of the magnetic microrobots is non-trivial. Modern controllers commonly require motion status-based feedback. To fully utilize those advanced approaches, motion state of one microrobot should be supplied, however it is still challenging in cases. It is noted that, during locomotion, one ferromagnetic microrobot can combine with others to form an unstructured larger one, namely growing magnetic microrobot (GMM), whose dynamic behavior keeps changing, and thus the model-based observers are never applicable. Besides, tracking and estimating states of those unstructured time-varying GMMs in complex surroundings are always challenging, especially for an uneven sampling scenario. In order to accurately estimate the GMM motion status in a complex environment via micro-vision, this study develops an indeterministic observer leveraging on the approach of discriminative correlation filter with channel/spatial reliability (CSR-DCF) and the variable-step finite-time sliding mode (FSM-V) state estimation theory. Experimental study verifies that the proposed observation scheme can effectively estimate motion states of one GMM moving in obstacle surroundings throughout.

Sub-structures such as micro-structured magnetic teeth fabricated with an artificial bacteria flagellum (ABF) are designed for achieving more motion modes, higher precision, and better controllability. To achieve these, a more precise model considering the non-circular cross-sectional features is setup without simplifying the structure as a helical filament with a circular cross-section as having been used in previous investigations, making it possible to include the effects of the substructures into the motion equation. Analyses and experiments verified the correctness. Besides of the geometric effects, our experimental observation also shows an anomalous step-out frequency appeared in an ABF. This asynchronous motion is attributed to the lag of magnetization with respect to the external rotating magnetic field due to the geometries and the soft-magnetic materials of the ribbons, which is different from the regular asynchronous motion solely caused by low Reynolds number of fluid to microscopic swimmers. While the lag of magnetization can be further attributed initiatively to the soft magnetic materials adopted, the feasibility to arrange the easy axis will enable many new possibilities, which is of particular interest in generating more modes for swarms such as cascade stepping out of ABFs with the same nominal overall sizes and for more precise positioning using stepping motion.

Stereo matching plays an important role in computer vision and robotics. Though substantial progress has been made on deep learning-based algorithms, the inherent semantic information within the ground truth of the training data for stereo matching has not been well explored. In this letter, we propose to use a pseudo segmentation sub-network to extract additional semantic information. More specifically, we divide the disparity label into groups and let each group correspond to a class for pseudo segmentation. To assist stereo matching with the semantic information obtained from pseudo segmentation, we inject the feature maps at the end of the pseudo segmentation sub-network into the cost volume that is used to infer the pixel-level disparity. To validate the effectiveness of the proposed approach, we select PSMNet (Chang and Chen, 2018)and GwcNet (Guoet al., 2019) as baselines and enhance them with the pseudo segmentation sub-network. Comprehensive experiments are carried out on the Scene Flow, KITTI 2015, and KITTI 2012 datasets, and the results show that our proposed method can improve the performance notably.

To date, untethered microrobots have been receiving tremendous attention for playing implacable roles of maneuverable tools in fields such as microfabrication and biomanipulation. Typical actuation of such untethered tiny robots is the magnetic field-based approaches, including gradient and rotational methods. Compared to the gradient type method, the rotational approach requires much less magnetic field strength to generate efficient actuation for magnetic microrobots. To actuate microrobots desirably, a precise periodic magnetic field should be provided. To generate precise periodic magnetic field with enhanced strength, this paper develops a prototype of 3D magnetic servoing system based on integrated solenoids, performance of which are enhanced by employing iron cores and extended number of coils. Each solenoid is equipped with a Hall sensor to provide real-time feedback signal for performing precise magnetic field control. To precisely regulate this setup, a decoupled asynchronous repetitive control (DARC) scheme is established to generate a desirable 3D periodic magnetic field with noise-level tracking error under the situation of missing execution opportunity randomly. Experimental results demonstrate the effectiveness of the proposed magnetic servoing system, which is promising for dynamic properties characterization of magnetic microrobots.

Scene generation is an important step of robotic drawing. Recent works have shown success in scene generation conditioned on text using a variety of approaches, with which the generated scenes cannot be revised after its generation. To allow modification on generated scenes, we model the scene generation process as a discrete event system. Instead of training text-to-pixel mappings using large datasets, the proposed approach uses object instances retrieved from the Internet to synthesize scenes. Evaluated on 128 experiments using MSCOCO evaluation dataset, the result shows the scene generation performance has been increased by 197%, 22.3%, and 55.7% compared with the state of the art approach on three standard metrics (CIDEr, ROUGH-L, METEOR), respectively. Human evaluation conducted on Amazon Mechanical Turk shows over 80% of generated scenes are considered to have higher recognizability and better alignment with natural language descriptions than baseline works.

Nanorobot has the potential to automate the manipulation and observation processes at molecular scale. Current drug discovery process is labor and cost intensive. Thus, a strong demand exists to automate this process. Here, we developed an Atomic Force Microscopy (AFM) based nanorobot for in situ sensing molecular interactions for drug discovery. The AFM tip and sample substrate are functionalized by the molecules of interest. Via measuring the interactive binding forces between these two molecules using the AFM based nanorobot, we are able to test the effectiveness of drug candidates on attenuating or enhancing the molecular interactions involved in cell signaling pathways. To measure the single molecular interactions precisely, we developed a new substrate coating method. The new method functionalizes the substrate much more evenly compared with the previous method. We further optimized settings during measuring the interactive binding force, such as coating with bovine serum albumin (BSA) and the level of indentation of AFM tip onto the substrate. With these progresses, this nanorobot measures single molecular interactions automated. We further used this nanorobot to test a small peptide, which was designed to attenuate the interactive binding force between focal adhesion kinase (FAK) and protein kinase B (AKT), two molecules involved in cell adhesion.

Atomic force microscopy (AFM) is a powerful measurement instrument, which has been widely implemented in various fields. To enhance the maneuverability, an AFM can be modified and its cantilever can be controlled as a robotic end-effector. To precisely control the nanorobots, their scanners inherent hysteresis, which is the main disadvantage, should be compensated sufficiently. Mostly, hysteresis compensators used in AFM control system only employ the symmetric models, which cannot well represent the asymmetric hysteresis phenomena of piezo-scanners. However, under many cases, the smart material based scanners possess asymmetric hysteresis slightly or seriously, which is far more complex than the regular symmetric case; in addition, for commercial or customized AFMs, the scanners are usually controlled without feedback. These drawbacks tackle further improvement of positioning accuracy of AFM systems. To effectively describe and further reduce the hysteresis of general cases, we propose a new type of generalized Prandtl-Ishlinskii (PI) operator based superposition model, named unparallel PI (UPI) model. The flexible edge of the UPI operator can be tilted freely within a defined range, which enables it to capture asymmetric hysteresis efficiently. To cancel the hysteresis effect in an open-loop system, three inverse compensation approaches are proposed, compared, and the corresponding stability is analyzed. Experiments with AFM based nanorobot verified the proposed approaches with convincing performance.