Magnetic microrobots (MMs) have emerged as promising tools for targeted therapies, including non-invasive in vivo treatments and precise drug delivery, owing to their untethered controllability and biocompatibility. Current actuation strategies for MMs primarily rely on two magnetic field (MF) generation approaches: gradient-based and rotational methods. Unlike the gradient method, rotational actuation enables efficient manipulation of MMs under significantly weaker magnetic fields. To fully leverage the potential of rotationally driven MMs, a comprehensive understanding of their fundamental spin motility is essential. Achieving accurate characterization of these MMs necessitates the development of an MF generation system equipped with rapid motion-tracking and broad-range measurement capabilities. This study proposes a high-speed rotating states observation scheme by developing a tracking-based optimal local imaging and estimation scheme, simultaneously meeting the broad-range observation capability and the high imaging speed requirement. Specifically, the CSR-DCF tracking method is adopted to detect the MM’s location, and based on this, the observation system adjusts the imaging region optimally. An estimation scheme based on the asynchronous rectification method is derived to measure the MM rotating states consistently using measured MF data and local optical images of the target. Experimental studies are carried out to validate the effectiveness of the proposed scheme.

To date, untethered micro/nanorobots have attracted considerable attention in various aspects due to their unique potential for in-vivo applications such as the targeted therapy. One of the most promising types of micro/nanorobots is the class of ferromagnetic microrobots which can be efficiently actuated via gradient/rotational magnetic field generated by less costly electromagnetic coil systems. For performing successful operations, locomotion control of the magnetic microrobots is non-trivial. Modern controllers commonly require motion status-based feedback. To fully utilize those advanced approaches, motion state of one microrobot should be supplied, however it is still challenging in cases. It is noted that, during locomotion, one ferromagnetic microrobot can combine with others to form an unstructured larger one, namely growing magnetic microrobot (GMM), whose dynamic behavior keeps changing, and thus the model-based observers are never applicable. Besides, tracking and estimating states of those unstructured time-varying GMMs in complex surroundings are always challenging, especially for an uneven sampling scenario. In order to accurately estimate the GMM motion status in a complex environment via micro-vision, this study develops an indeterministic observer leveraging on the approach of discriminative correlation filter with channel/spatial reliability (CSR-DCF) and the variable-step finite-time sliding mode (FSM-V) state estimation theory. Experimental study verifies that the proposed observation scheme can effectively estimate motion states of one GMM moving in obstacle surroundings throughout.

To date, untethered microrobots have been receiving tremendous attention for playing implacable roles of maneuverable tools in fields such as microfabrication and biomanipulation. Typical actuation of such untethered tiny robots is the magnetic field-based approaches, including gradient and rotational methods. Compared to the gradient type method, the rotational approach requires much less magnetic field strength to generate efficient actuation for magnetic microrobots. To actuate microrobots desirably, a precise periodic magnetic field should be provided. To generate precise periodic magnetic field with enhanced strength, this paper develops a prototype of 3D magnetic servoing system based on integrated solenoids, performance of which are enhanced by employing iron cores and extended number of coils. Each solenoid is equipped with a Hall sensor to provide real-time feedback signal for performing precise magnetic field control. To precisely regulate this setup, a decoupled asynchronous repetitive control (DARC) scheme is established to generate a desirable 3D periodic magnetic field with noise-level tracking error under the situation of missing execution opportunity randomly. Experimental results demonstrate the effectiveness of the proposed magnetic servoing system, which is promising for dynamic properties characterization of magnetic microrobots.

Scene generation is an important step of robotic drawing. Recent works have shown success in scene generation conditioned on text using a variety of approaches, with which the generated scenes cannot be revised after its generation. To allow modification on generated scenes, we model the scene generation process as a discrete event system. Instead of training text-to-pixel mappings using large datasets, the proposed approach uses object instances retrieved from the Internet to synthesize scenes. Evaluated on 128 experiments using MSCOCO evaluation dataset, the result shows the scene generation performance has been increased by 197%, 22.3%, and 55.7% compared with the state of the art approach on three standard metrics (CIDEr, ROUGH-L, METEOR), respectively. Human evaluation conducted on Amazon Mechanical Turk shows over 80% of generated scenes are considered to have higher recognizability and better alignment with natural language descriptions than baseline works.

In this paper, two techniques have been presented for slippage detection. They are independent of sensor signal type and are promising for general use on tactile array sensors. The first method is based on frequency analysis of the correlation coefficient sequence of sensor array data sampled as time evolves. The main idea is that a slippage causes heavier fluctuation in the sensor signal distribution and values than a static case. The second approach employs 2-D cross correlation to detect displacements of the sliding object from tactile images, which makes it possible to estimate the slippage velocity using commercially available sensors rather than custom hardware. Experiments have been implemented to evaluate the proposed approaches. It can be seen that the first method is capable of detecting both translational and rotational slippage. Also, it works well in dynamic environments. Additionally, the ability of the second method to detect slippage velocity has been confirmed in the experiment.

A new planning and control scheme for natural language control of robotic operations using the perceptive feedback is presented. Different from the traditional open-loop natural language control, the scheme incorporates the high-level planning and low-level control of the robotic systems and makes the high-level planning become a closed-loop process such that it is able to handle some unexpected events in the robotics system and the environment. The experimental results on a natural language controlled mobile manipulator clearly demonstrate the advantages of the proposed method.

Standard manipulators are restrained in many applications due to their limited working ranges. Adding mobile platforms, in particular nonholonomic mobile platforms, can expediently enlarge their working ranges but also introduces new challenges. The problem of the existing control methods for nonholonomic mobile manipulators is that they leave out the consideration of the differences between the mobile platform and the manipulator such as the dynamics differences and working condition differences. This may consequently result in some unnecessarily large errors for the motion tracking in the implementation. To address this problem, this paper proposes a new practical control method using the adaptive motion distribution and coordination between the mobile platform and manipulator to minimize the errors of the motion control and also automatically handle some unexpected events. The effectiveness and advantages of the proposed method were demonstrated through both simulation and experimental results.

Robots often have limited knowledge and need to continuously acquire new knowledge and skills in order to collaborate with its human partners. To address this issue, this paper describes an approach which allows human partners to teach a robot (i.e., a robotic arm) new high-level actions through natural language instructions. In particular, built upon the traditional planning framework, we propose a representation of high-level actions that only consists of the desired goal states rather than step-by-step operations (although these operations may be specified by the human in their instructions). Our empirical results have shown that, given this representation, the robot can reply on automated planning and immediately apply the newly learned action knowledge to perform actions under novel situations.

This paper describes an approach for a robotic arm to learn new actions through dialogue in a simplified blocks world. In particular, we have developed a three-tier action knowledge representation that on one hand, supports the connection be-tween symbolic representations of lan-guage and continuous sensorimotor repre-sentations of the robot; and on the other hand, supports the application of existing planning algorithms to address novel situ-ations. Our empirical studies have shown that, based on this representation the robot was able to learn and execute basic actions in the blocks world. When a human is engaged in a dialogue to teach the robot new actions, step-by-step instructions lead to better learning performance compared to one-shot instructions. 1