Noncontact trapping of micro objects has great application potential in fields like material science and biomedical engineering due to its label-freeness and biocompatibility. In this paper, an automated acoustic micro-particle trapping system implemented with phased transducer array (PTA) is prototyped. The system is incorporated with a stereo vision to provide visual feedback benefited from localization of the invisible acoustic field through hydrophone scanning. Binocular vision calibration and stereo matching are realized using image Jacobian matrix. An efficient phase modulation algorithm is proposed for the calculation of desired PTA phase profile in real-time and a spatiotemporal multiplexing control strategy is adopted to dynamically generate multiple trappings. Experimental results well demonstrated that the stable trapping of multiple particles can be robustly realized by the system, leading to the improvements of robotic noncontact manipulation with invisible acoustic end-effector. Note to Practitioners—This paper is motivated by the problem that previous classic acoustic trapping was achieved as a physical phenomenon that particles within the trapping zone would be automatically trapped and thus required people to place the particle into the invisible trapping zone, which is neither precision nor efficient. Such problem is a crucial factor that limits acoustic tweezer to be further readily usable in bioengineering, surface manufacturing, and quantitative micromechanical characterization. In this work, automated acoustic trapping is presented in the context of robotics, as grasping task in conventional industrial robots, that can generate the acoustic trap exactly in the location where particles are detected (by microscopic vision, or micro-CT or acoustic imaging, etc.). This paper proposes a full pipeline to automatically trap multiple particles using ultrasonic transducer array and binocular microscopic vision. The experiments verified the ability of proposed method in simultaneously trapping three micro particles with opposite acoustic properties. Such trapping method is the foundational technology for further acoustic manipulation such as arraying and sorting, which will be the tasks in our future work.

Knot tying is an essential task for robotic surgery, which is routinely realized by dual-arm robotic manipulation. Despite the well-established protocol and progress at macro scale so far, there remain challenges to further advance robotic knot tying technique, particularly in terms of decreasing space consumption with better dexterity, higher precision, and well biomechanical compatibility. In this paper, we propose a novel dual-arm nanorobotic system setup for automated knot tying performed on mini-incision under stereo microscope, featured by an additional rotation degree of freedom mounted on each arm. With this setup, an optimized motion trajectory planning under standard knot-tying protocol is also presented in order to support tying knots with shorter and thinner suture. Leveraging the natural advantage of nanorobotics and microscope, the proposed system is capable of tying consecutive throws with micro-suture on mini-incision, like in vascular anastomosis or microsurgery. We successfully evaluated the knot tying system on 2.0 mm wide bionic blood vessel with 30 mm long #8-0 micro-suture. We finally tested the mechanical strength of the knots for potential medical assessment.

Non-contact particle cluster manipulation holds significant promise in the realms of advanced manufacturing, chemistry, and pharmacy. However, achieving precise and dynamic control over the spatial kinematics of particle clusters remains a significant challenge, necessitating real-time and accurately programmable robotic end-effector. To this end, we develop an innovative non-contact, precise particle cluster manipulation system with ultrasonic phased array transducer (PAT) under microscope. This system combines a physics-based deep learning algorithm for real-time calculation of phase-only holograms (POHs), supporting PAT to dynamically form acoustic fields, namely holographic acoustic end-effector (HAE). Leveraging the dynamically and accurately generated HAEs by our system, kinematics control of particle clusters including aggregation, rotation, and translation is yielded. The extensive experiments well demonstrated the effectiveness of proposed system for particle cluster manipulation.

Acoustic holography (AH) is a promising technique for precise noncontact micro-nano robotic manipulation. It encodes a three-dimensional (3D) acoustic field acting as a virtual end-effector into a two-dimensional (2D) hologram, whereby the desired acoustic field reconstruction is made possible. Most traditional methods to implement AH, such as 3D printed holographic lens and phased array of transducers (PAT), have limitations of dynamic and dexterous manipulation. Furthermore, existing iterative optimization algorithms to calculate 2D holograms have inadequate accuracy and real-time performance. To address these issues, this paper proposes a physics-based deep learning method with a novel training framework for phase-only hologram (POH) calculation enabling further pushing forward the PAT-based AH for noncontact robotic manipulation. By implementing independent control of each channel on PAT referring real-time calculated POH by a well-trained network, the desired acoustic field can be reconstructed in real-time with high fidelity. The results both on a simulated dataset and a real dataset demonstrate that our method supports accurate and dynamic reconstruction of desired acoustic field with distinct morphologies, with an average reconstruction error of 0.085 and average POH computing time of 47 milliseconds on GPU. Indeed, this work shows the future potential of AH in the field of noninvasive medical therapy, exogenous material delivery, and miniaturized industrial assembly.Note to Practitioners—This paper addresses the challenge of noncontact micro-nano robotic manipulation by PAT-based AH, an intriguing technique in bioengineering, micro-assembly, and material characterization. However, existing approaches have limited precision and real-time performance. To overcome these limitations, this paper proposes a physics-based deep learning method with a novel training framework. Our method achieves excellent accuracy and real-time performance, enabling efficient reconstruction of various complicated acoustic field morphologies for precise and dynamic acoustic manipulation. Experimental results demonstrate its high manipulation flexibility due to the independent modulation of each channel of PAT and real-time precise control due to the ultrafast calculation of the proposed deep learning method, though the method has not yet been deployed into an acoustic manipulation system and tested in practice. Future research will focus on designing physical experiments for further evaluation. Overall, the proposed method provides a novel and promising basis for desired acoustic field generation.

Phase-only acoustic holography is a fundamental and promising technique for contactless robotic manipulation. Through independently controlling phase-only hologram (POH) of phase array of transducers (PAT) and simultaneously driving each channel by sophisticated circuits, a certain acoustic field is dynamically generated in working medium (e.g., air, water or biological tissues) at certain moment. The phase profile of PAT is required dynamically and precisely as per arbitrary expected acoustic field for the sake of versatile and stable robotic manipulation. However, the most conventional methods rely on iterative optimization algorithms which are inevitably time-consuming and probably non-convergent, moreover hindering versatility and fidelity of acoustic robotic manipulation. To address these issues, this paper reports a real-time phase-only acoustic holography algorithm by virtue of iterative unsupervised learning. Using a physics model to construct two queues, which we refer to as experience pools, data pairs consisting of a target acoustic amplitude hologram in expected acoustic field and corresponding POH of PAT are collected on-the-fly, circumventing costly preparation of annotated dataset in advance. With iterative learning between neural network training and experience pools update, both the solution of objective inverse mapping and the adaptation for arbitrary desired acoustic field are mutually enhanced. The experiments and results validated that the proposed approach surpasses previous algorithms in terms of real time and precision.

Ultrafast acoustic holography (AH) enabling dynamic contactless micro-nano robotic manipulation has recently attracted wide attention. As an advanced technique, AH encodes specific three-dimensional (3D) acoustic field on a two-dimensional (2D) hologram whereby realizing holographic reconstruction with high fidelity. However, current approaches face the limitation of encoding time, accuracy and flexibility, thus, leading to inapplicability for dynamic and precise robotic manipulation. Here, we develop an approach to overcome these issues. Its basic idea is to use a convolutional neural network trained in a self-supervised manner with iterative interaction with virtual physical environment. Energy conservation is incorporated to access the physical constrain during wave propagation. The experimental results demonstrate that the proposed method circumvents laborious annotated dataset preparation and boosts the reinforcement from physics model. By the validation and comparison on distinct acoustic fields with various patterns, the accuracy and real-time performance of the proposed method are confirmed supporting dynamic and precise robotic manipulation.

Acoustic holography is a newly emerging and promising technique to dynamically generate arbitrary desired holographic acoustic field in 3D space for contactless robotic manipulation. The latest technology supporting complex dynamic holographic acoustic field reconstruction is through phased transducer array (PTA), where the phase profile of emitted acoustic wave from discrete transducers is controlled independently by sophisticated circuits to modulate the acoustic interference field. While the forward kinematics of a phased array based robotic manipulation system is simple and straightforward, the inverse kinematics of the required holographic acoustic field is mathematically non-linear and unsolvable, which substantially limits the application of dynamic holographic acoustic field for robot manipulation. In this work, we propose a physics-based deep learning framework for this phase retrieval inverse kinematics problem so that the target complex hologram could be reconstructed precisely with average MAE of 0.022 and in real time with prediction time of 47 milliseconds on GPU. The accuracy and real time of the proposed method for dynamic holographic acoustic field reconstruction from PTA are demonstrated experimentally.