Cell rotation plays a crucial role in micromanipulation. Among manual cell rotation techniques, single-micropipette cell rotation is widely adopted due to its high efficiency and flexibility. However, there is currently no method capable of achieving automated single-micropipette cell rotation. In this study, we developed the first three-dimensional (3D) simulation system for single-micropipette cell rotation. Based on this simulation system, we successfully achieved single-micropipette cell rotation imitation learning (IL) for the first time. Specifically, we first analyze the forces acting on cells in the fluid, establishing a dynamic model that describes the cell's behavior in response to the flow velocity at the holding micropipette's orifice, the relative position of the micropipette, and time. We then developed the cell rotation simulation environment by discretizing the model and designing the simulation's cell and holding micropipette models based on real-world conditions. Finally, we designed a network architecture for IL using this model, achieving single-micropipette cell rotation in simulation. The results demonstrate that the simulation system exhibits a relative error range of 5.34% to 12.21% compared to real-world experiments, indicating a high degree of accuracy. Additionally, the single-micropipette cell rotation task achieved a success rate of 69% with an average completion time of 17.13 seconds, closely matching the expert data's average time of 17.69 seconds, confirming the feasibility of the simulation system.

Zebrafish are widely used in the biomedical field, as an ideal model for microinjection. In automated zebrafish microinjection, posture adjustment is the first and key step, which takes a lot of skill, and injection success assessment is a challenging task. Constrained by these two aspects, it is difficult to further enhance the efficiency and success rate of injection. In this study, we propose an automated dual-micropipette coordination microinjection system. Zebrafish are randomly arranged in our system, reducing the operational difficulty, and the yolk is positioned using a pose estimation algorithm, followed by injection accomplished with dual-micropipette. Due to the reduction of posture adjustment time by half, the proposed system achieves the shortest injection time of 15.2s. Moreover, the simplicity of the system and the ease of operation contribute to the clinical feasibility of our system.

Patch clamp technique is a ”gold standard” approach to the study of ion channel biophysics and pharmacology. Due to the complexity of the operation and the heavy reliance on the experience of the experimenter, more and more researchers are focusing on patch clamping automation. In manual patch clamping, the operators patch the neurons in the specific brain region by experience, which lacks quantitative indicators of brain region selection. At present, the existing automated patch clamp focuses more on searching and measuring single cells under a high magnification lens, but ignores the localization of different brain regions under a low magnification lens, affecting the efficiency and application of the automated system. Addressing this, a 2D-3D registration method is proposed to register the experimental microscopic image of brain slices onto the brain atlases. This method is then embedded into the patch clamp system for brain region selection and electrophysiological recordings of neurons in multiple brain regions. It is the first time to achieve automated selection of brain regions in patch clamp experiments. The experimental results indicate that brain slice registration and brain region selection completely meet the requirements of the patch clamp experiment. Compared with the original robotic patch clamp process, the total time for the whole-cell patch clamp initialization with the improved process is reduced by 280 seconds, and the efficiency is increased by 42%. Meanwhile, the time for brain region switching is reduced to$1/6$of the original. In the future, this method will be applied to automatic image-guided patch clamp systems.Note to Practitioners—Calibrating the depth of the experimental brain slice in the brain and dividing the distribution of each brain region in the brain slice are the most important preparations for electrophysiological experiments. In this paper, we design 2D-3D registration to establish a method for brain region selection and robotic patch clamping of different brain regions in one brain slice. We have successfully detected the electrophysiological signals of neurons in 3 major brain regions, including the visual region, auditory region, and hippocampus. The experimental results show that the method proposed in this paper is suitable for the existing robotic patch clamp system and greatly improves the degree of automation.

Zebrafish microinjection is widely used in vascular biology, neurology, and other research areas. The microinjection of zebrafish larvae is a complicated 3D task because zebrafish larvae are independent individuals with complex body structures compared with biological cells. In this study, we propose an automatic high-throughput injection system for zebrafish larvae under an optical inverted microscope. The proposed system combines the fixing device design, object detection, visual positioning, and 3D injection path planning to improve the survival rate of zebrafish after injection. Experimental results demonstrated the capability and efficiency of the proposed system, which achieved a success rate of 95.0% and a survival rate of 98.7% for zebrafish injection. The system can be applied to various biomedical and biochemical experiments.Note to Practitioners—Zebrafish larvae have long been an important model organism in neuroscience, biomedicine, and drug discovery. In various methods for external substance transportation into the zebrafish, microinjection is more efficient but also more difficult. At present, high-throughput injection of zebrafish larvae is mainly based on traditional manual operations. The injection speed and success rate of manual injection gradually decrease with the increase in zebrafish numbers. In this study, an automatic injection system is designed and implemented for high-throughput zebrafish larvae injection. We first use a simply designed mold to fix batch zebrafish larvae. Then we propose a coarse-fine two-step positioning method and perform a 3D injection path planning to achieve precise microinjection. The proposed injection system frees the operators from the repetitive and tedious injection work, providing technical support for the life science experiment.

Cell rotation is widely used to adjust cell posture in sub-cellular micromanipulations. The trajectory planning of the injection micropipette is needed, so that the cells can be rotated with the minimum deformation to reduce cell damage and keep cell viability. Due to the uncertainty of cell properties and manipulation environment, it is difficult to identify the parameters of the mechanical models in traditional robotic cell rotation methods. In this paper, deep reinforcement learning is introduced into cell manipulation for the first time to perform trajectory planning of the micropipette. We first abstract the cell rotation process by using the mechanical model and microscopic vision techniques and build a cell rotation simulation environment. Then we design a reward function by combining various factors of cell rotation and implement a reinforcement learning framework based on deep Q-learning (DQL). Finally, we train the cell rotation process based on the deep reinforcement learning algorithm. The simulation results indicate the proposed DQL agent achieved an average success rate of 97% without useless exploration. Moreover, the proposed method rotated the cells in a way that causes less mechanical damage than humans, demonstrating the DRL ability for cell rotation with high efficiency and low cell damage.

Cell aspiration is a common technique in cell manipulation for cell transfer or intracellular property measurement. In this paper, we present a robotic micromanipulation system for cell aspiration and positioning by a micropipette. Considering the relative motion of the object and the fluid, we first establish an overall dynamic model of microbead motion inside and outside the micropipette based on computational fluid dynamics (CFD). Then we design an adaptive sliding mode controller (ASMC) for microbead aspiration outside the micropipette and positioning inside the pipette based on the dynamic model. The controller is proven to achieve asymptotic stability by Lyapunov techniques. Simulation and experimental results demonstrate the effectiveness of the fluid model and the performance of the designed control system. Note to Practitioners—Cell aspiration with a micropipette is a key technology in cell manipulation. Generally, there is relative motion between the aspirated object and the fluid, resulting in large overshoot even aspiration failure. In this paper, we set up an overall dynamic model of microbead motion inside and outside the micropipette, combining microbead motion dynamics, fluid dynamics and pneumatic pump modeling. Based on this model, we design an ASMC for microbead aspiration outside the micropipette and positioning inside the pipette. In simulations and experiments, the positioning errors of the microbead of different sizes converge to zero without overshoot, revealing the strong robustness of the controller. Applications for this technology include cell or sperm aspiration and injection.

Visual localization, which is a key technology to realize the automation of cell manipulation, has been widely studied. Since the depth of field of the microscope is narrow, the planar localization and depth estimation are usually coupled together. At present, most methods adopt the serial working mode of focusing first and then planar localization, but they usually do not have good real-time performance and stability. In this paper, a simultaneous depth estimation and localization network was developed for cell manipulation. The network takes a focused image and a defocus-offset image as inputs, and outputs the defocus in the depth direction and the offset in the plane at the same time after going through defocus-offset information extraction, defocus classification mapping and offset regression mapping. To train and test our network, we also create two datasets: An Adherent Cell dataset and an Injection Micropipette dataset. The experimental results demonstrated that the proposed method achieves the detection of all test samples with a frame rate of more than 40Hz, and the maximum errors of depth estimation and localization are$\boldsymbol{2.44\mu m}$and$\boldsymbol{0.49\mu m}$, respectively. The proposed method has good stability, which is mainly reflected in its strong generalization ability and anti-noise ability.

Zebrafish (Danio Rerio) larvae have long been an important model organism for biomedicine and drug discovery. It is difficult to deliver the external materials into the circulatory system by conventional exposing administration, while vein microinjection is more efficient but more challenging. In this paper, a robotic cardinal vein microinjection system was presented for zebrafish larvae. The key points of injection pipette during penetration were analyzed, and their 3D positions were identified online by combining motion analysis and visual calibration. 3D path planning of the pipette tip was designed automatedly before injection. The injection system was evaluated by injecting green fluorescent microspheres into 20 zebrafish larvae. The experiment results show that the fluorescent microspheres spread rapidly in the blood circulation system after injection, achieving a success rate of 85.0% and a survival rate of 94.1%. The superior performance provided by the system will significantly facilitate tumor xenografts and canner studies on zebrafish.