Segmenting individual trophectoderm (TE) cells is essential for developing quantitative metrics to assess the developmental potential of human blastocysts. The elongated shape and circular arrangement of TE cells lead to continuously varying orientations across the image, posing challenges for existing cell instance segmentation methods that assume uniformly oriented cells. As a result, most methods segment the TE as a single region, and the development of quantitative, cell-level metrics predictive of live birth potential remains unexplored. In this work, we propose an instance segmentation model that represents elongate, circularly arranged TE cells using elliptical distance maps, with which superior performance in both segmentation accuracy and metric extraction was achieved, compared with state-of-the-art methods. The extracted metrics, including TE cell number, the mean and standard deviation of cell length, width, area, and mean inter-cell distance, serve as effective predictors of a blastocyst’s live birth potential. When used to predict live birth, these metrics achieved a significantly higher area under the receiver operating characteristic curve (AUC = 0.693) than traditional TE morphological grades (AUC = 0.585). The source code is publicly available at https://github.com/robotVisionHang/TESeg.

Current numerical methods for solving kinematic identification (KI) and inverse kinematics (IK) are limited in accuracy, convergence rates, and robustness, necessitating further enhancement. This paper presents a modified Halley method for solving the KI and IK problems of serial robots based on the product of exponentials formula, achieving quintic convergence. Specifically, a general error model is first established based on exponential coordinates, and KI and IK are reformulated as root-finding problems. Next, the modified Halley method, which we prove to be a fifth-order method and incorporates a damping strategy, is proposed to resolve the singularity issue and enhance robustness. Subsequently, the Jacobian and Hessian matrices required for the proposed method are analytically derived based on the time differential of exponentials. Furthermore, highly simplified explicit formulas for these matrices are presented for the IK problem. Simulations on serial robots with various configurations validate the proposed method's accuracy, convergence rates, and robustness in solving KI and IK problems, as well as its advantages over the state-of-the-art. Additionally, experimental validation of KI on two physical robots further demonstrates the effectiveness of the proposed method. Our custom-written MATLAB and C++ codebases are made publicly available for download.

Video object detection (VOD) of motile cells (e.g., bacteria and sperm) under microscopy is challenging due to motion blur, sporadic out-of-focus, and pose variations. Compared with VOD in generic scenes, the lower contrast and smaller color space of microscopy imaging further introduce feature overlap between the foreground objects and the background objects (e.g., impurity cells and contaminants). Transformer-based methods have achieved great success in the VOD of generic scenes by utilizing object queries to model the inner-frame objects and the inter-frame objects. However, the appearance overlap problem in microscopy video frames significantly compromises the inter-frame query aggregation by introducing background features into the object query. To tackle this challenge, this paper reports a static-dynamic query-based VOD network that treats object queries of the current video frame and reference video frames differently. Specifically, a two-stage framework is implemented that first generates high-quality object queries of reference frames with a static Transformer decoder pre-trained on a still image dataset. The network is then trained on a per-frame annotated dataset using a dynamic Transformer decoder to model the object queries of the current frame. A Reference Query Relation Module is further proposed to enhance the reference queries for more effective aggregation with the current query. Experiments on clinically collected biopsied sperm datasets validated the effectiveness of the proposed method.

Sperm morphology measurement is vital for diagnosing male infertility, which involves quantification of multiple subcellular parts for each sperm. Instance-aware part segmentation networks have been introduced to address this task by automatically identifying individual sperm and segmenting their subcellular parts. However, major limitations of state-of-the-art instance-aware part segmentation networks include: 1) they are time-consuming and computational expensive due to sequential processing and multi-stage frameworks; 2) they perform poorly for densely packed sperm that overlap or cross over one another. To overcome these challenges, this paper proposes 1) integrating instance identification and subcellular part segmentation within a single-stage framework to save inference time and memory usage; 2) dividing a sperm target into simpler components (head and tail) to improve prediction accuracy, followed by a contrastive learning-based matching method to pair the head and tail. Experimental results on our clinically collected human sperm dataset demonstrated that the proposed network not only outperformed state-of-the-art CP-Net (by 3.5% APp vol) but also achieved realtime inference (48.0 frames per second), effectively meeting the clinical requirements for automated parts segmentation of sperm. final part segmentation results. 2) Since the sperm head and tail have simpler shapes, they are detected separately to improve segmentation accuracy. A contrastive learning-based method is then designed to pair head and tail based on similarity of feature embeddings extracted from the proposed instance prediction branch. The proposed method significantly outperformed existing networks, particularly in handling densely packed sperm. The presented method has applicability to analyzing sperm and more broadly other cell types.

Automated morphological measurement of cellular and subcellular structures in live cells is important for evaluating cell functions. Due to their small size and transparent appearance, visualizing cellular and subcellular structures often requires high magnification microscopy and fluorescent staining. However, high magnification microscopy gives a limited field of view, and fluorescent staining alters cell viability and/or activity. Therefore, microscopy image enhancement methods have been developed to predict detailed intracellular structures in live cells. Existing image enhancement networks are mostly CNN-based models lacking global information or Transformer-based models lacking local information. For these purposes, a novel CNN-Transformer combined bilateral U-Net (CTBUnet) is proposed to effectively aggregate both local and global information. Experiments on the collected sperm cell enhancement dataset demonstrate the effectiveness of proposed network for both super-resolution and virtual staining prediction.

Automated positioning systems play a pivotal role in micro-scale cell manipulation. In clinical intracytoplasmic sperm injection (ICSI) for infertility treatment, a motile sperm needs to be immobilized by glass micropipette tapping for subsequent surgical steps. The process requires accurate tracking of the target sperm and precise alignment between the sperm tail and the micropipette. Manual sperm immobilization suffers from inconsistent success rates, and current robotic systems developed for the task fail to comply with the standard clinical setup. Instead of using a motorized micromanipulator as in existing robotic systems, this paper presents an automated and compact three-dimensional (3-D) positioning stage for sperm immobilization that can be seamlessly integrated into standard clinical platforms. To tackle the challenge of accurately tracking the target sperm with degraded detection quality due to the complex 3-D motion of the positioning stage, a multi-stage sperm tracking scheme is designed for detection-to-tracklet association. To prevent physical contact between the sperm head and the micropipette, an adaptive tail-tapping planning strategy based on the sperm head orientation analysis is established. A visual servo controller equipped with a dynamic sperm motion observer is further employed to achieve precise positioning of the target sperm during the immobilization process. Experimental results demonstrated that the proposed system achieved a sperm tracking accuracy of 88.12%, and a sperm positioning accuracy of$2.3~\pm ~1.2~\mu $m. Further experiments revealed the system achieved a success rate of 93.5% and a time cost of 5.5 s for automated sperm immobilization. Note to Practitioners—This work presents an automated positioning system for the robotic immobilization of live human sperm in the intracytoplasmic sperm injection (ICSI) process. Conventional robotic systems developed for sperm immobilization use motorized stages which only provide two degrees of freedom (DOF) and disturb standard clinical setups with bulky sizes and additional equipment. Leveraging the advantages of piezoelectric positioners, a compact 3-D positioning system is developed to perform robotic sperm immobilization in an all-in-one manner. A sperm immobilization tracker is developed to track the target sperm during the immobilization process. The proposed multi-stage data association metric can effectively resist the noisy detection results due to occlusion and out-of-focus blur introduced by the 3-D movement of the positioning stage. Based on the analysis of the sperm head orientation, an adaptive tail-tapping planning strategy is established to avoid the risk of contacting the sperm head where DNA is contained. The developed positioning system can be easily integrated into a standard microscopy operation platform for biological cell manipulation. The proposed tracking scheme is applicable in various microscopy cell analysis scenarios where the detection results are inevitably affected by the degraded image quality.

Achieving efficient and robust grasping manipulation in microscopic scenarios is crucial for robotic applications. As a typical end-effector, the microgripper that can perform gripping-holding-releasing operations, plays an essential role in the area of micromanipulation. To facilitate the development of industrial microassembly and cell micromanipulation technologies, we propose a robotic microgripper (RM) with a two-finger structure based on magnetic programmable soft materials. First, we synthesize and characterize the magnetic programmable soft material, and fabricate the microfinger based on the soft material. The analytical model of the microgripper was developed using beam deformation theory and magnetic field modeling. By combining the deformation mechanism of flexible microfingers with visual feedback, we modeled the gripping force of the microgripper and proposed a feedback adaptive grasping strategy (FAGS). The cooperative control of two flexible microfingers via remote magnetic field actuation achieves the grasping of microobjects with various complex shapes and in different media. A micromanipulation robotic system was constructed by combining RM with a micromanipulation arm, enabling targeted grasping, transportation, and posture control of microobjects in complex scenarios. Note to Practitioners—The microgripper plays a pivotal role in advancing automated robotic cell micromanipulation and microassembly. This article is motivated to develop a microgripper based on soft programmable magnetic materials. The proposed microgripper features a straightforward structural design, which simplifies the manufacturing process. The actuation is implemented by visual feedback and controlled variations in magnetic field intensity. Compared to existing rigid microgrippers, soft microgrippers provide higher stability of contact with the object being manipulated. For cell manipulation, a robotic dexterous micromanipulation system is devised by integrating the RM onto a multi-degree-of-freedom micromanipulator arm, enabling precise operations such as grasping, releasing, transporting, and orientation control of zebrafish embryonic cells. In microassembly tasks, the microgripper demonstrates the ability to handle micro-parts of diverse shapes. The proposed microgripper holds substantial promise in advancing the automation of robotic cell micromanipulation and microassembly processes.

Traditional sperm morphology analysis is based on tedious manual annotation. Automated morphology analysis of a high number of sperm requires accurate segmentation of each sperm part and quantitative morphology evaluation. State-of-the-art instance-aware part segmentation networks follow a "detect-then-segment" paradigm. However, due to sperm’s slim shape, their segmentation suffers from large context loss and feature distortion due to bounding box cropping and resizing during ROI Align. Moreover, morphology measurement of sperm tail is demanding because of the long and curved shape and its uneven width. This paper presents automated techniques to measure sperm morphology parameters automatically and quantitatively. A novel attention-based instance-aware part segmentation network is designed to reconstruct lost contexts outside bounding boxes and to fix distorted features, by refining preliminary segmented masks through merging features extracted by feature pyramid network. An automated centerline-based tail morphology measurement method is also proposed, in which an outlier filtering method and endpoint detection algorithm are designed to accurately reconstruct tail endpoints. Experimental results demonstrate that the proposed network outperformed the state-of-the-art top-down RP-R-CNN by 9.2% ${\mathbf{AP}}_{vol}^p$, and the proposed automated tail morphology measurement method achieved high measurement accuracies of 95.34%,96.39%,91.20% for length, width and curvature, respectively.

Automated positioning systems play a pivotal role in micro-scale cell manipulation. In clinical intracytoplasmic sperm injection (ICSI) of in vitro fertilization (IVF) treatment, a motile sperm needs to be immobilized by glass micropipette tapping for subsequent surgical steps. The process requires accurate tracking of the target sperm and precise alignment between the sperm tail and the micropipette. Manual sperm immobilization suffers from inconsistent success rates, and current robotic systems developed for the task fail to comply with the standard clinical setup. Instead of using a motorized micromanipulator as in existing robotic systems, this paper presents an automated, compact three-dimensional positioning stage for sperm immobilization that can be seamlessly integrated into standard clinical platforms. Based on the analysis of the sperm head orientation, an adaptive tail tapping planning strategy is established to avoid the risk of touching the sperm head where DNA is contained. A visual servo controller equipped with a dynamic sperm motion observer is employed to achieve precise tracking and positioning of the target sperm three-dimensionally. Experimental results revealed the system achieved a success rate of 93.5% and a time cost of 5.5 s for automated sperm immobilization.

Instance segmentation of biological cells is important in medical image analysis for identifying and segmenting individual cells, and quantitative measurement of subcellular structures requires further cell-level subcellular part segmentation. Subcellular structure measurements are critical for cell phenotyping and quality analysis. For these purposes, instance-aware part segmentation network is first introduced to distinguish individual cells and segment subcellular structures for each detected cell. This approach is demonstrated on human sperm cells since the World Health Organization has established quantitative standards for sperm quality assessment. Specifically, a novel Cell Parsing Net (CP-Net) is proposed for accurate instance-level cell parsing. An attention-based feature fusion module is designed to alleviate contour misalignments for cells with an irregular shape by using instance masks as spatial cues instead of as strict constraints to differentiate various instances. A coarse-to-fine segmentation module is developed to effectively segment tiny subcellular structures within a cell through hierarchical segmentation from whole to part instead of directly segmenting each cell part. Moreover, a sperm parsing dataset is built including 320 annotated sperm images with five semantic subcellular part labels. Extensive experiments on the collected dataset demonstrate that the proposed CP-Net outperforms state-of-the-art instance-aware part segmentation networks.

Sperm immobilization is a crucial procedure in clinical cell surgery for infertility treatment. Current immobilization is implemented by tapping the sperm tail with a glass micropipette, but its effectiveness is restricted by sperm orientation and ineffective membrane ablation. Ineffective ablation leads to limited release of oocyte activating factors and lowers fertilization rate; and sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. This paper reports automated piezo-assisted sperm immobilization with enhanced efficacy of cell membrane ablation and sperm orientation control. The designed piezo drill consists of two orthogonal vibration modules to generate controlled micropipette vibration along axial and lateral axes. Through stiffness modeling, the flexure joints guide the motion of the central beam of each vibration module. To achieve sperm orientation control, whirl flow is induced by both axial and lateral vibration of the micropipette tip. To immobilize sperm, only micropipette’s axial vibration is generated to prevent lateral vibration from damaging sperm head. A visual servoing scheme is developed by decoupling sperm wiggling from positioning error for immobilization. Experimental results showed that sperm orientation control by the piezo drill achieved an error of 1.4$^{\circ}$and a time cost of 2.5 s. Visual servoing with sperm wiggling decoupling achieved a positioning error of 1.7$\mu$m. Furthermore, the piezo-assisted sperm immobilization technique led to effective membrane ablation. With membrane-impermeable stains, it took 5.6 s for the immobilized sperm to be stained after piezo-assisted immobilization, significantly less than 49.2 s by conventional micropipette tapping.Note to Practitioners—This work tackled the challenge of ineffective membrane ablation and orientation limit in clinical cell surgeries. Conventional manual immobilization suffers from low membrane ablation efficacy, which leads to limited release of oocyte activating factors and lowers fertilization rate. Moreover, sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. In this paper, we propose automation techniques for effective membrane ablation and orientation control of sperm. A clinically compatible piezo drill is developed to generate controllable micropipette motion along both axial and lateral directions. The whirl flow generated by micropipette vibration is employed to rotate sperm, which greatly increased the number of available sperm for immobilization. A visual servoing controller is developed to keep the sperm at the center of field of view for immobilization by decoupling sperm wiggling from positioning error. The developed methods can be generalized to the manipulation of other types of cells. The piezo drill can be used for effective membrane ablation of oocyte, embryo, yeast cell and so on. The orientation control strategy leveraging piezo-induced whirl flow is applicable to non-contact rotation of a variety of microorganism.

This paper reports, for the first time, automated grading of human blastocysts (day-5 embryos) from multi-focus images. Based on a novel attention module, a convolutional neural network (CNN) was developed to predict the morphological grade of a blastocyst. The attention module integrates high-level features extracted from the blastocyst’s multi-focus images. Experimental results revealed that multi-focus blastocyst images help improve the grading accuracies than a single blastocyst image. Comparisons of the accuracy achieved by the model and the average accuracy of five embryologists demonstrated that the proposed model can outperform embryologists in the morphological grading of blastocysts (88% versus 86% for development stage prediction, 83% versus 79% for inner cell mass grade prediction, 89% versus 82% for trophectoderm grade prediction).Note to Practitioners—This work was motivated by the subjectivity and significant intra-and inter-evaluator variations in manual morphological grading of blastocysts. Existing approaches to automate the grading process mainly use a single blastocyst image although multi-focus images captured at different focal planes reveal more morphological features of a blastocyst than a single blastocyst image. This paper describes a new CNN-based method using multi-focus images to improve the grading accuracy. The accuracy of the proposed method was verified on multi-focus images of human blastocysts captured by a standard time-lapse incubator at fixed focal depths ($-$45$\mu$m,$-$30$\mu$m,$-$15$\mu$m, 0$\mu$m, 15$\mu$m, 30$\mu$m, 45$\mu$m).

Automated manipulation of deformable objects is challenging due to the object’s deformation behavior. Different from still deformable objects such as wires and cloth, biological organisms such as sperm and worms are both deformable and motile, requiring the control of both deformation and motion. This paper reports automated orientation control of live sperm, as an example of motile deformable cells. Robotic manipulation of human sperm was performed by using a glass micropipette, which is a standard clinical tool, to rotate individual motile sperm. Sperm rotation must be performed before immobilization, as required in clinical cell surgery for infertility treatment. To control tail deformation during sperm rotation, a path planner was designed based on kinematic analysis and manipulation point update. To deal with the intrinsic motion of a motile sperm, a motorized stage was controlled to compensate for sperm swimming motion, and an observer was designed to decouple sperm orientation from its wiggling motion. A sliding mode controller was designed to cope with stiffness variances along the sperm tail and among different sperm. Deep neural networks were developed for robust sperm tail detection, and Kalman filter was used to predict tail motion. Experimental results demonstrated that automated sperm manipulation achieved an orientation error of 0.8° and operation time of 6.8 s, both significantly less than those of manual operation. The designed observer was effective to reduce sperm orientation error by reducing the disturbance from sperm wiggling motion. The developed sliding mode controller outperformed the PID controller in operation time, reducing the time of oocyte exposure to the ambient environment.Note to Practitioners—This work tackled the challenge of rotating a fast-swimming and deformable sperm in clinical cell surgeries. Automated manipulation of deformable objects has wide applications in industrial and service settings such as manipulating wires and folding cloth. However, the intrinsic motion of a motile sperm and the lack of a rotational degree of freedom in standard micromanipulators pose difficulties to automated sperm manipulation. In this paper, we propose automation techniques for sperm orientation control. For sperm tail detection, deep learning was used to handle the variances of shape and length among different sperm. A path planning strategy and a controller were designed to achieve automated rotation of motile sperm, with its deformation and motion both controlled. The developed methods can be generalized to the manipulation of other deformable objects such as wires, cables and cloth. These objects exhibit significant variance of mechanical properties, and calibration is often time-consuming. The designed controller can be used to manipulate deformable objects with robustness to varied mechanical parameters. Path planning was designed by updating the manipulation point based on the object’s deformation behavior, and is suitable in manipulation where constraints are imposed such as the object’s strain.

Soft tissue cutting is used for incision, separation and removal of tissues or cells. Due to high deformation of soft tissues resulting from their viscosity and elasticity, it is challenging to accurately cut the tissue along a desired path and control the force applied to the tissue for reducing invasiveness, especially at the microscale. This paper presents a robotic biopsy system for cutting and collecting trophectoderm cells from a highly deformable blastocyst. The system, for the first time, enables TE cell junction detection for laser ablation throughout the blastocyst biopsy process by using a convolutional neural network. The overall detection error was 2.13% in every 1,000 cell junctions with position RMSE of$1.63\ \mu \mathrm{m}\pm 0.29\ \mu \mathrm{m}$. A dynamics model was developed to describe the motion of the trophectoderm cells inside a biopsy micropipette. Based on this model, an adaptive control method was developed for trophectoderm cell aspiration and positioning inside the biopsy micropipette. Experimental results revealed that the controller was capable of effectively compensating for the cell positioning error by updating the varying system parameters according to the adaptation law. The success rate was 100%, the cell aggregate positioning accuracy was$\pm 1\ \mu \mathrm{m}$, the average settling time was 2 s, and the largest overshoot was$4.3\ \mu \mathrm{m}$. Compared to manual blastocyst biopsy, the robotic biopsy system shortened the blastocyst's recovery time (35 min vs. 50 min) which indicates lower invasiveness.

The robotic manipulation of deformable linear objects is a classic and challenging topic. Apart from synthetic objects, such as wires and cables, linear objects are also commonly found in biological cells and organisms. Biomanipulation of such objects is hampered by difficulties, such as limited degrees of freedom of micromanipulators and varied mechanical properties of the biological entities to manipulate. This article presents a robotic manipulation of human sperm, which are deformable cells with a linear shape. The shape and movement of the cell are recapitulated by our developed geometric and kinematic models. Under unfixed constraints between the end-effector and the cell, path planning is designed to update the manipulation point to control cell deformation. A state transition function is formulated in path planning to handle the stiffness variations of sperm without force sensing. A model-predictive controller is designed to minimize the orientation error and manipulation path length. To detect sperm tail for visual feedback, an accuracy of 98% was achieved via deep neural networks. The robotic manipulation of human sperm was performed using a standard clinical setup of a glass micropipette to rotate a sperm to the target orientation. Experimental results showed that robotic sperm rotation achieved an orientation error of 0.8$^{\circ }$, a tail curvedness of 0.14$\mu$m$^{-1}$, and an operation time of 5.6 s, all significantly less than those of the manual approach. The less orientation error and tail curvedness after robotic rotation led to a significantly lower speed of sperm entering the micropipette during sperm aspiration, resulting in a higher success rate of 97% (versus 76% after manual rotation) for aspiration control.

Precise rotational positioning of end-effectors under microscopy is crucial for robotic micromanipulation. However, the end-effector is presently limited to a fixed orientation, which is manually set before a given micromanipulation task, lacking accuracy and versatility ofin situreorientation. In this article, we present a unified framework for rotationally positioning the end-effector in three dimensions by establishing a general rotational model, developing a detection method within the limited field of view under microscopy, and designing a three-loop control strategy that adapts to different experimental requirements and model parameters. In experiments, a standard angled micropipette was used as the end-effector to verify the validity of the proposed methods. The performance was evaluated experimentally where the micropipette was robotically rotated to an arbitrarily desired orientation with an average orientation error less than$2^\circ$. In the experiments of sperm manipulation, thein situmicropipette orientation control capability improved the success rate of sperm immobilization and achieved dexterous robotic sperm orientation for facile aspiration.

Cell manipulation is a key technology in many biomedical and clinical applications, in which end-effector alignment is a critical procedure. Presently, end-effector alignment is performed manually and suffers from large misalignment error and inconsistency. Manual alignment often undesirably moves the end-effector (e.g., a glass micropipette) out of the limited field of view under microscopy and risks breaking the fragile end-effector. This paper presents automated end-effector alignment for robotic cell manipulation. A rotational degree of freedom was added to a micromanipulator with translational degrees of freedom. The kinematic model of end-effector’s rotation was established, and the unknown model parameters were calibrated and updated via quadratic optimization. A controller was designed based on the kinematics modeling and parameter optimization to compensate for rotation-induced translation and achieve end-effector alignment. Experimental results demonstrate that the robotic alignment technique achieved an accuracy of 0.6±0.3° and a time cost of 18.5 ± 10.2 s, both significantly less than manual alignment. The developed controller cost significantly less time for micropipette alignment than the PID controller. A glass micropipette was used as the end-effector for human sperm immobilization, a critical procedure in clinical cell surgery. The success rate of sperm immobilization was 97% by robotic micropipette alignment, higher than the success rate of 90% by manual alignment due to the higher accuracy of robotic alignment.

In biology, fluorescent dyes are routinely used for biochemical measurements such as pH and ion concentrations. They, especially when used for detecting a low concentration of ions, suffer from low signal-to-noise ratios (SNR); and increasing the concentration of fluorescent dyes causes more sever cytotoxicity. We invented a new approach that uses a low amount of fluorescent dye-coated magnetic nanoparticles for on-demand, accurately aggregating the nanoparticles and thus fluorescent dyes in a local region inside a cell for intracellular measurement. Experiments proved this approach is capable of achieving a significantly higher SNR and lower cytotoxicity. Different from existing magnetic micromanipulation systems that generate large swarms (several microns and above) or cannot move the generated swarm to an arbitrary position, we developed a five-pole magnetic micromanipulation system and technique for generating a small swarm (e.g., 1 μm; capable of generating a magnetic swarm from 0.52 μm to 52.7 μm with an error <; 7.5 %) and accurately positioning the small swarm (position control accuracy: 0.76 μm). As an example, the system performed intracellular pH mapping using a 1 μm swarm of pH sensitive fluorescent dye-coated magnetic nanoparticles. The swarm had an SNR inside a cell 10 times that by the traditional method, i.e., global dye treatment, with both cases using the same fluorescent dye concentration. Our intracellular measurement results, for the first time, quantitatively revealed the existence of pH gradient and polarized pH distribution in live migrating cells.

Robotic manipulation of deformable objects has been a classic topic in robotics. Compared to synthetic deformable objects such as rubber balls and clothes, biological cells are highly deformable and more prone to damage. This article presents robotic manipulation of deformable cells for orientation control (both out-of-plane and in-plane), which is required in both clinical (e.g., in vitro fertilization) and biomedical (e.g., clone) applications. Compared to manual cell orientation control based on empirical experience, the robotic approach, based on modeling and path planning, effectively rotates a cell, while consistently maintaining minimal cell deformation to avoid cell damage. A force model is established to determine the minimal force applied by the micropipette to rotate a spherical or, more generally, ellipsoidal oocyte. The force information is translated into indentation through a contact mechanics model, and the manipulation path of the micropipette is formed by connecting the indentation positions on the oocyte. An optimal controller is designed to compensate for the variations of mechanical properties across oocytes. The polar body of an oocyte is detected by deep neural networks with robustness to shape and size differences. In experiments, the system achieved an accuracy of 97.6% in polar body detection and an accuracy of 0.7° in oocyte orientation control with maximum oocyte deformation of 2.70 μm throughout the orientation control process.

Robotic manipulation of deformable objects (vs. rigid objects) has been a classic topic in robotics. Compared to deformable synthetic objects such as rubber balls and clothes, biological cells are highly deformable and more prone to damage. This paper presents robotic manipulation of deformable cells for orientation control (both out-of-plane and in-plane), which is required in both clinical (e.g., in vitro fertilization) and biomedical (e.g., clone) applications. Compared to manual cell rotation control based on empirical experience, the robotic approach, based on mathematical modeling and path planning, effectively rotates a cell while consistently maintaining minimal cell deformation to avoid cell damage. A force model is established to determine the minimal force applied by the micropipette to rotate a spherical or more generally, an ellipsoidal mouse oocyte. The force information is translated into indentation through a contact mechanics model, and the manipulation path of the micropipette is formed by connecting the indentation positions on the oocyte. A compensation controller is designed to compensate for the variations of mechanical properties across cells. The polar body of an oocyte is detected by deep neural networks with robustness to shape and size differences. Experimental results demonstrate that the system achieved an accuracy of 97.6% in polar body detection and an accuracy of 0.7° in oocyte orientation control with maximum oocyte deformation of 2.69 μm.

Measuring the motility and morphology parameters of motile cells is important for revealing their functional characteristics. This paper presents automation techniques that, for the first time, enable automated, non-invasive measurement of motility and morphology parameters of individual sperms. Compared to the status quo of qualitative estimation of single sperm's motility and morphology based on embryologists' empirical experience, the automation techniques provide quantitative data in nearly real time. An adapted joint probabilistic data association filter (JPDAF) was used for multi-sperm tracking and tackled challenges of identifying sperms that intersect or have small spatial distances. Since the standard differential interference contrast (DIC) imaging method has side illumination effect which causes inherent inhomogeneous image intensity and poses difficulties for accurate sperm morphology measurement, we integrated total variation norm into the quadratic cost function method, which together effectively removed inhomogeneous image intensity and retained sperm's subcellular structures after DIC image reconstruction. In order to relocate the same sperm of interest identified under low magnification after switching to high magnification, coordinate transformation was conducted to handle the changes in the field of view caused by magnification switch. Experimental results demonstrated an accuracy of 95.6% in sperm motility measurement and errors <;10% in morphology measurement.

Magnetic micromanipulation is an untethered technique and has enabled numerous applications in the scale of millimeters to micrometers from the tissue level to cell level. However, existing systems are not capable of maneuvering a sub-micrometer object for precise force control, preventing the realization of intracellular manipulation or `fantastic voyage' inside a single cell. The magnetic micromanipulation task achieved in this work is sub-micrometer position control and piconewton force control of a sub-micron (0.7 μm) magnetic bead inside a single human bladder cancer cell (RT4). The magnetic bead was 3D positioned in the cell using a generalized predictive controller that effectively tackled the control challenge caused by the slow visual feedback (1 Hz) from high-resolution confocal microscopy. The average positioning error was quantified to be 0.43 μm, which is slightly larger than Brownian motion-imposed constraint (0.31 μm). The system is capable of three-dimensionally applying a maximum force of 60 pN with a resolution of 4 pN. In experiments, a 0.7 μm magnetic bead was controlled to move from an initial position in a cell to target positions on the cell nucleus. Force-displacement data were obtained from multiple locations along the cell nucleus' major and minor axes. The results revealed, for the first time, significantly higher stiffness exists in the cell nucleus' major axis than the minor axis. This stiffness polarity was likely attributed to the aligned stress fibers of actin filament inside the cells.

Manipulation of motile cells such as bacteria and sperm is required in both cell biology and clinical applications. For immobilizing a motile sperm, the sperm head and tail positions must be accurately tracked, interference of proximal sperms on the target sperm must be tackled, and the orientation of the sperm must be properly aligned with the manipulation tool in order not to damage the sperm head where DNA is contained. Manual operation of sperm immobilization has stringent skill requirements, and both manual operation and existing robotic sperm immobilization suffer from inconsistent success rates and incapability of manipulating sperms swimming in all directions. This paper presents a robotic system for fully automated tracking, orientation control, and immobilization of motile sperms. Algorithms were developed for robustly tracking the sperm head and estimating the sperm tail positions under interfering conditions. A new visual servo control strategy was developed to enable the robotic system to actively adjust sperm orientation for immobilizing a sperm swimming in any direction. Experimental results from robotic immobilization of 400 sperms confirmed that the robotic system achieved a consistent success rate of 94.5 %, independent of sperm velocity or swimming direction.

Measuring cell motility and morphology is important for revealing their functional characteristics. This paper presents automation techniques that enable automated, non-invasive measurement of motility and morphology parameters of single sperm. Compared to the status quo of qualitative estimation of single sperm's motility and morphology manually, the automation techniques provide quantitative data for embryologists to select a single sperm for intracytoplasmic sperm injection. An adapted joint probabilistic data association filter was used for multi-sperm tracking and tackled challenges of identifying sperms that intersect or have small spatial distances. Since the standard differential interference contrast (DIC) imaging method has side illumination effect which causes inherent inhomogeneous image intensity and poses difficulties for accurate sperm morphology measurement, we integrated total variation norm into the quadratic cost function method, which together effectively removed inhomogeneous image intensity and retained sperm's subcellular structures after DIC image reconstruction. In order to relocate the same sperm of interest identified under low magnification after switching to high magnification, coordinate transformation was conducted to handle the changes in the field of view caused by magnification switch. The sperm's position after magnification switch was accurately predicted by accounting for the sperm's swimming motion during magnification switch. Experimental results demonstrated an accuracy of 95.6% in sperm motility measurement and an error <10% in morphology measurement.