Grating displacement sensing is regarded as one of the key technologies for achieving cross-scale nanopositioning. This paper proposes a real-time grating sensing correction and compensation technology to enhance the performance of the embedded grating displacement sensor in Cross-scale piezoelectric actuators (CSPAs), thereby enabling nano-scale motion positioning of CSPAs. Firstly, based on the principle of diffracted image reflection, a miniaturized grating sensing unit that can be monolithically integrated with the CSPA structure is designed. Secondly, an online self-correction algorithm based on amplitude iteration is proposed to dynamically eliminate DC offset and amplitude imbalance errors in the signals. Furthermore, a real-time error compensation strategy is constructed to compensate for inherent periodic errors and measurement lag errors of the system induced by stick-slip effects. Experimental results demonstrate that, with the proposed technology, the embedded grating displacement sensor can achieve a detection resolution of 0.9 nm within the full stroke. The CSPA integrated with this sensor achieved a positioning accuracy within ±1.3 nm over its scanning range, and a full-stroke bidirectional positioning consistency of 3.093 ± 1.358 nm.

Sperm morphology measurement is vital for diagnosing male infertility, which involves quantification of multiple subcellular parts for each sperm. Instance-aware part segmentation networks have been introduced to address this task by automatically identifying individual sperm and segmenting their subcellular parts. However, major limitations of state-of-the-art instance-aware part segmentation networks include: 1) they are time-consuming and computational expensive due to sequential processing and multi-stage frameworks; 2) they perform poorly for densely packed sperm that overlap or cross over one another. To overcome these challenges, this paper proposes 1) integrating instance identification and subcellular part segmentation within a single-stage framework to save inference time and memory usage; 2) dividing a sperm target into simpler components (head and tail) to improve prediction accuracy, followed by a contrastive learning-based matching method to pair the head and tail. Experimental results on our clinically collected human sperm dataset demonstrated that the proposed network not only outperformed state-of-the-art CP-Net (by 3.5% APp vol) but also achieved realtime inference (48.0 frames per second), effectively meeting the clinical requirements for automated parts segmentation of sperm. final part segmentation results. 2) Since the sperm head and tail have simpler shapes, they are detected separately to improve segmentation accuracy. A contrastive learning-based method is then designed to pair head and tail based on similarity of feature embeddings extracted from the proposed instance prediction branch. The proposed method significantly outperformed existing networks, particularly in handling densely packed sperm. The presented method has applicability to analyzing sperm and more broadly other cell types.

Automated morphological measurement of cellular and subcellular structures in live cells is important for evaluating cell functions. Due to their small size and transparent appearance, visualizing cellular and subcellular structures often requires high magnification microscopy and fluorescent staining. However, high magnification microscopy gives a limited field of view, and fluorescent staining alters cell viability and/or activity. Therefore, microscopy image enhancement methods have been developed to predict detailed intracellular structures in live cells. Existing image enhancement networks are mostly CNN-based models lacking global information or Transformer-based models lacking local information. For these purposes, a novel CNN-Transformer combined bilateral U-Net (CTBUnet) is proposed to effectively aggregate both local and global information. Experiments on the collected sperm cell enhancement dataset demonstrate the effectiveness of proposed network for both super-resolution and virtual staining prediction.

This paper presents an automated point-of-care semen analysis method that uses smartphone imaging to visually measure sperm concentration and motility of semen samples. The proposed method follows the same visual tracking scheme as laboratory semen analysis systems, aiming to match clinical standards while being suitable for point-of-care use. A boundary-sensitive segmentation network is developed to identify and distinguish sperm from impurities in raw semen. A novel occlusion-aware multi-sperm tracking algorithm is proposed to tackle challenges posed by smartphone imaging and undiluted raw semen samples. For automated motility measurement, an occlusion-awareness module is proposed to robustly track multiple sperm during frequent sperm crossover/occlusion. The module combines the segmented contour and kinematic-based probabilistic modeling to determine the occlusion status of both targets and measurements, facilitating fundamental improvement to feasible joint event enumeration to enable robust data association. The proposed method achieved a high success rate of 95.14% for tracking occluded sperm, with low mean errors for sperm concentration (2.03 million/ml) and motility (1.58%), outperforming existing multi-sperm tracking methods. In clinical tests involving 50 participants, our method exhibited good agreement with clinical standards (Spearman rank correlation coefficients of 0.94 for concentration and 0.89 for motility) even when used by inexperienced users.Note to Practitioners—Semen analysis is the gold standard method for assessing male reproductive capacity. Clinical semen analysis routinely uses professional computer-assisted semen analysis (CASA) systems to examine sperm concentration and motility; however, clinical visits for semen analysis are not always feasible due to the unavailability of such professional systems and the mental stress brought by clinical visits. This work provides a point-of-care semen analysis method. For hardware, a smartphone microscopic imaging modality was developed to enable clear visualization of sperm with the built-in smartphone camera. For software, an occlusion-aware multi-sperm tracking algorithm was proposed to automatically measure sperm concentration and motility. In addition to intensively validating the proposed point-of-care method against clinical CASA systems, this work also analyzed different types of tracking failure and quantified their effects on the automated evaluation of sperm concentration and motility. The techniques pave the way for further improvement in both point-of-care and clinical semen analysis. Overall, this work offers an accessible and reliable tool for automated male fertility evaluation.

This paper reports an asymmetric Bouc-Wen (ABW) hysteresis model and a hybrid control algorithm based on multi-modal Bayesian gradient optimization (MBGO) for trajectory tracking in the micro-positioning phase of a piezoelectric positioning platform. First, a system-level dynamic model capable of expressing hysteresis nonlinearity is established based on the asymmetric Bouc-Wen model. Second, an MBGO parameter identification algorithm based on Particle Swarm Optimization (PSO) is proposed to improve the characterization capability of the hysteresis model. Subsequently, a feedforward adaptive fuzzy PID (FF-AFPID) composite controller is designed by compensating the hysteresis nonlinearity through the ABW inverse model while dynamically adjusting PID parameters with adaptive fuzzy rules. Through triangular and sinusoidal trajectory tracking experiments, the root mean square errors were reduced to 0.112 nm and 0.103 nm by the FF-AFPID, with an improvement of 74.944%, 57.088% (triangular) and 77.511%, 57.083% (sinusoidal) over the FF-PID and FF-FPID algorithms, respectively. The results demonstrate that the trajectory tracking of performance the positioning platform in micro-positioning phase was significantly enhanced by FF-AFPID, with the maximum error being suppressed to sub-nanometer levels.

The fabrication of Nanoimprint Lithography (NIL) masters serves as the initial process in the manufacturing of devices such as silicon photonic chips using NIL. Repairing defects and modifying structures on an NIL master accurately is critical for NIL manufacturing. This study proposes an innovative scanning electron microscope (SEM)-based in-situ nanomanipulation technique to address the macro-micro-nano cross-scale nanopositioning issues necessary for repair functions. A macro/micro closed-loop control system was designed, which includes a frequency/voltage (f/u) proportional controller, a real-time direct inverse hysteresis compensation feedforward controller, a grating displacement sensor, and a cross-scale nanopositioning platform. The performance testing of the nano-manipulation approach resulted in a repair range of 22.18×20.92×10.06 mm3, meeting the repair requirements for most silicon photo chip NIL masters in terms of size. The system achieved a repair accuracy at 4.946 nm (X-axis), 4.663 nm (Y-axis), and 4.679 nm (Z-axis). As a demonstrate, the repair functionality testing confirmed the ability of the system to remove target structures such as cantilever beams and detach adhered particles from a master surface.

Point cloud clustering is a promising method for 3D surface defect diagnosis in manufacturing but requires manual clustering parameter selection, reducing usability. This paper proposes an automatic point cloud clustering method to address this issue. It employs a strategy that progresses from coarse to fine. In the coarse searching stage, a K-Nearest Neighbor (KNN) graph analysis technique is developed to recognize potential defective regions in parallel. Moving on to the fine stage of extracting detailed defects, a modified DBSCAN algorithm is proposed, in which the clustering parameters are calculated automatically from the KNN graph analysis results. Experimental results showed that the proposed method achieved cloud clustering with automatically calculated clustering parameters for surface defect diagnosis. The proposed method outperformed the traditional region growing algorithm in accuracy (0.942 vs. 0.680) and processing speed (21500 points/sec vs. 8740 points/sec) without requiring manual intervention.Note to Practitioners—This paper presents a method for diagnosing defects on automobile and flat steel surfaces. Current 3D point cloud techniques for surface defect diagnosis require manual parameter adjustments, reducing usability. This paper proposes an automatic method without manual intervention. The proposed method uses a coarse-to-fine strategy. The 3D point cloud is divided into sub-blocks to locate potential defects, and a clustering algorithm then extracts detailed defects with automatically determined parameters. We mathematically characterize changes in point density caused by surface defects and show how these features can be used for clustering parameter calculation. Experimental results demonstrate the method’s efficiency on flat as well as some curved surfaces, but it has yet to be evaluated on complex structures. Future work will aim to broaden its application to include a more extensive variety of surfaces and integrate it with robotic vision systems.

Contouring control of 3D trajectory is critical in multi-axial machine tool, scanning stage and other precision automation systems. Currently, most contouring controllers are based on Frenet frames, thus their limited applicability to nonzero-curvature 3D trajectories rather than zero-curvature ones which exist ubiquitously in multi-axial motion systems. This paper proposes an optimal vector-referenced moving frame based contouring controller (OVRMFCC) suitable for contouring control of 3D zero-curvature trajectories. Firstly, the optimal vector-referenced moving frame (OVRMF) capable of framing arbitrary finite-length smooth 3D trajectory regardless of its curvature was proposed. Then, a contouring controller (OVRMFCC) based on OVRMF was designed, followed by derivation of its analytical form and proof of its convergence. Finally, this controller was deployed to an FPGA-based controller target with comprehensive comparison experiments on a triaxial system. Experimental results indicates that OVRMFCC reduces at least 46.6% maximum contour error, and 25.0% root-mean-square contour error compared to cross-coupled controller. Besides, OVRMFCC achieves almost the same precision on trajectories with nonzero curvature or curvature singularities compared to TCF. It still maintains high-precision contour tracking on trajectories with continuous zero-curvature segments or planned discrete trajectories with sharp curvature changes, while TCF crashes or leads to several times larger contour error. Note to Practitioners—This work is motivated by the increasing need of contouring control of 3D trajectory in precision automation systems. The mainstream 3D contouring controllers, like task coordinated frame method and model predicted contouring controller, are invalid for zero-curvature 3D trajectory which ubiquitously exists in motion system since they are based on Frenet frame which fails to be defined where curvature is zero. Although cross-coupled controller can tackle zero-curvature 3D trajectory, it proves inefficient in reducing contour error as it is commonly model-free and not specifically designed. To solve this problem, we proposed an optimal vector referenced moving frame (OVRMF) for arbitrary infinite-length smooth 3D trajectory framing and proves its existence strictly. The main advantage of OVRMF lies in its independence of curvature, thus its existence everywhere as long as the curve is$C^{1}$continuous. And based on OVRMF a contouring controller (OVRMFCC) is designed which combines the benefit of both TCF and OVRMF. With this, OVRMFCC can track any infinite-length$C^{3}$-continuous 3D trajectory, which fill the gap of traditional 3D contouring controller. Experimental results reveal that OVRMFCC maintains high precision regardless of trajectory curvature, which outperforms both cross-coupled controller and TCF.

Instance segmentation of biological cells is important in medical image analysis for identifying and segmenting individual cells, and quantitative measurement of subcellular structures requires further cell-level subcellular part segmentation. Subcellular structure measurements are critical for cell phenotyping and quality analysis. For these purposes, instance-aware part segmentation network is first introduced to distinguish individual cells and segment subcellular structures for each detected cell. This approach is demonstrated on human sperm cells since the World Health Organization has established quantitative standards for sperm quality assessment. Specifically, a novel Cell Parsing Net (CP-Net) is proposed for accurate instance-level cell parsing. An attention-based feature fusion module is designed to alleviate contour misalignments for cells with an irregular shape by using instance masks as spatial cues instead of as strict constraints to differentiate various instances. A coarse-to-fine segmentation module is developed to effectively segment tiny subcellular structures within a cell through hierarchical segmentation from whole to part instead of directly segmenting each cell part. Moreover, a sperm parsing dataset is built including 320 annotated sperm images with five semantic subcellular part labels. Extensive experiments on the collected dataset demonstrate that the proposed CP-Net outperforms state-of-the-art instance-aware part segmentation networks.

Sperm immobilization is a crucial procedure in clinical cell surgery for infertility treatment. Current immobilization is implemented by tapping the sperm tail with a glass micropipette, but its effectiveness is restricted by sperm orientation and ineffective membrane ablation. Ineffective ablation leads to limited release of oocyte activating factors and lowers fertilization rate; and sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. This paper reports automated piezo-assisted sperm immobilization with enhanced efficacy of cell membrane ablation and sperm orientation control. The designed piezo drill consists of two orthogonal vibration modules to generate controlled micropipette vibration along axial and lateral axes. Through stiffness modeling, the flexure joints guide the motion of the central beam of each vibration module. To achieve sperm orientation control, whirl flow is induced by both axial and lateral vibration of the micropipette tip. To immobilize sperm, only micropipette’s axial vibration is generated to prevent lateral vibration from damaging sperm head. A visual servoing scheme is developed by decoupling sperm wiggling from positioning error for immobilization. Experimental results showed that sperm orientation control by the piezo drill achieved an error of 1.4$^{\circ}$and a time cost of 2.5 s. Visual servoing with sperm wiggling decoupling achieved a positioning error of 1.7$\mu$m. Furthermore, the piezo-assisted sperm immobilization technique led to effective membrane ablation. With membrane-impermeable stains, it took 5.6 s for the immobilized sperm to be stained after piezo-assisted immobilization, significantly less than 49.2 s by conventional micropipette tapping.Note to Practitioners—This work tackled the challenge of ineffective membrane ablation and orientation limit in clinical cell surgeries. Conventional manual immobilization suffers from low membrane ablation efficacy, which leads to limited release of oocyte activating factors and lowers fertilization rate. Moreover, sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. In this paper, we propose automation techniques for effective membrane ablation and orientation control of sperm. A clinically compatible piezo drill is developed to generate controllable micropipette motion along both axial and lateral directions. The whirl flow generated by micropipette vibration is employed to rotate sperm, which greatly increased the number of available sperm for immobilization. A visual servoing controller is developed to keep the sperm at the center of field of view for immobilization by decoupling sperm wiggling from positioning error. The developed methods can be generalized to the manipulation of other types of cells. The piezo drill can be used for effective membrane ablation of oocyte, embryo, yeast cell and so on. The orientation control strategy leveraging piezo-induced whirl flow is applicable to non-contact rotation of a variety of microorganism.

Automated manipulation of deformable objects is challenging due to the object’s deformation behavior. Different from still deformable objects such as wires and cloth, biological organisms such as sperm and worms are both deformable and motile, requiring the control of both deformation and motion. This paper reports automated orientation control of live sperm, as an example of motile deformable cells. Robotic manipulation of human sperm was performed by using a glass micropipette, which is a standard clinical tool, to rotate individual motile sperm. Sperm rotation must be performed before immobilization, as required in clinical cell surgery for infertility treatment. To control tail deformation during sperm rotation, a path planner was designed based on kinematic analysis and manipulation point update. To deal with the intrinsic motion of a motile sperm, a motorized stage was controlled to compensate for sperm swimming motion, and an observer was designed to decouple sperm orientation from its wiggling motion. A sliding mode controller was designed to cope with stiffness variances along the sperm tail and among different sperm. Deep neural networks were developed for robust sperm tail detection, and Kalman filter was used to predict tail motion. Experimental results demonstrated that automated sperm manipulation achieved an orientation error of 0.8° and operation time of 6.8 s, both significantly less than those of manual operation. The designed observer was effective to reduce sperm orientation error by reducing the disturbance from sperm wiggling motion. The developed sliding mode controller outperformed the PID controller in operation time, reducing the time of oocyte exposure to the ambient environment.Note to Practitioners—This work tackled the challenge of rotating a fast-swimming and deformable sperm in clinical cell surgeries. Automated manipulation of deformable objects has wide applications in industrial and service settings such as manipulating wires and folding cloth. However, the intrinsic motion of a motile sperm and the lack of a rotational degree of freedom in standard micromanipulators pose difficulties to automated sperm manipulation. In this paper, we propose automation techniques for sperm orientation control. For sperm tail detection, deep learning was used to handle the variances of shape and length among different sperm. A path planning strategy and a controller were designed to achieve automated rotation of motile sperm, with its deformation and motion both controlled. The developed methods can be generalized to the manipulation of other deformable objects such as wires, cables and cloth. These objects exhibit significant variance of mechanical properties, and calibration is often time-consuming. The designed controller can be used to manipulate deformable objects with robustness to varied mechanical parameters. Path planning was designed by updating the manipulation point based on the object’s deformation behavior, and is suitable in manipulation where constraints are imposed such as the object’s strain.

The robotic manipulation of deformable linear objects is a classic and challenging topic. Apart from synthetic objects, such as wires and cables, linear objects are also commonly found in biological cells and organisms. Biomanipulation of such objects is hampered by difficulties, such as limited degrees of freedom of micromanipulators and varied mechanical properties of the biological entities to manipulate. This article presents a robotic manipulation of human sperm, which are deformable cells with a linear shape. The shape and movement of the cell are recapitulated by our developed geometric and kinematic models. Under unfixed constraints between the end-effector and the cell, path planning is designed to update the manipulation point to control cell deformation. A state transition function is formulated in path planning to handle the stiffness variations of sperm without force sensing. A model-predictive controller is designed to minimize the orientation error and manipulation path length. To detect sperm tail for visual feedback, an accuracy of 98% was achieved via deep neural networks. The robotic manipulation of human sperm was performed using a standard clinical setup of a glass micropipette to rotate a sperm to the target orientation. Experimental results showed that robotic sperm rotation achieved an orientation error of 0.8$^{\circ }$, a tail curvedness of 0.14$\mu$m$^{-1}$, and an operation time of 5.6 s, all significantly less than those of the manual approach. The less orientation error and tail curvedness after robotic rotation led to a significantly lower speed of sperm entering the micropipette during sperm aspiration, resulting in a higher success rate of 97% (versus 76% after manual rotation) for aspiration control.

Precise rotational positioning of end-effectors under microscopy is crucial for robotic micromanipulation. However, the end-effector is presently limited to a fixed orientation, which is manually set before a given micromanipulation task, lacking accuracy and versatility ofin situreorientation. In this article, we present a unified framework for rotationally positioning the end-effector in three dimensions by establishing a general rotational model, developing a detection method within the limited field of view under microscopy, and designing a three-loop control strategy that adapts to different experimental requirements and model parameters. In experiments, a standard angled micropipette was used as the end-effector to verify the validity of the proposed methods. The performance was evaluated experimentally where the micropipette was robotically rotated to an arbitrarily desired orientation with an average orientation error less than$2^\circ$. In the experiments of sperm manipulation, thein situmicropipette orientation control capability improved the success rate of sperm immobilization and achieved dexterous robotic sperm orientation for facile aspiration.

Three-dimensional structured light (SL) measurement of highly reflective surface is a challenge faced in industrial metrology. The high dynamic range (HDR) technique provides a solution by fusing images under multiple exposures; however, the process is highly time-consuming. This article reports a new SL-based method to measure parts with highly reflective surfaces from only a single exposure. A new quantitative metric is defined to optimally select camera exposure for capturing input single-exposure images. Different from existing image gradient or entropy-based metrics, the new metric incorporates both intensity modulation and overexposure. A skip pyramid context aggregation network (SP-CAN) is proposed to enhance the single exposure-captured images. Compared with existing image enhancement methods, SP-CAN effectively preserves detailed encoded phase information near edges and corners during enhancement. Experiments with various industrial parts demonstrated that the average time cost of the proposed method was 0.6 s, which was only one tenth of the HDR method (ten exposures), and the two methods achieved similar coverage rates (97.6% versus 98.0%) and measurement accuracy (0.040 mm versus 0.038 mm).

This article reports an automated optically-induced dielectrophoresis (ODEP) system for characterizing the specific membrane capacitance (SMC) of individual cells. The simulation of cell motion is conducted to analyze the electrokinetic forces acting on the cell. A self-developed visual tracking algorithm for multicells is used to realize an automated process for determining the frequency-sweeping range, crossover frequencies, and cell radii. The SMC values of malignant bladder cancer cells (T24 and RT4) and normal urothelial cells (SV-HUC-1) were quantified using the automated system, demonstrating that the system has a measurement speed of ~1 cell/s, an accuracy of 1 kHz for the crossover frequency determination, and an accuracy of 0.2 μm for the cell radius measurement.

Robotic manipulation of deformable objects has been a classic topic in robotics. Compared to synthetic deformable objects such as rubber balls and clothes, biological cells are highly deformable and more prone to damage. This article presents robotic manipulation of deformable cells for orientation control (both out-of-plane and in-plane), which is required in both clinical (e.g., in vitro fertilization) and biomedical (e.g., clone) applications. Compared to manual cell orientation control based on empirical experience, the robotic approach, based on modeling and path planning, effectively rotates a cell, while consistently maintaining minimal cell deformation to avoid cell damage. A force model is established to determine the minimal force applied by the micropipette to rotate a spherical or, more generally, ellipsoidal oocyte. The force information is translated into indentation through a contact mechanics model, and the manipulation path of the micropipette is formed by connecting the indentation positions on the oocyte. An optimal controller is designed to compensate for the variations of mechanical properties across oocytes. The polar body of an oocyte is detected by deep neural networks with robustness to shape and size differences. In experiments, the system achieved an accuracy of 97.6% in polar body detection and an accuracy of 0.7° in oocyte orientation control with maximum oocyte deformation of 2.70 μm throughout the orientation control process.

Measuring cell motility and morphology is important for revealing their functional characteristics. This paper presents automation techniques that enable automated, non-invasive measurement of motility and morphology parameters of single sperm. Compared to the status quo of qualitative estimation of single sperm's motility and morphology manually, the automation techniques provide quantitative data for embryologists to select a single sperm for intracytoplasmic sperm injection. An adapted joint probabilistic data association filter was used for multi-sperm tracking and tackled challenges of identifying sperms that intersect or have small spatial distances. Since the standard differential interference contrast (DIC) imaging method has side illumination effect which causes inherent inhomogeneous image intensity and poses difficulties for accurate sperm morphology measurement, we integrated total variation norm into the quadratic cost function method, which together effectively removed inhomogeneous image intensity and retained sperm's subcellular structures after DIC image reconstruction. In order to relocate the same sperm of interest identified under low magnification after switching to high magnification, coordinate transformation was conducted to handle the changes in the field of view caused by magnification switch. The sperm's position after magnification switch was accurately predicted by accounting for the sperm's swimming motion during magnification switch. Experimental results demonstrated an accuracy of 95.6% in sperm motility measurement and an error <10% in morphology measurement.

Ampoule injection is a routinely used treatment in hospitals due to its rapid effect after intravenous injection. During manufacturing, tiny foreign particles can be present in the ampoule injection. Therefore, strict inspection must be performed before ampoule injections can be sold for hospital use. In the quality control inspection process, most ampoule enterprises still rely on manual inspection which suffers from inherent inconsistency and unreliability. This paper reports an automated system for inspecting foreign particles within ampoule injections. A custom-designed hardware platform is applied for ampoule transportation, particle agitation, and image capturing and analysis. Constructed trajectories of moving objects within liquid are proposed for use to differentiate foreign particles from air bubbles and random noise. To accurately classify foreign particles, multiple features including particle area, mean gray value, geometric invariant moments, and wavelet packet energy spectrum are used in supervised learning to generate feature vectors. The results show that the proposed algorithm is effective in classifying foreign particles and reducing false positive rates. The automated inspection system inspects over 150 ampoule injections per minute (versus ~ 12 ampoule injections per minute by technologist) with higher accuracy and repeatability. In addition, the automated system is capable of diagnosing impurity types while existing inspection systems are not able to classify detected particles.

Mammalian sperms reorient against fluid flow in the female reproductive tract, known as rheotaxis. Compared to chemotaxis that provides short-distance guidance, rheotaxis provides long-distance guidance for a sperm to find the egg cell. However, only a low number of sperms are capable of rheotaxis and their tail behavior during reorientation is not yet known. We have developed an automated system to manipulate human sperm orientation in fluid flow and quantitatively reveal sperm behavior changes during rheotaxis. The system automatically detects multiple sperms, selects the sperm for analysis, controls fluid flow, and quantifies sperm tail behavior. Sperm head angle is used as feedback to control fluid flow and select reorienting sperms. High accuracy of head angle tracking and automated sperm selection enables the capturing of dynamic sperm turning behavior in a large sample size. Algorithms are developed to track sperm tail skeletons and quantify tail beating amplitude and asymmetry, based on which the first quantitative analysis of sperm tail behavior in rheotaxis is obtained. Experimental results reveal, for the first time, that the sperms that are capable of reorienting against fluid flow beat their tails more asymmetrically than those sperms that are unable to reorient against fluid flow while no significant difference was found in their tail beating amplitudes.

This paper reports the first robotic system for vitrification of mammalian embryos. Vitrification is a technique for preserving oocytes and embryos in clinical IVF (in vitro fertilization). The procedure involves multiple steps of stringently timed pick-and-place operation for processing an oocyte/embryo in vitrification media. In IVF clinics, vitrification is conducted manually by highly skilled embryologists. Processing one oocyte/embryo occupies the embryologist 15–20 minutes, depending on protocols chosen to implement. Due to poor reproducibility and inconsistency across operators, success rates and survival rates also vary significantly. Through collaboration with IVF clinics, we are in process to realize robotic vitrification and aim ultimately to standardize clinical vitrification from manual operation to fully automated robotic operation. Our robotic system is embedded with two contact detection methods to determine the relative Z positions of the vitrification micropipette, embryo, and vitrification straw. A 3D tracking algorithm is developed for visually servoed embryo transfer and real-time monitoring of embryo volume changes during vitrification. Excess medium is automatically removed from around the vitrified embryo on the vitrification straw to achieve a high cooling rate. Tests on mouse embryos demonstrate that the system is capable of performing vitrification with a throughput at least three times that of manual operation and achieved a high survival rate (88.9%) and development rate (93.8%).

Most mammalian cells (e.g., cancer cells and cardiomyocytes) adhere to a culturing surface. Compared to robotic injection of suspended cells (e.g., embryos and oocytes), fewer attempts were made to automate the injection of adherent cells due to their smaller size, highly irregular morphology, small thickness (a few micrometers thick), and large variations in thickness across cells. This paper presents a recently developed robotic system for automated microinjection of adherent cells. The system is embedded with several new capabilities: automatically locating micropipette tips; robustly detecting the contact of micropipette tip with cell culturing surface and directly with cell membrane; and precisely compensating for accumulative positioning errors. These new capabilities make it practical to perform adherent cell microinjection truly via computer mouse clicking in front of a computer monitor, on hundreds and thousands of cells per experiment (vs. a few to tens of cells as state-of-the-art). System operation speed, success rate, and cell viability rate were quantitatively evaluated based on robotic microinjection of over 4,000 cells. This paper also reports the use of the new robotic system to perform cell-cell communication studies using large sample sizes. The gap junction function in a cardiac muscle cell line (HL-1 cells), for the first time, was quantified with the system.

In robotic micromanipulation, end-effector tips must be first located under microscopy imaging before manipulation is performed. The tip of micromanipulation tools is typically a few micrometers in size and highly delicate. In all existing micromanipulation systems, the process of locating the end-effector tip is conducted by a skilled operator, and the automation of this task has not been attempted. This paper presents a technique to automatically locate end-effector tips. The technique consists of programmed sweeping patterns, motion history image end-effector detection, active contour to estimate end-effector positions, autofocusing and quad-tree search to locate an end-effector tip, and, finally, visual servoing to position the tip to the center of the field of view. Two types of micromanipulation tools (micropipette that represents single-ended tools and microgripper that represents multiended tools) were used in experiments for testing. Quantitative results are reported in the speed and success rate of the autolocating technique, based on over 500 trials. Furthermore, the effect of factors such as imaging mode and image processing parameter selections was also quantitatively discussed. Guidelines are provided for the implementation of the technique in order to achieve high efficiency and success rates.

Pick-place of single nanowires inside scanning electron microscopes (SEM) is useful for prototyping functional devices and characterizing nanowires's properties. Nanowire pick-place has been typically performed via teleoperation, which is time-consuming and highly skill-dependent. This paper presents an automated approach to the pick-place of single nanowires. Through SEM visual detection and vision-based motion control, the system automatically transferred individual silicon nanowires from their growth substrate to a microelectromechanical systems (MEMS) device that characterized the nanowires's electromechanical properties. The performance of the nanorobotic pick-up and placement procedures was experimentally quantified.

This paper presents a nanomanipulation system for operation inside scanning electron microscopes (SEM). The system is small in size, capable of being mounted onto and demounted from an SEM through the specimen exchange chamber without breaking the high vacuum of the SEM. This advance eliminates frequent opening of the high-vacuum chamber, thus, incurs less contamination to the SEM, avoids lengthy pumping, and significantly eases the exchange of end-effectors (e.g., nano probes and grippers). The system consists of two independent 3-DOF Cartesian nanomanipulators based on piezo motors and piezo actuators. High-resolution optical encoders are integrated into the nanomanipulators to provide position feedback for closed-loop control. A look-then-move control system and a contact detection algorithm are implemented for horizontal and vertical nanopositioning. The system design, system characterization details, and system performance are described.