Segmenting individual trophectoderm (TE) cells is essential for developing quantitative metrics to assess the developmental potential of human blastocysts. The elongated shape and circular arrangement of TE cells lead to continuously varying orientations across the image, posing challenges for existing cell instance segmentation methods that assume uniformly oriented cells. As a result, most methods segment the TE as a single region, and the development of quantitative, cell-level metrics predictive of live birth potential remains unexplored. In this work, we propose an instance segmentation model that represents elongate, circularly arranged TE cells using elliptical distance maps, with which superior performance in both segmentation accuracy and metric extraction was achieved, compared with state-of-the-art methods. The extracted metrics, including TE cell number, the mean and standard deviation of cell length, width, area, and mean inter-cell distance, serve as effective predictors of a blastocyst’s live birth potential. When used to predict live birth, these metrics achieved a significantly higher area under the receiver operating characteristic curve (AUC = 0.693) than traditional TE morphological grades (AUC = 0.585). The source code is publicly available at https://github.com/robotVisionHang/TESeg.

Quantitative analysis of cleavage-stage embryos is a critical step for embryo evaluation in in vitro fertilization (IVF) treatment. Deep learning-based instance segmentation methods have emerged as a promising solution. However, existing methods, typically designed for opaque instances in nature scenes, face challenges when applied to segmenting overlapping cells in cleavage-stage embryos. The semi-transparency of the cells in cleavage-stage embryos leads to vague boundaries and complex overlaps between cells, making it difficult for existing models to segment these cells without merging or missing cells. In this work, we propose a novel instance segmentation network for segmenting overlapping, semi-transparent cleavage-stage embryo cells. Specifically, the network introduces two specialized branches that learn complementary representations: an overlap branch for addressing inter-instance overlaps and a boundary branch for dealing with intra-instance shapes. Then, an attention-based feature fusion module is designed to integrate learned overlap and boundary features to enhance the instance representation for the segmentation branch. Finally, an overlap-aware post-processing module leverages the outputs from the overlap and boundary branches to adaptively adjust suppression thresholds, ensuring the preservation of distinct yet highly overlapping cells. Experiments on two cleavage-stage embryo datasets demonstrate that our method outperformed state-of-the-art methods in both segmentation accuracy and recall, highlighting its potential for quantitative embryo evaluation in clinical IVF.

Untethered capsules are capable of entering the gastrointestinal (GI) tract and collecting fluid samples containing microbial communities from specific locations, facilitating the study of chronic diseases. However, existing sampling capsules are designed for single-site sampling, making it challenging to gather samples from multiple targets. This paper reports a magnetic-driven capsule for multiple sampling within the GI tract and an on-demand magnetic-triggered fluid sampling strategy. The capsule consists of a body, a magnetic-triggered negative pressure unit, and a reservoir unit. Composed of an elastic membrane and Magnet I, the negative pressure unit controls pressure change inside the capsule cavity on demand to pump the sample by switching the magnetic field, while the embedded Magnet I also enables real-time magnetic localization for regional targeting and position tracking. The reservoir unit integrates three sampling papers for fluid absorption, two waterproof layers that maintain contamination levels below 25% to ensure reliable multi-site sampling, and a rotating arm embedded with Magnet II for posture adjustment of the sampling paper. The pumping and storage performance of the capsule was systematically evaluated and optimized. Meanwhile, the capsule, actuated by an external magnetic field, was evaluated for its active locomotion performance. Finally, the feasibility of using the capsule to perform active navigation and multi-target sampling in a porcine intestine was validated viaex vivoexperiments.

Current numerical methods for solving kinematic identification (KI) and inverse kinematics (IK) are limited in accuracy, convergence rates, and robustness, necessitating further enhancement. This paper presents a modified Halley method for solving the KI and IK problems of serial robots based on the product of exponentials formula, achieving quintic convergence. Specifically, a general error model is first established based on exponential coordinates, and KI and IK are reformulated as root-finding problems. Next, the modified Halley method, which we prove to be a fifth-order method and incorporates a damping strategy, is proposed to resolve the singularity issue and enhance robustness. Subsequently, the Jacobian and Hessian matrices required for the proposed method are analytically derived based on the time differential of exponentials. Furthermore, highly simplified explicit formulas for these matrices are presented for the IK problem. Simulations on serial robots with various configurations validate the proposed method's accuracy, convergence rates, and robustness in solving KI and IK problems, as well as its advantages over the state-of-the-art. Additionally, experimental validation of KI on two physical robots further demonstrates the effectiveness of the proposed method. Our custom-written MATLAB and C++ codebases are made publicly available for download.

The aspiration to replicate the capabilities of the human hand has driven innovations in the design of soft robotic hands. Despite these advancements, many existing designs of soft hands still lack effective in-hand vision and the ability for each finger to achieve active multi-degree-of-freedom motion. This paper proposes a cable-driven soft robotic hand that can achieve dexterous grasping and manipulation, vision-guided grasping, vision-based slip detection and compensation, as well as visually servoed in-hand manipulation. The hand has five soft fingers, each capable of independent flexion/extension motion and bidirectional ad/abduction motion. A red-green-blue-depth (RGB-D) camera is integrated into the palm of the soft hand to enable in-hand vision capability. Modeling of the soft hand is established to analyze its kinematics, statics, and manipulability. A series of experiments are conducted to demonstrate its dexterous grasping and manipulation capabilities on a variety of objects. Using 3D point cloud data from the in-palm camera, an effective vision-guided grasping strategy is developed to grasp objects on a table. The in-hand vision also enables slip detection and compensation during grasping to maintain the grasp stability. Furthermore, a hierarchical, visually servoed controller is developed to perform closed-loop in-hand object manipulation. With its high dexterity and visual feedback capabilities, the soft hand will find important applications such as household object manipulation and food picking/sorting, and may also be used as a prosthetic hand or an auxiliary hand for humans.

Robotic steering of magnetic guidewires has shown great potential in accelerating endovascular interventions, enhancing the success rate of time-sensitive surgeries such as stroke treatment. Incomplete state feedback of the guidewire from 2D perspective images and unknown interactions with the surrounding vessel wall raise challenges in modeling and steering control. These two factors, however, are commonly overlooked by existing works. In this paper, 2D perspective images of the guidewire, which comply with prevalent medical imaging modalities, are used as the only feedback. A model-based external force observer is proposed that allows the guidewire to perceive the unknown interactions, and a compliance controller is subsequently designed to handle the external force while steering the guidewire. Experiments conducted in a human-sized phantom demonstrate how the compliance controller preserves stability and safety by adapting to the estimated external force.

Quantifying sperm flagellar beating behavior (e.g., beating amplitude, frequency, and wavelength) plays a crucial role in biological research, clinical diagnostics, and the design of sperm-inspired microrobots. However, existing computational methods struggle to accurately and efficiently analyze the highly dynamic, complex, and fine structures of sperm flagella, especially when portions of the flagellum become invisible due to three-dimensional out-of-focus beating. This paper proposes an automated high-throughput tool for quantitative analysis of sperm flagellar beating. The core innovation is continuous convolution (CConv), which adaptively captures the irregular, time-varying patterns of sperm flagella while ensuring continuity in segmentation outputs, even in the presence of locally invisible regions caused by out-of-focus motion. CConv can be integrated into various neural network architectures as a plug-and-play module. Extensive experiments demonstrate that integrating CConv consistently improves the accuracy and continuity of flagella segmentation across different networks. Furthermore, utilizing a curvature-based approach, we quantified key flagellar beating parameters, including length, amplitude, frequency, and wavelength. Applying the high-throughput tool on 1200 sperm revealed that sperm from fertile donors had significantly higher flagellar beating frequency than sperm from infertile patients. The proposed automated tool unlocks high-throughput, quantitative analysis of sperm flagellar beating, showing the potential for applications in reproductive biology and engineering research. The codes and datasets will be released at https://github.com/Goldfish-Yu/CConv.

Video object detection (VOD) of motile cells (e.g., bacteria and sperm) under microscopy is challenging due to motion blur, sporadic out-of-focus, and pose variations. Compared with VOD in generic scenes, the lower contrast and smaller color space of microscopy imaging further introduce feature overlap between the foreground objects and the background objects (e.g., impurity cells and contaminants). Transformer-based methods have achieved great success in the VOD of generic scenes by utilizing object queries to model the inner-frame objects and the inter-frame objects. However, the appearance overlap problem in microscopy video frames significantly compromises the inter-frame query aggregation by introducing background features into the object query. To tackle this challenge, this paper reports a static-dynamic query-based VOD network that treats object queries of the current video frame and reference video frames differently. Specifically, a two-stage framework is implemented that first generates high-quality object queries of reference frames with a static Transformer decoder pre-trained on a still image dataset. The network is then trained on a per-frame annotated dataset using a dynamic Transformer decoder to model the object queries of the current frame. A Reference Query Relation Module is further proposed to enhance the reference queries for more effective aggregation with the current query. Experiments on clinically collected biopsied sperm datasets validated the effectiveness of the proposed method.

Sperm morphology measurement is vital for diagnosing male infertility, which involves quantification of multiple subcellular parts for each sperm. Instance-aware part segmentation networks have been introduced to address this task by automatically identifying individual sperm and segmenting their subcellular parts. However, major limitations of state-of-the-art instance-aware part segmentation networks include: 1) they are time-consuming and computational expensive due to sequential processing and multi-stage frameworks; 2) they perform poorly for densely packed sperm that overlap or cross over one another. To overcome these challenges, this paper proposes 1) integrating instance identification and subcellular part segmentation within a single-stage framework to save inference time and memory usage; 2) dividing a sperm target into simpler components (head and tail) to improve prediction accuracy, followed by a contrastive learning-based matching method to pair the head and tail. Experimental results on our clinically collected human sperm dataset demonstrated that the proposed network not only outperformed state-of-the-art CP-Net (by 3.5% APp vol) but also achieved realtime inference (48.0 frames per second), effectively meeting the clinical requirements for automated parts segmentation of sperm. final part segmentation results. 2) Since the sperm head and tail have simpler shapes, they are detected separately to improve segmentation accuracy. A contrastive learning-based method is then designed to pair head and tail based on similarity of feature embeddings extracted from the proposed instance prediction branch. The proposed method significantly outperformed existing networks, particularly in handling densely packed sperm. The presented method has applicability to analyzing sperm and more broadly other cell types.

Automated morphological measurement of cellular and subcellular structures in live cells is important for evaluating cell functions. Due to their small size and transparent appearance, visualizing cellular and subcellular structures often requires high magnification microscopy and fluorescent staining. However, high magnification microscopy gives a limited field of view, and fluorescent staining alters cell viability and/or activity. Therefore, microscopy image enhancement methods have been developed to predict detailed intracellular structures in live cells. Existing image enhancement networks are mostly CNN-based models lacking global information or Transformer-based models lacking local information. For these purposes, a novel CNN-Transformer combined bilateral U-Net (CTBUnet) is proposed to effectively aggregate both local and global information. Experiments on the collected sperm cell enhancement dataset demonstrate the effectiveness of proposed network for both super-resolution and virtual staining prediction.

This paper presents an automated point-of-care semen analysis method that uses smartphone imaging to visually measure sperm concentration and motility of semen samples. The proposed method follows the same visual tracking scheme as laboratory semen analysis systems, aiming to match clinical standards while being suitable for point-of-care use. A boundary-sensitive segmentation network is developed to identify and distinguish sperm from impurities in raw semen. A novel occlusion-aware multi-sperm tracking algorithm is proposed to tackle challenges posed by smartphone imaging and undiluted raw semen samples. For automated motility measurement, an occlusion-awareness module is proposed to robustly track multiple sperm during frequent sperm crossover/occlusion. The module combines the segmented contour and kinematic-based probabilistic modeling to determine the occlusion status of both targets and measurements, facilitating fundamental improvement to feasible joint event enumeration to enable robust data association. The proposed method achieved a high success rate of 95.14% for tracking occluded sperm, with low mean errors for sperm concentration (2.03 million/ml) and motility (1.58%), outperforming existing multi-sperm tracking methods. In clinical tests involving 50 participants, our method exhibited good agreement with clinical standards (Spearman rank correlation coefficients of 0.94 for concentration and 0.89 for motility) even when used by inexperienced users.Note to Practitioners—Semen analysis is the gold standard method for assessing male reproductive capacity. Clinical semen analysis routinely uses professional computer-assisted semen analysis (CASA) systems to examine sperm concentration and motility; however, clinical visits for semen analysis are not always feasible due to the unavailability of such professional systems and the mental stress brought by clinical visits. This work provides a point-of-care semen analysis method. For hardware, a smartphone microscopic imaging modality was developed to enable clear visualization of sperm with the built-in smartphone camera. For software, an occlusion-aware multi-sperm tracking algorithm was proposed to automatically measure sperm concentration and motility. In addition to intensively validating the proposed point-of-care method against clinical CASA systems, this work also analyzed different types of tracking failure and quantified their effects on the automated evaluation of sperm concentration and motility. The techniques pave the way for further improvement in both point-of-care and clinical semen analysis. Overall, this work offers an accessible and reliable tool for automated male fertility evaluation.

Automated positioning systems play a pivotal role in micro-scale cell manipulation. In clinical intracytoplasmic sperm injection (ICSI) for infertility treatment, a motile sperm needs to be immobilized by glass micropipette tapping for subsequent surgical steps. The process requires accurate tracking of the target sperm and precise alignment between the sperm tail and the micropipette. Manual sperm immobilization suffers from inconsistent success rates, and current robotic systems developed for the task fail to comply with the standard clinical setup. Instead of using a motorized micromanipulator as in existing robotic systems, this paper presents an automated and compact three-dimensional (3-D) positioning stage for sperm immobilization that can be seamlessly integrated into standard clinical platforms. To tackle the challenge of accurately tracking the target sperm with degraded detection quality due to the complex 3-D motion of the positioning stage, a multi-stage sperm tracking scheme is designed for detection-to-tracklet association. To prevent physical contact between the sperm head and the micropipette, an adaptive tail-tapping planning strategy based on the sperm head orientation analysis is established. A visual servo controller equipped with a dynamic sperm motion observer is further employed to achieve precise positioning of the target sperm during the immobilization process. Experimental results demonstrated that the proposed system achieved a sperm tracking accuracy of 88.12%, and a sperm positioning accuracy of$2.3~\pm ~1.2~\mu $m. Further experiments revealed the system achieved a success rate of 93.5% and a time cost of 5.5 s for automated sperm immobilization. Note to Practitioners—This work presents an automated positioning system for the robotic immobilization of live human sperm in the intracytoplasmic sperm injection (ICSI) process. Conventional robotic systems developed for sperm immobilization use motorized stages which only provide two degrees of freedom (DOF) and disturb standard clinical setups with bulky sizes and additional equipment. Leveraging the advantages of piezoelectric positioners, a compact 3-D positioning system is developed to perform robotic sperm immobilization in an all-in-one manner. A sperm immobilization tracker is developed to track the target sperm during the immobilization process. The proposed multi-stage data association metric can effectively resist the noisy detection results due to occlusion and out-of-focus blur introduced by the 3-D movement of the positioning stage. Based on the analysis of the sperm head orientation, an adaptive tail-tapping planning strategy is established to avoid the risk of contacting the sperm head where DNA is contained. The developed positioning system can be easily integrated into a standard microscopy operation platform for biological cell manipulation. The proposed tracking scheme is applicable in various microscopy cell analysis scenarios where the detection results are inevitably affected by the degraded image quality.

Point cloud clustering is a promising method for 3D surface defect diagnosis in manufacturing but requires manual clustering parameter selection, reducing usability. This paper proposes an automatic point cloud clustering method to address this issue. It employs a strategy that progresses from coarse to fine. In the coarse searching stage, a K-Nearest Neighbor (KNN) graph analysis technique is developed to recognize potential defective regions in parallel. Moving on to the fine stage of extracting detailed defects, a modified DBSCAN algorithm is proposed, in which the clustering parameters are calculated automatically from the KNN graph analysis results. Experimental results showed that the proposed method achieved cloud clustering with automatically calculated clustering parameters for surface defect diagnosis. The proposed method outperformed the traditional region growing algorithm in accuracy (0.942 vs. 0.680) and processing speed (21500 points/sec vs. 8740 points/sec) without requiring manual intervention.Note to Practitioners—This paper presents a method for diagnosing defects on automobile and flat steel surfaces. Current 3D point cloud techniques for surface defect diagnosis require manual parameter adjustments, reducing usability. This paper proposes an automatic method without manual intervention. The proposed method uses a coarse-to-fine strategy. The 3D point cloud is divided into sub-blocks to locate potential defects, and a clustering algorithm then extracts detailed defects with automatically determined parameters. We mathematically characterize changes in point density caused by surface defects and show how these features can be used for clustering parameter calculation. Experimental results demonstrate the method’s efficiency on flat as well as some curved surfaces, but it has yet to be evaluated on complex structures. Future work will aim to broaden its application to include a more extensive variety of surfaces and integrate it with robotic vision systems.

Vehicle trajectory prediction is gaining significant attention from academia and industry because of its vital role in autonomous driving. However, current theories face two challenges. First, they generally underperform when faced with longer historical trajectory inputs, especially in large-scale scenarios. Second, they usually ignore the temporal continuity of the target vehicle itself. To address these issues, our study proposed a novel extended temporal sequence fusion attention (ETSFA) network. This network can fully capture the information from the historical trajectory and the dynamic influences of adjacent agents. In addition, the novel dual-channel decoupled model can precisely characterize the intricate spatiotemporal interplay of on-road vehicles. Specifically, the proposed network consists of two main parts. For temporal analysis, the linear inference network (LIN) is reparameterized into complex diagonal forms at the state–space model (SSM) layer to express the linear recurrence capability, thus effectively mining the long-term historical trajectory temporal features of the target vehicle. For spatial analysis, an advanced spatial perception module (SPM) based on graph attention networks (GATs) is proposed to aggregate vehicle and intervehicle interaction features. In addition, a spatial inference module (SIM) based on a convolutional linear inference unit (CONVLIN) is customized for spatiotemporal graph features. Finally, the proposed ETSFA is trained and validated across diverse public datasets, including HighD and NGSIM, demonstrating a marked improvement in the prediction accuracy of the proposed ETSFA over existing methods.

Traditional sperm morphology analysis is based on tedious manual annotation. Automated morphology analysis of a high number of sperm requires accurate segmentation of each sperm part and quantitative morphology evaluation. State-of-the-art instance-aware part segmentation networks follow a "detect-then-segment" paradigm. However, due to sperm’s slim shape, their segmentation suffers from large context loss and feature distortion due to bounding box cropping and resizing during ROI Align. Moreover, morphology measurement of sperm tail is demanding because of the long and curved shape and its uneven width. This paper presents automated techniques to measure sperm morphology parameters automatically and quantitatively. A novel attention-based instance-aware part segmentation network is designed to reconstruct lost contexts outside bounding boxes and to fix distorted features, by refining preliminary segmented masks through merging features extracted by feature pyramid network. An automated centerline-based tail morphology measurement method is also proposed, in which an outlier filtering method and endpoint detection algorithm are designed to accurately reconstruct tail endpoints. Experimental results demonstrate that the proposed network outperformed the state-of-the-art top-down RP-R-CNN by 9.2% ${\mathbf{AP}}_{vol}^p$, and the proposed automated tail morphology measurement method achieved high measurement accuracies of 95.34%,96.39%,91.20% for length, width and curvature, respectively.

Automated positioning systems play a pivotal role in micro-scale cell manipulation. In clinical intracytoplasmic sperm injection (ICSI) of in vitro fertilization (IVF) treatment, a motile sperm needs to be immobilized by glass micropipette tapping for subsequent surgical steps. The process requires accurate tracking of the target sperm and precise alignment between the sperm tail and the micropipette. Manual sperm immobilization suffers from inconsistent success rates, and current robotic systems developed for the task fail to comply with the standard clinical setup. Instead of using a motorized micromanipulator as in existing robotic systems, this paper presents an automated, compact three-dimensional positioning stage for sperm immobilization that can be seamlessly integrated into standard clinical platforms. Based on the analysis of the sperm head orientation, an adaptive tail tapping planning strategy is established to avoid the risk of touching the sperm head where DNA is contained. A visual servo controller equipped with a dynamic sperm motion observer is employed to achieve precise tracking and positioning of the target sperm three-dimensionally. Experimental results revealed the system achieved a success rate of 93.5% and a time cost of 5.5 s for automated sperm immobilization.

Obtaining three-dimensional information, especially the z-axis depth information, is crucial for robotic micromanipulation. Due to the unavailability of depth sensors such as lidars in micromanipulation setups, traditional depth acquisition methods such as depth from focus or depth from defocus directly infer depth from microscopic images and suffer from poor resolution. Alternatively, micromanipulation tasks obtain accurate depth information by detecting the contact between an end-effector and an object (e.g., a cell). Despite its high accuracy, only sparse depth data can be obtained due to its low efficiency. This paper aims to address the challenge of acquiring dense depth information during robotic cell micromanipulation. A weakly-supervised depth completion network is proposed to take cell images and sparse depth data obtained by contact detection as input to generate a dense depth map. A two-stage data augmentation method is proposed to augment the sparse depth data, and the depth map is optimized by a network refinement method. The experimental results show that the MAE value of the depth prediction error is less than 0.3 µm, which proves the accuracy and effectiveness of the method. This deep learning network pipeline can be seamlessly integrated with the robotic micromanipulation tasks to provide accurate depth information.

Instance segmentation of biological cells is important in medical image analysis for identifying and segmenting individual cells, and quantitative measurement of subcellular structures requires further cell-level subcellular part segmentation. Subcellular structure measurements are critical for cell phenotyping and quality analysis. For these purposes, instance-aware part segmentation network is first introduced to distinguish individual cells and segment subcellular structures for each detected cell. This approach is demonstrated on human sperm cells since the World Health Organization has established quantitative standards for sperm quality assessment. Specifically, a novel Cell Parsing Net (CP-Net) is proposed for accurate instance-level cell parsing. An attention-based feature fusion module is designed to alleviate contour misalignments for cells with an irregular shape by using instance masks as spatial cues instead of as strict constraints to differentiate various instances. A coarse-to-fine segmentation module is developed to effectively segment tiny subcellular structures within a cell through hierarchical segmentation from whole to part instead of directly segmenting each cell part. Moreover, a sperm parsing dataset is built including 320 annotated sperm images with five semantic subcellular part labels. Extensive experiments on the collected dataset demonstrate that the proposed CP-Net outperforms state-of-the-art instance-aware part segmentation networks.

Sperm immobilization is a crucial procedure in clinical cell surgery for infertility treatment. Current immobilization is implemented by tapping the sperm tail with a glass micropipette, but its effectiveness is restricted by sperm orientation and ineffective membrane ablation. Ineffective ablation leads to limited release of oocyte activating factors and lowers fertilization rate; and sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. This paper reports automated piezo-assisted sperm immobilization with enhanced efficacy of cell membrane ablation and sperm orientation control. The designed piezo drill consists of two orthogonal vibration modules to generate controlled micropipette vibration along axial and lateral axes. Through stiffness modeling, the flexure joints guide the motion of the central beam of each vibration module. To achieve sperm orientation control, whirl flow is induced by both axial and lateral vibration of the micropipette tip. To immobilize sperm, only micropipette’s axial vibration is generated to prevent lateral vibration from damaging sperm head. A visual servoing scheme is developed by decoupling sperm wiggling from positioning error for immobilization. Experimental results showed that sperm orientation control by the piezo drill achieved an error of 1.4$^{\circ}$and a time cost of 2.5 s. Visual servoing with sperm wiggling decoupling achieved a positioning error of 1.7$\mu$m. Furthermore, the piezo-assisted sperm immobilization technique led to effective membrane ablation. With membrane-impermeable stains, it took 5.6 s for the immobilized sperm to be stained after piezo-assisted immobilization, significantly less than 49.2 s by conventional micropipette tapping.Note to Practitioners—This work tackled the challenge of ineffective membrane ablation and orientation limit in clinical cell surgeries. Conventional manual immobilization suffers from low membrane ablation efficacy, which leads to limited release of oocyte activating factors and lowers fertilization rate. Moreover, sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. In this paper, we propose automation techniques for effective membrane ablation and orientation control of sperm. A clinically compatible piezo drill is developed to generate controllable micropipette motion along both axial and lateral directions. The whirl flow generated by micropipette vibration is employed to rotate sperm, which greatly increased the number of available sperm for immobilization. A visual servoing controller is developed to keep the sperm at the center of field of view for immobilization by decoupling sperm wiggling from positioning error. The developed methods can be generalized to the manipulation of other types of cells. The piezo drill can be used for effective membrane ablation of oocyte, embryo, yeast cell and so on. The orientation control strategy leveraging piezo-induced whirl flow is applicable to non-contact rotation of a variety of microorganism.

This paper reports, for the first time, automated grading of human blastocysts (day-5 embryos) from multi-focus images. Based on a novel attention module, a convolutional neural network (CNN) was developed to predict the morphological grade of a blastocyst. The attention module integrates high-level features extracted from the blastocyst’s multi-focus images. Experimental results revealed that multi-focus blastocyst images help improve the grading accuracies than a single blastocyst image. Comparisons of the accuracy achieved by the model and the average accuracy of five embryologists demonstrated that the proposed model can outperform embryologists in the morphological grading of blastocysts (88% versus 86% for development stage prediction, 83% versus 79% for inner cell mass grade prediction, 89% versus 82% for trophectoderm grade prediction).Note to Practitioners—This work was motivated by the subjectivity and significant intra-and inter-evaluator variations in manual morphological grading of blastocysts. Existing approaches to automate the grading process mainly use a single blastocyst image although multi-focus images captured at different focal planes reveal more morphological features of a blastocyst than a single blastocyst image. This paper describes a new CNN-based method using multi-focus images to improve the grading accuracy. The accuracy of the proposed method was verified on multi-focus images of human blastocysts captured by a standard time-lapse incubator at fixed focal depths ($-$45$\mu$m,$-$30$\mu$m,$-$15$\mu$m, 0$\mu$m, 15$\mu$m, 30$\mu$m, 45$\mu$m).

Multidirectional jumping is commonly found in living creatures and desirable to be integrated into mobile robots for enhanced agility. Existing jumping robots mostly employ complex or cumbersome structures and modular designs to achieve multidirectional jumping. There is a lack of a simple, lightweight, and compact actuator design for multidirectional jumping robots. Here, we present a multidirectional jumping soft robot (MDJSR) driven by a biaxial electrohydraulic actuator (BEHA). The BEHA has a simple structure, i.e., a thin plastic frame-guided film pouch with four pairs of distributed electrodes and enclosed with a dielectric liquid. Inspired by gall midge larvae, the MDJSR exhibits two switchable locomotion strategies, including continuous non-energy-storing jumping to move fast and energy-storing jumping to cross obstacles. Its multidirectional jumping capability was demonstrated in the navigation through a labyrinth with two ways of obstaclecrossing and obstacle-circumventing in different terrain environments. In addition, the robot can be deployed to detect unknown space and collect environmental factors. This work provides an enabling solution to miniature and lightweight multimodal jumping soft robots for various robotic tasks

Automated manipulation of deformable objects is challenging due to the object’s deformation behavior. Different from still deformable objects such as wires and cloth, biological organisms such as sperm and worms are both deformable and motile, requiring the control of both deformation and motion. This paper reports automated orientation control of live sperm, as an example of motile deformable cells. Robotic manipulation of human sperm was performed by using a glass micropipette, which is a standard clinical tool, to rotate individual motile sperm. Sperm rotation must be performed before immobilization, as required in clinical cell surgery for infertility treatment. To control tail deformation during sperm rotation, a path planner was designed based on kinematic analysis and manipulation point update. To deal with the intrinsic motion of a motile sperm, a motorized stage was controlled to compensate for sperm swimming motion, and an observer was designed to decouple sperm orientation from its wiggling motion. A sliding mode controller was designed to cope with stiffness variances along the sperm tail and among different sperm. Deep neural networks were developed for robust sperm tail detection, and Kalman filter was used to predict tail motion. Experimental results demonstrated that automated sperm manipulation achieved an orientation error of 0.8° and operation time of 6.8 s, both significantly less than those of manual operation. The designed observer was effective to reduce sperm orientation error by reducing the disturbance from sperm wiggling motion. The developed sliding mode controller outperformed the PID controller in operation time, reducing the time of oocyte exposure to the ambient environment.Note to Practitioners—This work tackled the challenge of rotating a fast-swimming and deformable sperm in clinical cell surgeries. Automated manipulation of deformable objects has wide applications in industrial and service settings such as manipulating wires and folding cloth. However, the intrinsic motion of a motile sperm and the lack of a rotational degree of freedom in standard micromanipulators pose difficulties to automated sperm manipulation. In this paper, we propose automation techniques for sperm orientation control. For sperm tail detection, deep learning was used to handle the variances of shape and length among different sperm. A path planning strategy and a controller were designed to achieve automated rotation of motile sperm, with its deformation and motion both controlled. The developed methods can be generalized to the manipulation of other deformable objects such as wires, cables and cloth. These objects exhibit significant variance of mechanical properties, and calibration is often time-consuming. The designed controller can be used to manipulate deformable objects with robustness to varied mechanical parameters. Path planning was designed by updating the manipulation point based on the object’s deformation behavior, and is suitable in manipulation where constraints are imposed such as the object’s strain.

Recounts the career and contributions of Peter Luh.

Nanomaterials possess superior mechanical, electrical, and optical properties suitable for device applications in different fields such as nanoelectronics, photonics, and sensors. Characterizing the multiphysical properties of single nanomaterials and nanostructures provides experimental guidelines for synthesis and device applications of functional nanomaterials. Nanomanipulation techniques under scanning electron microscopy (SEM) have enabled the testing of mechanical and electrical properties of various nanomaterials. However, the introduction of micro-photoluminescence ($\mu $-PL) measurement into an SEM setup for in-situ single nanomaterial characterization is still experimentally challenging; in particular, the seamless integration of the mechanical, electrical, and$\mu $-PL testing techniques inside an SEM for multi-field-coupled characterization of single nanostructures is still unexplored. In this work, we report the first SEM-based nanomanipulation system for multiphysical characterization of single nanomaterials. A custom-made, optical-microfiber-based$\mu $-PL setup is integrated onto a nanomanipulation system with four nanomanipulators inside an SEM. The system is also equipped with a conductive nanoprobe and a conductive atomic force microscopy (AFM) probe for electrical nanoprobing and electroluminescence (EL) measurement of single nanomaterials with contact force feedback. Using the system, field-coupled characterization (i.e., optomechanical, optoelectronic, electromechanical, and mechano-optoelectronic testing) of single InGaN/GaN nanowires (NWs) are conducted; and, for the first time, the effect of mechanical compression applied to individual InGaN/GaN NWs on its optoelectronic property is revealed. Note to Practitioners—With the rapid advances of nanophotonics and nanoelectronics, the optical and optoelectronic characterization of semiconductive nanomaterials becomes widely used for guiding the material synthesis and improving the nanodevice performance. However, few studies on optical-relevant characterization were carried out in SEM, mainly due to the limited space of an SEM chamber, making it challenging to integrate optical components for effective optical excitation and luminescence measurement. To address this issue, space-saving optical microfibers were integrated into the SEM chamber for in-situ optoelectronic characterization of semiconductor NWs, along with the seamless integration of mechanical and electrical nanoprobing tools for electromechanical characterization. The developed nanomanipulation system will greatly facilitate the multiphysical testing of semiconductor nanomaterials, and thus expedite their synthesis optimization processes and broaden their optoelectronic device applications.

Soft wall-climbing robots have been limited in their ability to perform complex locomotion in diverse environments due to their structure and weight. Thus far, soft wall-climbing robots with integrated functions that can locomote in complex 3-D environments are yet to be developed. This article addresses this challenge by presenting a lightweight (2.57 g) soft wall-climbing robot with integrated linear, turning, and transitioning motion capabilities. The soft robot employs three pneumatic bending actuators and two adaptive electroadhesion pads, which enable it to flip forward, transition between two walls, turn in two directions, and adhere to various surfaces. Different motion and control strategies are proposed based on a theoretical model. The experimental results demonstrate that the robot can move at an average speed of 3.85 mm/s (0.08 body length/s) on horizontal, vertical, and inverted walls and make transitions between walls with different pinch angles within 180°. Additionally, the soft robot can carry a miniature camera on vertical walls to perform detection and surveillance tasks. This article provides a reliable structure and control strategy to enhance the multifunctionality of soft wall-climbing robots and enable their applications in unstructured environments.

Soft tissue cutting is used for incision, separation and removal of tissues or cells. Due to high deformation of soft tissues resulting from their viscosity and elasticity, it is challenging to accurately cut the tissue along a desired path and control the force applied to the tissue for reducing invasiveness, especially at the microscale. This paper presents a robotic biopsy system for cutting and collecting trophectoderm cells from a highly deformable blastocyst. The system, for the first time, enables TE cell junction detection for laser ablation throughout the blastocyst biopsy process by using a convolutional neural network. The overall detection error was 2.13% in every 1,000 cell junctions with position RMSE of$1.63\ \mu \mathrm{m}\pm 0.29\ \mu \mathrm{m}$. A dynamics model was developed to describe the motion of the trophectoderm cells inside a biopsy micropipette. Based on this model, an adaptive control method was developed for trophectoderm cell aspiration and positioning inside the biopsy micropipette. Experimental results revealed that the controller was capable of effectively compensating for the cell positioning error by updating the varying system parameters according to the adaptation law. The success rate was 100%, the cell aggregate positioning accuracy was$\pm 1\ \mu \mathrm{m}$, the average settling time was 2 s, and the largest overshoot was$4.3\ \mu \mathrm{m}$. Compared to manual blastocyst biopsy, the robotic biopsy system shortened the blastocyst's recovery time (35 min vs. 50 min) which indicates lower invasiveness.

This article addresses the trajectory consensus problem of coordinating the trajectories of vehicles at multiple future time points. The objective is the consensus of the geometry of the vehicles' planned trajectories. The geometric feature of trajectories is parameterized by a set of trajectory states defined as required lengths along the trajectory to reduce the distance to its ending point to specific values. To solve this special consensus problem involving coupled state variables, the conventional consensus model is extended by attaching it to a mapping from the state variables to the trajectory's geometry. This mapping is established using a homotopic structure that creates a compact and efficient form for the mapping. The geometry of the homotopic structure is based on the shapes of its envelopes, and the elements in the structure are derived from their deformation. Through a homotopic search in the structure, an asymptotic consensus of trajectory states is achieved. Simulation results show the proposed coupled state consensus method can achieve better performance on the consensus of multiple vehicles than the conventional isolated state consensus method.

The robotic manipulation of deformable linear objects is a classic and challenging topic. Apart from synthetic objects, such as wires and cables, linear objects are also commonly found in biological cells and organisms. Biomanipulation of such objects is hampered by difficulties, such as limited degrees of freedom of micromanipulators and varied mechanical properties of the biological entities to manipulate. This article presents a robotic manipulation of human sperm, which are deformable cells with a linear shape. The shape and movement of the cell are recapitulated by our developed geometric and kinematic models. Under unfixed constraints between the end-effector and the cell, path planning is designed to update the manipulation point to control cell deformation. A state transition function is formulated in path planning to handle the stiffness variations of sperm without force sensing. A model-predictive controller is designed to minimize the orientation error and manipulation path length. To detect sperm tail for visual feedback, an accuracy of 98% was achieved via deep neural networks. The robotic manipulation of human sperm was performed using a standard clinical setup of a glass micropipette to rotate a sperm to the target orientation. Experimental results showed that robotic sperm rotation achieved an orientation error of 0.8$^{\circ }$, a tail curvedness of 0.14$\mu$m$^{-1}$, and an operation time of 5.6 s, all significantly less than those of the manual approach. The less orientation error and tail curvedness after robotic rotation led to a significantly lower speed of sperm entering the micropipette during sperm aspiration, resulting in a higher success rate of 97% (versus 76% after manual rotation) for aspiration control.

Precise rotational positioning of end-effectors under microscopy is crucial for robotic micromanipulation. However, the end-effector is presently limited to a fixed orientation, which is manually set before a given micromanipulation task, lacking accuracy and versatility ofin situreorientation. In this article, we present a unified framework for rotationally positioning the end-effector in three dimensions by establishing a general rotational model, developing a detection method within the limited field of view under microscopy, and designing a three-loop control strategy that adapts to different experimental requirements and model parameters. In experiments, a standard angled micropipette was used as the end-effector to verify the validity of the proposed methods. The performance was evaluated experimentally where the micropipette was robotically rotated to an arbitrarily desired orientation with an average orientation error less than$2^\circ$. In the experiments of sperm manipulation, thein situmicropipette orientation control capability improved the success rate of sperm immobilization and achieved dexterous robotic sperm orientation for facile aspiration.

Cell manipulation is a key technology in many biomedical and clinical applications, in which end-effector alignment is a critical procedure. Presently, end-effector alignment is performed manually and suffers from large misalignment error and inconsistency. Manual alignment often undesirably moves the end-effector (e.g., a glass micropipette) out of the limited field of view under microscopy and risks breaking the fragile end-effector. This paper presents automated end-effector alignment for robotic cell manipulation. A rotational degree of freedom was added to a micromanipulator with translational degrees of freedom. The kinematic model of end-effector’s rotation was established, and the unknown model parameters were calibrated and updated via quadratic optimization. A controller was designed based on the kinematics modeling and parameter optimization to compensate for rotation-induced translation and achieve end-effector alignment. Experimental results demonstrate that the robotic alignment technique achieved an accuracy of 0.6±0.3° and a time cost of 18.5 ± 10.2 s, both significantly less than manual alignment. The developed controller cost significantly less time for micropipette alignment than the PID controller. A glass micropipette was used as the end-effector for human sperm immobilization, a critical procedure in clinical cell surgery. The success rate of sperm immobilization was 97% by robotic micropipette alignment, higher than the success rate of 90% by manual alignment due to the higher accuracy of robotic alignment.

Three-dimensional structured light (SL) measurement of highly reflective surface is a challenge faced in industrial metrology. The high dynamic range (HDR) technique provides a solution by fusing images under multiple exposures; however, the process is highly time-consuming. This article reports a new SL-based method to measure parts with highly reflective surfaces from only a single exposure. A new quantitative metric is defined to optimally select camera exposure for capturing input single-exposure images. Different from existing image gradient or entropy-based metrics, the new metric incorporates both intensity modulation and overexposure. A skip pyramid context aggregation network (SP-CAN) is proposed to enhance the single exposure-captured images. Compared with existing image enhancement methods, SP-CAN effectively preserves detailed encoded phase information near edges and corners during enhancement. Experiments with various industrial parts demonstrated that the average time cost of the proposed method was 0.6 s, which was only one tenth of the HDR method (ten exposures), and the two methods achieved similar coverage rates (97.6% versus 98.0%) and measurement accuracy (0.040 mm versus 0.038 mm).

In biology, fluorescent dyes are routinely used for biochemical measurements such as pH and ion concentrations. They, especially when used for detecting a low concentration of ions, suffer from low signal-to-noise ratios (SNR); and increasing the concentration of fluorescent dyes causes more sever cytotoxicity. We invented a new approach that uses a low amount of fluorescent dye-coated magnetic nanoparticles for on-demand, accurately aggregating the nanoparticles and thus fluorescent dyes in a local region inside a cell for intracellular measurement. Experiments proved this approach is capable of achieving a significantly higher SNR and lower cytotoxicity. Different from existing magnetic micromanipulation systems that generate large swarms (several microns and above) or cannot move the generated swarm to an arbitrary position, we developed a five-pole magnetic micromanipulation system and technique for generating a small swarm (e.g., 1 μm; capable of generating a magnetic swarm from 0.52 μm to 52.7 μm with an error <; 7.5 %) and accurately positioning the small swarm (position control accuracy: 0.76 μm). As an example, the system performed intracellular pH mapping using a 1 μm swarm of pH sensitive fluorescent dye-coated magnetic nanoparticles. The swarm had an SNR inside a cell 10 times that by the traditional method, i.e., global dye treatment, with both cases using the same fluorescent dye concentration. Our intracellular measurement results, for the first time, quantitatively revealed the existence of pH gradient and polarized pH distribution in live migrating cells.

Existing approaches for robotic control of magnetic swarms are not capable of generating magnetic aggregates precisely in an arbitrarily specified target region in a fluidic flow environment. Such a swarm control capability is demanded by medical applications such as clinical embolization (i.e., localized clogging of blood vessels). This paper presents a new magnetic swarm control strategy to generate aggregates only in a specified target region under fluidic flow. Within the target region, the magnetic field generates sufficiently large magnetic forces among magnetic particles to maintain the aggregates' integrity at the junctions of blood vessels. In contrast, unintended aggregates outside the target region are disassembled by fluidic shear. The aggregation control approach achieved a mean absolute error of 0.15 mm in positioning a target region and a mean absolute error of 0.30 mm in controlling the target region's radius. With thrombin coating, 1 μm magnetic particles were controlled to perform embolization both in vitro (using microfluidic channel networks) and ex vivo (using porcine tissue). Experiments proved the effectiveness of the swarm control technique for on-demand, targeted embolization.

A 3D measurement system consisting of a 3D scanner and an industrial robot (eye-in-hand) is commonly used to scan large object under test (OUT) from multiple fieldof-views (FOVs) for complete measurement. A data stitching process is required to align multiple FOVs into a single coordinate system. Marker-free stitching assisted by robot’s accurate positioning becomes increasingly attractive since it bypasses the cumbersome traditional fiducial marker-based method. Most existing methods directly use initial Denavit-Hartenberg (DH) parameters and hand-eye calibration to calculate the transformations between multiple FOVs. Since accuracy of DH parameters deteriorates over time, such methods suffer from high stitching errors (e.g., 0.2 mm) in long-term routine industrial use. This paper reports a new robot-scanner calibration approach to realize such measurement with low data stitching errors. During long-term continuous measurement, the robot periodically moves towards a 2D standard calibration board to optimize kinematic model’s parameters to maintain a low stitching error. This capability is enabled by several techniques including virtual arm-based robot-scanner kinematic model, trajectory-based robot-world transformation calculation, nonlinear optimization. Experimental results demonstrated a low data stitching error (< 0.1 mm) similar to the cumbersome marker-based method and a lower system downtime (< 60 seconds vs. 10-15 minutes by traditional DH and hand-eye calibration).

Functional nanomaterials possess exceptional multi-physical (e.g., mechanical, electrical and optical) properties compared with their bulk counterparts. To facilitate both synthesis and device applications of these nanomaterials, it is highly desired to characterize their multi-physical properties with high accuracy and efficiency. The nanomanipulation techniques under scanning electron microscopy (SEM) has enabled the testing of mechanical and electrical properties of various nanomaterials. However, the seamless integration of mechanical, electrical, and optical testing techniques into an SEM for triple-field-coupled characterization of single nanostructures is still unexplored. In this work, we report the first SEM-based nanomanipulation system for high-resolution mechano-optoelectronic testing of single semiconductor InGaN/GaN nanowires (NWs). A custom-made optical measurement setup was integrated onto a four-probe nanomanipulator inside an SEM, with two optical microfibers actuated by the nanomanipulator for NW excitation and emission measurement. A conductive tungsten nanoprobe and a conductive atomic force microscopy (AFM) cantilever probe were integrated onto the nanomanipulator for electrical nanoprobing of single NWs for electroluminescence (EL) measurement. The AFM probe also served as a force sensor for quantifying the contact force applied to the NW during nanoprobing. Using this unique system, we examined, for the first time, the effect of mechanical compression applied to an InGaN/GaN NW on its optoelectronic properties.

This article reports an automated optically-induced dielectrophoresis (ODEP) system for characterizing the specific membrane capacitance (SMC) of individual cells. The simulation of cell motion is conducted to analyze the electrokinetic forces acting on the cell. A self-developed visual tracking algorithm for multicells is used to realize an automated process for determining the frequency-sweeping range, crossover frequencies, and cell radii. The SMC values of malignant bladder cancer cells (T24 and RT4) and normal urothelial cells (SV-HUC-1) were quantified using the automated system, demonstrating that the system has a measurement speed of ~1 cell/s, an accuracy of 1 kHz for the crossover frequency determination, and an accuracy of 0.2 μm for the cell radius measurement.

Robotic manipulation of deformable objects has been a classic topic in robotics. Compared to synthetic deformable objects such as rubber balls and clothes, biological cells are highly deformable and more prone to damage. This article presents robotic manipulation of deformable cells for orientation control (both out-of-plane and in-plane), which is required in both clinical (e.g., in vitro fertilization) and biomedical (e.g., clone) applications. Compared to manual cell orientation control based on empirical experience, the robotic approach, based on modeling and path planning, effectively rotates a cell, while consistently maintaining minimal cell deformation to avoid cell damage. A force model is established to determine the minimal force applied by the micropipette to rotate a spherical or, more generally, ellipsoidal oocyte. The force information is translated into indentation through a contact mechanics model, and the manipulation path of the micropipette is formed by connecting the indentation positions on the oocyte. An optimal controller is designed to compensate for the variations of mechanical properties across oocytes. The polar body of an oocyte is detected by deep neural networks with robustness to shape and size differences. In experiments, the system achieved an accuracy of 97.6% in polar body detection and an accuracy of 0.7° in oocyte orientation control with maximum oocyte deformation of 2.70 μm throughout the orientation control process.

Robotic manipulation of deformable objects (vs. rigid objects) has been a classic topic in robotics. Compared to deformable synthetic objects such as rubber balls and clothes, biological cells are highly deformable and more prone to damage. This paper presents robotic manipulation of deformable cells for orientation control (both out-of-plane and in-plane), which is required in both clinical (e.g., in vitro fertilization) and biomedical (e.g., clone) applications. Compared to manual cell rotation control based on empirical experience, the robotic approach, based on mathematical modeling and path planning, effectively rotates a cell while consistently maintaining minimal cell deformation to avoid cell damage. A force model is established to determine the minimal force applied by the micropipette to rotate a spherical or more generally, an ellipsoidal mouse oocyte. The force information is translated into indentation through a contact mechanics model, and the manipulation path of the micropipette is formed by connecting the indentation positions on the oocyte. A compensation controller is designed to compensate for the variations of mechanical properties across cells. The polar body of an oocyte is detected by deep neural networks with robustness to shape and size differences. Experimental results demonstrate that the system achieved an accuracy of 97.6% in polar body detection and an accuracy of 0.7° in oocyte orientation control with maximum oocyte deformation of 2.69 μm.

Measuring the motility and morphology parameters of motile cells is important for revealing their functional characteristics. This paper presents automation techniques that, for the first time, enable automated, non-invasive measurement of motility and morphology parameters of individual sperms. Compared to the status quo of qualitative estimation of single sperm's motility and morphology based on embryologists' empirical experience, the automation techniques provide quantitative data in nearly real time. An adapted joint probabilistic data association filter (JPDAF) was used for multi-sperm tracking and tackled challenges of identifying sperms that intersect or have small spatial distances. Since the standard differential interference contrast (DIC) imaging method has side illumination effect which causes inherent inhomogeneous image intensity and poses difficulties for accurate sperm morphology measurement, we integrated total variation norm into the quadratic cost function method, which together effectively removed inhomogeneous image intensity and retained sperm's subcellular structures after DIC image reconstruction. In order to relocate the same sperm of interest identified under low magnification after switching to high magnification, coordinate transformation was conducted to handle the changes in the field of view caused by magnification switch. Experimental results demonstrated an accuracy of 95.6% in sperm motility measurement and errors <;10% in morphology measurement.

Magnetic micromanipulation is an untethered technique and has enabled numerous applications in the scale of millimeters to micrometers from the tissue level to cell level. However, existing systems are not capable of maneuvering a sub-micrometer object for precise force control, preventing the realization of intracellular manipulation or `fantastic voyage' inside a single cell. The magnetic micromanipulation task achieved in this work is sub-micrometer position control and piconewton force control of a sub-micron (0.7 μm) magnetic bead inside a single human bladder cancer cell (RT4). The magnetic bead was 3D positioned in the cell using a generalized predictive controller that effectively tackled the control challenge caused by the slow visual feedback (1 Hz) from high-resolution confocal microscopy. The average positioning error was quantified to be 0.43 μm, which is slightly larger than Brownian motion-imposed constraint (0.31 μm). The system is capable of three-dimensionally applying a maximum force of 60 pN with a resolution of 4 pN. In experiments, a 0.7 μm magnetic bead was controlled to move from an initial position in a cell to target positions on the cell nucleus. Force-displacement data were obtained from multiple locations along the cell nucleus' major and minor axes. The results revealed, for the first time, significantly higher stiffness exists in the cell nucleus' major axis than the minor axis. This stiffness polarity was likely attributed to the aligned stress fibers of actin filament inside the cells.

Manipulation of motile cells such as bacteria and sperm is required in both cell biology and clinical applications. For immobilizing a motile sperm, the sperm head and tail positions must be accurately tracked, interference of proximal sperms on the target sperm must be tackled, and the orientation of the sperm must be properly aligned with the manipulation tool in order not to damage the sperm head where DNA is contained. Manual operation of sperm immobilization has stringent skill requirements, and both manual operation and existing robotic sperm immobilization suffer from inconsistent success rates and incapability of manipulating sperms swimming in all directions. This paper presents a robotic system for fully automated tracking, orientation control, and immobilization of motile sperms. Algorithms were developed for robustly tracking the sperm head and estimating the sperm tail positions under interfering conditions. A new visual servo control strategy was developed to enable the robotic system to actively adjust sperm orientation for immobilizing a sperm swimming in any direction. Experimental results from robotic immobilization of 400 sperms confirmed that the robotic system achieved a consistent success rate of 94.5 %, independent of sperm velocity or swimming direction.

Measuring cell motility and morphology is important for revealing their functional characteristics. This paper presents automation techniques that enable automated, non-invasive measurement of motility and morphology parameters of single sperm. Compared to the status quo of qualitative estimation of single sperm's motility and morphology manually, the automation techniques provide quantitative data for embryologists to select a single sperm for intracytoplasmic sperm injection. An adapted joint probabilistic data association filter was used for multi-sperm tracking and tackled challenges of identifying sperms that intersect or have small spatial distances. Since the standard differential interference contrast (DIC) imaging method has side illumination effect which causes inherent inhomogeneous image intensity and poses difficulties for accurate sperm morphology measurement, we integrated total variation norm into the quadratic cost function method, which together effectively removed inhomogeneous image intensity and retained sperm's subcellular structures after DIC image reconstruction. In order to relocate the same sperm of interest identified under low magnification after switching to high magnification, coordinate transformation was conducted to handle the changes in the field of view caused by magnification switch. The sperm's position after magnification switch was accurately predicted by accounting for the sperm's swimming motion during magnification switch. Experimental results demonstrated an accuracy of 95.6% in sperm motility measurement and an error <10% in morphology measurement.

Magnetic micromanipulation has the advantage of untethered control, high precision, and biocompatibility and has recently undergone great advances. The magnetic micromanipulation task to tackle in this paper is to three dimensionally navigate a 5-μm magnetic bead inside a mouse embryo and accurately apply forces to intraembryonic structures to perform mechanical measurements at multiple locations. Existing technologies are not able to achieve these navigation and measurement goals because of poor magnetic force scaling and/or lacking the capability of applying an accurately controlled force. This paper reports a three-dimensional magnetic tweezer system that enables, for the first time, intraembryonic magnetic navigation and force application. A single magnetic bead was introduced into a mouse embryo via robotic microinjection. The magnetic tweezer system accurately controlled the position of the magnetic bead via visually servoed magnetic control. By moving the magnetic bead with known forces inside the embryo, cytoplasm viscosity was measured, which is eight times the viscosity of water. For performing mechanical measurements on the cellular structures inside the mouse embryo, the system should be capable of applying forces up to 120 pN with a resolution of 4 pN. The results revealed that the middle region is significantly more deformable than the side regions of the inner cell mass.

Magnetic micromanipulation has the advantage of untethered control, high precision, and biocompatibility and has recently undergone great advances. The magnetic micromanipulation task to tackle in this work is to three-dimensionally navigate a 5-micrometer magnetic bead inside a mouse embryo and perform mechanical measurements at multiple locations. Existing technologies are not able to achieve these navigation and measurement goals because of poor magnetic force scaling and/or lacking the capability of applying an accurately controlled force. This paper reports a robotic magnetic tweezer system that enables, for the first time, intra- embryonic magnetic navigation and force application. A single magnetic bead was introduced into a mouse embryo via robotic microinjection. The robotic magnetic tweezer system accurately controls the position of the magnetic bead via visually servoed magnetic control. The system is also capable of applying forces up to 120 pN with a resolution of 1.78 pN for performing mechanical measurements on the cellular structures inside the mouse embryo, revealing that the middle region is more deformable than the side regions of the inner cell mass.

Ampoule injection is a routinely used treatment in hospitals due to its rapid effect after intravenous injection. During manufacturing, tiny foreign particles can be present in the ampoule injection. Therefore, strict inspection must be performed before ampoule injections can be sold for hospital use. In the quality control inspection process, most ampoule enterprises still rely on manual inspection which suffers from inherent inconsistency and unreliability. This paper reports an automated system for inspecting foreign particles within ampoule injections. A custom-designed hardware platform is applied for ampoule transportation, particle agitation, and image capturing and analysis. Constructed trajectories of moving objects within liquid are proposed for use to differentiate foreign particles from air bubbles and random noise. To accurately classify foreign particles, multiple features including particle area, mean gray value, geometric invariant moments, and wavelet packet energy spectrum are used in supervised learning to generate feature vectors. The results show that the proposed algorithm is effective in classifying foreign particles and reducing false positive rates. The automated inspection system inspects over 150 ampoule injections per minute (versus ~ 12 ampoule injections per minute by technologist) with higher accuracy and repeatability. In addition, the automated system is capable of diagnosing impurity types while existing inspection systems are not able to classify detected particles.

This Special Issue draws six papers from the Thirteenth IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). SSRR is an international forum for furthering the study of key issues underpinning the research of safety, security, and rescue robotics as well as solutions necessary for the fielding of robots and sensor systems across a variety of challenging application areas. We are very pleased to have selected these subset of papers from an extremely strong technical program focused on automation themes for SSRR applications.

Mammalian sperms reorient against fluid flow in the female reproductive tract, known as rheotaxis. Compared to chemotaxis that provides short-distance guidance, rheotaxis provides long-distance guidance for a sperm to find the egg cell. However, only a low number of sperms are capable of rheotaxis and their tail behavior during reorientation is not yet known. We have developed an automated system to manipulate human sperm orientation in fluid flow and quantitatively reveal sperm behavior changes during rheotaxis. The system automatically detects multiple sperms, selects the sperm for analysis, controls fluid flow, and quantifies sperm tail behavior. Sperm head angle is used as feedback to control fluid flow and select reorienting sperms. High accuracy of head angle tracking and automated sperm selection enables the capturing of dynamic sperm turning behavior in a large sample size. Algorithms are developed to track sperm tail skeletons and quantify tail beating amplitude and asymmetry, based on which the first quantitative analysis of sperm tail behavior in rheotaxis is obtained. Experimental results reveal, for the first time, that the sperms that are capable of reorienting against fluid flow beat their tails more asymmetrically than those sperms that are unable to reorient against fluid flow while no significant difference was found in their tail beating amplitudes.

This paper reports the first robotic system for vitrification of mammalian embryos. Vitrification is a technique for preserving oocytes and embryos in clinical IVF (in vitro fertilization). The procedure involves multiple steps of stringently timed pick-and-place operation for processing an oocyte/embryo in vitrification media. In IVF clinics, vitrification is conducted manually by highly skilled embryologists. Processing one oocyte/embryo occupies the embryologist 15–20 minutes, depending on protocols chosen to implement. Due to poor reproducibility and inconsistency across operators, success rates and survival rates also vary significantly. Through collaboration with IVF clinics, we are in process to realize robotic vitrification and aim ultimately to standardize clinical vitrification from manual operation to fully automated robotic operation. Our robotic system is embedded with two contact detection methods to determine the relative Z positions of the vitrification micropipette, embryo, and vitrification straw. A 3D tracking algorithm is developed for visually servoed embryo transfer and real-time monitoring of embryo volume changes during vitrification. Excess medium is automatically removed from around the vitrified embryo on the vitrification straw to achieve a high cooling rate. Tests on mouse embryos demonstrate that the system is capable of performing vitrification with a throughput at least three times that of manual operation and achieved a high survival rate (88.9%) and development rate (93.8%).

Mechanical force is an integral part of tissue morphogenesis and patterning. We have developed an automated micro-aspiration system to investigate how mouse limb bud tissue responds to extrinsic forces in order to understand whether tissue-generated forces can be a part of the mechanism causing oriented cell behaviors observed in mouse limb bud morphogenesis. The system is capable of performing automated micropipette tracking, tissue-tip contact detection, pressure control, and prolonged application of constant pressure. A three-dimensional tissue tracking algorithm is developed based on the processing of time-lapsed confocal Z-stack images. 3D visual feedback from confocal microscopy imaging, for the first time, is used to realize 3D visual servoing to control the micropipette position to compensate for tissue movement. This enables stable force application in a biologically relevant time scale (e.g., 60 minutes) during which cell remodeling occurs. Experimental results demonstrate that micro-aspiration on mouse limb bud is capable of creating tension anisotropy which causes force-responsive cells to dynamically remodel through polarized cell division and rosette resolution.

The Kleihauer-Betke test (KBT) is a widely used method for measuring fetal-maternal hemorrhage (FMH) in maternal care. In hospitals, KBT is performed by a certified technologist to count a minimum of 2,000 fetal and maternal red blood cells (RBCs) on a blood smear. Manual counting is inherently inconsistent and subjective. This paper presents a system for automated counting and distinguishing fetal and maternal RBCs on clinical KB slides. A custom-adapted hardware platform is used for KB slide scanning and image capturing. Spatial-color pixel classification with spectral clustering is proposed to separate overlapping cells. Optimal clustering number and total cell number are obtained through maximizing cluster validity index. To accurately identify fetal RBCs from maternal RBCs, multiple features including cell size, shape, gradient and saturation difference are used in supervised learning to generate feature vectors, to tackle cell color, shape and contrast variations across clinical KB slides. The results show that the automated system is capable of completing the counting of over 60,000 cells (vs. 2,000 by technologists) within 5 minutes (vs. 15 minutes by technologists). The counting results are highly accurate and correlate strongly with those from benchmarking flow cytometry measurement.

Nanomanipulation under scanning electron microscopy (SEM) has been demonstrated as an enabling technique for the manipulation and characterization of nanomaterials. We recently developed nanomanipulation techniques for the extraction and identification of DNA contained within sub-nuclear locations of a single cell nucleus. In nanomanipulation of DNA, a key step is target identification through SEM-fluorescence correlative imaging. Existing image correlation techniques often require fiducial marks and/or manual feature selection or data training, which are unsuitable for DNA nanomanipulation. This paper presents an approach for correlating SEM-fluorescence microscopy images, proven effective in processing images taken under poor SEM imaging conditions imposed by the necessity of preserving DNA's biochemical integrity. The performance of the image correlation approach under different imaging conditions was quantitatively evaluated. Compared to manual correlation by skilled operators, the automated correlation approach demonstrated an order of magnitude higher speed. The SEM-fluorescence correlation approach enables targeted nanomanipulation of sub-cellular structures under SEM.

Most mammalian cells (e.g., cancer cells and cardiomyocytes) adhere to a culturing surface. Compared to robotic injection of suspended cells (e.g., embryos and oocytes), fewer attempts were made to automate the injection of adherent cells due to their smaller size, highly irregular morphology, small thickness (a few micrometers thick), and large variations in thickness across cells. This paper presents a recently developed robotic system for automated microinjection of adherent cells. The system is embedded with several new capabilities: automatically locating micropipette tips; robustly detecting the contact of micropipette tip with cell culturing surface and directly with cell membrane; and precisely compensating for accumulative positioning errors. These new capabilities make it practical to perform adherent cell microinjection truly via computer mouse clicking in front of a computer monitor, on hundreds and thousands of cells per experiment (vs. a few to tens of cells as state-of-the-art). System operation speed, success rate, and cell viability rate were quantitatively evaluated based on robotic microinjection of over 4,000 cells. This paper also reports the use of the new robotic system to perform cell-cell communication studies using large sample sizes. The gap junction function in a cardiac muscle cell line (HL-1 cells), for the first time, was quantified with the system.

Probing nanometer-sized structures to evaluate the performance of integrated circuits (IC) for design verification and manufacturing quality monitoring demands precision nanomanipulation technologies. To minimize electron-induced damage and improve measurement accuracy, scanning electron microscopy (SEM) imaging parameters must be cautiously chosen to ensure low electron energy and dosage. This results in significant image noise and drift. This paper presents automated nanoprobing with a nanomanipulation system inside a standard SEM. We achieved SEM image denoising and drift compensation in real time. This capability is necessary for achieving robust visual tracking and servo control of nanomanipulators for probing nanostructures in automated operation. This capability also proves highly useful to conventional manual operation by rendering real-time SEM images that have little noise and drift. The automated system probed nanostructures on an SEM metrology chip as surrogates of electronic features on IC chips. Success rates in visual tracking and Z-contact detection under various imaging conditions were quantitatively discussed. The experimental results demonstrate the system's capability for automated probing of nanostructures under IC-chip-probing relevant electron microscope imaging conditions.

In robotic micromanipulation, end-effector tips must be first located under microscopy imaging before manipulation is performed. The tip of micromanipulation tools is typically a few micrometers in size and highly delicate. In all existing micromanipulation systems, the process of locating the end-effector tip is conducted by a skilled operator, and the automation of this task has not been attempted. This paper presents a technique to automatically locate end-effector tips. The technique consists of programmed sweeping patterns, motion history image end-effector detection, active contour to estimate end-effector positions, autofocusing and quad-tree search to locate an end-effector tip, and, finally, visual servoing to position the tip to the center of the field of view. Two types of micromanipulation tools (micropipette that represents single-ended tools and microgripper that represents multiended tools) were used in experiments for testing. Quantitative results are reported in the speed and success rate of the autolocating technique, based on over 500 trials. Furthermore, the effect of factors such as imaging mode and image processing parameter selections was also quantitatively discussed. Guidelines are provided for the implementation of the technique in order to achieve high efficiency and success rates.

Nanomanipulation inside electron microscopes enables a multitude of precision applications. The semiconductor industry employs this capability to probe sub-micrometer-sized features to evaluate the performance of integrated circuits (IC) for design/quality monitoring. In electron microscopy imaging, the use of low accelerating voltages and high magnifications, as required for IC nanoprobing tasks, results in significant image noise and drift. This paper presents automated nanoprobing with a nanomanipulation system inside a standard scanning electron microscope (SEM). We achieved SEM image denoising and drift compensation in real time. This capability is necessary for achieving robust visual tracking and servo control of nanoprobes for probing nanostructures in automated operation. This capability also proves highly useful to conventional manual operation by rendering real-time SEM images that have little noise and drift. The automated system probed nanostructures on an SEM metrology chip as surrogates of electronic features on IC chips. Success rates in visual tracking and Z-contact detection under various imaging conditions were quantitatively discussed. The experimental results demonstrate the system's capability for automated probing of nanostructures under IC-chip-probing relevant EM imaging conditions.

Locating end-effector tips is a prerequisite step in micromanipulation. The tip of micromanipulation tools is typically a few micrometers in size and highly delicate. In all existing automated micromanipulation systems, the process of locating the end-effector tip is conducted by a skilled operator, and the automation of this task has not been attempted. This paper presents a technique for automatically locating end-effector tips. The technique consists of programmed sweeping patterns, MHI (motion history image) end-effector detection, active contour for estimating end-effector positions, autofocusing and quad-tree search for locating end-effector tip, and finally visual servoing to position the tip to the center of the field of view. Two types of micromanipulation tools (micropipette representing single-ended tools and microgripper representing multi-ended tools) were used in experiments for testing. Quantitative results were reported in the speed and success rate of the auto-locating technique, based on over 500 trials. Furthermore, the effect of factors such as imaging mode and image processing parameter selections was also quantitatively discussed. Guidelines are provided for the implementation of the technique in order to achieve high efficiency and success rates.

Pick-place of single nanowires inside scanning electron microscopes (SEM) is useful for prototyping functional devices and characterizing nanowires's properties. Nanowire pick-place has been typically performed via teleoperation, which is time-consuming and highly skill-dependent. This paper presents an automated approach to the pick-place of single nanowires. Through SEM visual detection and vision-based motion control, the system automatically transferred individual silicon nanowires from their growth substrate to a microelectromechanical systems (MEMS) device that characterized the nanowires's electromechanical properties. The performance of the nanorobotic pick-up and placement procedures was experimentally quantified.

Pick-place of single nanowires inside scanning electron microscopes (SEM) is useful for prototyping functional devices and characterizing nanowires' properties. Nanowire pick-place has been typically performed via teleoperation, which is time-consuming and highly skill-dependent. This paper presents a robotic system capable of automated pickplace of single nanowires. Through SEM visual detection and vision-based motion control, the system transferred individual silicon nanowires from their growth substrate to a microelectromechanical systems (MEMS) device that characterized the nanowires' electromechanical properties. The performance of the nanorobotic pick-up and placement procedures was quantified by experiments. The system demonstrated automated nanowire pick-up and placement with high reliability.

Manipulating single cells with a micropipette is the oldest, yet still a widely used technique. This paper discusses the positioning of a single cell to a target position inside the micropipette after the cell is aspirated into the micropipette. Due to the small volume of a single cell (pico-liter) and nonlinear dynamics involved, this task has high skill requirements and is labor intensive in manual operation that is solely based on trial and error and has high failure rates. We present automated techniques in this paper for achieving this task. Computer vision algorithm was developed to track a single cell inside a micropipette for automated single-cell positioning. A closed-loop robust controller integrating the dynamics of cell motion was designed to accurately and efficiently position the cell to a target position inside the micropipette. The system achieved high success rates of 97% for cell tracking (n=100) and demonstrated its capability of accurately positioning a cell inside the micropipette within 8 seconds (vs. 25 seconds by highly skilled operators).

Nanowire field-effect transistors (nano-FETs) are nano devices capable of highly sensitive, label-free sensing of molecules. However, significant variations in sensitivity across devices can result from poor control over device parameters, such as nanowire diameter and the number of electrode-bridging nanowires. This paper presents a fabrication approach that uses wafer-scale nanowire contact printing for throughput and uses automated nanomanipulation for precision control of nanowire number and diameter. The process requires only one photolithography mask. Using nanowire contact printing and post processing (i.e., nanomanipulation inside scanning electron microscope), we are able to produce devices all with a single nanowire and similar diameters at a speed of ∼1 min/device with a success rate of 95% (n=500). This technology represents a seamless integration of wafer-scale microfabrication and automated nanorobotic manipulation for producing nano-FET sensors with consistent response across devices.

This paper is the first report of robotic ICSI (intracytoplasmic sperm injection). ICSI is a clinical procedure performed worldwide in fertility clinics, requiring pick-up of a single sperm and insert it into oocyte (i.e., an egg cell). Since its invention 20 years ago, ICSI has been conducted manually by a handful of highly skilled embryologists; however, success rates vary significantly among clinics due to poor reproducibility and inconsistency across operators. We leverage our work in robotic cell injection to realize robotic ICSI and aim ultimately, to standardize how clinical ICSI is performed. This paper presents some of the technical aspects of our robotic ICSI system, including a cell holding device and motion control and computer vision algorithms. The system performs visual tracking of single sperm, robotic immobilization of sperm, aspiration of sperm with pico-liter volume, and insertion of sperm into an oocyte with a high degree of reproducibility. The system requires minimal human involvement (requiring only a few computer mouse clicking), and is human operator skill independent. Using the hamster oocyte-human sperm model in preliminary trials, the robotic system demonstrated a high success rate of 90.0% and survival rate of 90.7% (n=120).

Biological experiments and drug screen require the transfer of individual zebrafish embryos into standard multi-well microplates. Manually pipetting embryos into wells is tedious and time consuming. This paper reports a prototype cooperative robotic system capable of transferring zebrafish embryos in parallel and depositing a single embryo per well in a 96-well microplate. A cell holding device was developed to trap multiple embryos in a regular pattern. The cell holding device and a microplate are positioned and aligned along multiple axes by the system. Embryo release strategies were systematically studied and compared. Experiments demonstrated that out of the 1,056 zebrafish embryos used in experiments (i.e., 44 times parallel transfer into 11 96-well plates), 996 wells were successfully filled with one and only one zebrafish embryo, representing a success rate of 94.3%. Further experiments confirmed that the transferred embryos were able to develop into zebrafish with 100% survival rate.

This paper presents a nanomanipulation system for operation inside scanning electron microscopes (SEM). The system is small in size, capable of being mounted onto and demounted from an SEM through the specimen exchange chamber without breaking the high vacuum of the SEM. This advance eliminates frequent opening of the high-vacuum chamber, thus, incurs less contamination to the SEM, avoids lengthy pumping, and significantly eases the exchange of end-effectors (e.g., nano probes and grippers). The system consists of two independent 3-DOF Cartesian nanomanipulators based on piezo motors and piezo actuators. High-resolution optical encoders are integrated into the nanomanipulators to provide position feedback for closed-loop control. A look-then-move control system and a contact detection algorithm are implemented for horizontal and vertical nanopositioning. The system design, system characterization details, and system performance are described.

This paper reports a prototype cooperative robotic system capable of transferring zebrafish embryos in parallel and depositing a single embryo per well in a 96-well microplate. A cell holding device was developed to trap multiple embryos in a regular pattern. The cell holding device and a microplate were positioned and aligned along multiple axes by the system. Embryo release strategies were systematically studied and compared. Experiments demonstrated that out of the 1056 zebrafish embryos used in experiments (i.e., 44 times parallel transfer into 11 96-well plates), 996 wells were successfully filled with one and only one zebrafish embryo, representing a success rate of 94.3%. Further experiments confirmed that the transferred embryos were able to develop into zebrafish with 100% survival rate.

Many microfabricated devices have been developed to quantify cellular response to a multitude of stimuli at a single-cell level in a high throughput manner. These single-cell studies require cells to be individually positioned at defined locations on a microdevice. This paper presents a micromanipulation system for automated pick-place of single cells. Integrating computer vision and motion control algorithms, the system visually tracks a cell in real time and controls multiple motion devices coordinately. Via fine manipulation of picoliter fluids and pressure of a few Pascals, the system accurately picks up a single cell, transfers the cell, and deposits it at a target location at a speed of 15-30 sec/cell. The micromanipulation system has the advantages of non-invasiveness, high specificity, and high precision. It is suitable to pick-place both non-labeled and labeled cells and applicable to standard cell culture substrates and microdevices with an open top.

This paper presents a robotic system that is capable of both picking up and releasing microobjects with high accuracy, reliability, and speed. Due to force-scaling laws, large adhesion forces at the microscale make rapid, accurate release of microobjects a long-standing challenge in micromanipulation, thus representing a hurdle toward automated robotic pick-and-place of micrometer-sized objects. The system employs a novel microelectromechanical systems (MEMS) microgripper with a controllable plunging structure to impact a microobject that gains sufficient momentum to overcome adhesion forces. The performance was experimentally quantified through the manipulation of 7.5-10.9 ¿m borosilicate glass spheres in an ambient environment. Experimental results demonstrate that the system, for the first time, achieves a 100% success rate in release (which is based on 700 trials) and a release accuracy of 0.45 ± 0.24 ¿m. High-speed, automated microrobotic pick-and-place was realized by visually recognizing the microgripper and microspheres, by visually detecting the contact of the microgripper with the substrate, and by vision-based control. Example patterns were constructed through automated microrobotic pick-and-place of microspheres, achieving a speed of 6 s/sphere, which is an order of magnitude faster than the highest speed that has been reported in the literature.

Due to force scaling laws, rapid, accurate release of micro objects has been a long-standing challenge for microrobotic manipulation. This paper presents an active release technique that for the first time, achieves 100% repeatability and a release accuracy of 0.70plusmn0.46mum, experimentally quantified through the manipulation of 10mum glass spheres under an optical microscope. Using a new MEMS (microelectromechanical systems) microgripper, this technique employs a controllable plunging mechanism for the micro object to gain sufficient momentum to overcome adhesion forces. Experimental results also confirmed that this technique is not substrate dependent. Theoretical analyses were conducted to understand the release principle. Based on this preliminary study, the technique may also prove to be an effective solution to active release of sub-micron objects in robotic pick-place.

This paper presents the use of a monolithic, force-feedback MEMS (microelectomechanical systems) microgripper for characterizing both elastic and viscoelastic properties of highly deformable hydrogel microcapsules (15–25µm) at wet state during micromanipulation. The single-chip microgripper integrates an electrothermal microactuator and two capacitive force sensors, one for contact detection (force resolution: 38.5nN) and the other for gripping force measurements (force resolution: 19.9nN). Through nanoNewton force measurements, closed-loop force control, and visual tracking, the system quantified Young's modulus values and viscoelastic parameters of alginate microcapsules, demonstrating an easy-to-operate, accurate compression testing technique for characterizing soft, micrometer-sized biomaterials.

Taking a different architecture than manual operation and existing microrobotic systems, this paper presents the first automated system that employs novel microfabricated cell holding devices and vision-position based control of multiple motion control devices to achieve easy sample immobilization, rapid cell orientation, and fast injection of mouse embryos. The system requires minimal human involvement through a maximum of three computer mouse clicking per mouse embryo, is human operator skill independent, and is immune from fatigue. While no robotic systems have provided performance close to manual operation, according to the preliminary experimental results (n = 90) from this study, this microrobotic system demonstrated: (i) an injection speed of 9 embryos/min vs. 2 embryos/min in typical manual operation, (ii) a success rate of 98.9%, a higher rate than the best success rate (90%) achieved by proficient injection technicians with over 10 years' experience, and (iii) a high survival rate of 82.1%, a rate comparable with the best survival rate (~80%) achieved by proficient injection technicians. Further improvement of the automated system will change the way of how mouse embryos are injected and promise its practical use in biology laboratories and mouse facilities.

This paper presents an automated cell contour visual measurement technique and a data synchronization mechanism for real-time, high-accuracy mechanical characterization of individual cells with micropipette aspiration. A computer vision tracking algorithm is developed for automatically measuring cell deformation parameters in real time (30 Hz) with a resolution down to 0.21 pixel, significantly enhancing the accuracy and efficiency of micropipette aspiration. To achieve a high characterization accuracy, the cell deformations and applied pressure changes are precisely synchronized using a data synchronization mechanism. Experimental results on both solid-like cells (interstitial cells) and liquid-like cells (neutrophils) quantitatively demonstrate that the visual tracking algorithm is capable of significantly increasing the efficiency and accuracy of micropipette aspiration. Among several characterized mechanical parameters, the viscoelastic properties of porcine aortic valve interstitial cells were, for the first time, quantified in this study.

The ability of efficiently delivering soluable/insoluable drug compounds or biomolecules into individual biological cells and quantifying their cellular responses is important for genetics, proteomics, and drug discovery. This paper presents a fully automated system for zebrafish embryo injection, which overcomes the problems inherent in manual injection, such as human fatigue and large variations in success rates due to poor reproducibility. Based on ldquolooking-then-movingrdquo control, the microrobotic system performs injection at a speed of 15 zebrafish embryos (chorion unremoved) per minute. Besides a high injection speed that compares favorably with that of a highly proficient injection technician, a vacuum-based embryo holding device enables fast immobilization of a large number of zebrafish embryos, shortening the embryo patterning process from minutes to seconds. The recognition of embryo structures from image processing identifies a desired destination inside the embryo for material deposition, together with precise motion control resulting in a success rate of 100%. Carefully tuning suction pressure levels as well as injection and retraction speeds produced a high survival rate of 98%. The quantitative performance evaluation of the automated system was based on the continuous injection of 250 zebrafish embryos. The technologies can be extended to other biological injection applications such as the injection of mouse embryos,Drosophilaembryos, andC.elegansto enable high-throughput biological and pharmaceutical research.

This paper presents the first demonstration of force-controlled micrograsping at the microNewton force level. The system manipulates highly deformable biomaterials (hydrogel microcapsules and biological cells) in an aqueous environment using a MEMS-based microgripper with integrated force feedback along two axes. The microgripper integrates an electrothermal V-beam microactuator and two capacitive force sensors, one for contact detection (force resolution: 38.5 nN) and the other for gripping force measurements (force resolution: 19.9 nN). The MEMS-based microgripper and the force control system experimentally demonstrate the capability of rapid contact detection and reliable force-controlled micrograsping to accommodate variations in sizes and mechanical properties of objects with a high reproducibility. Cell viability testing validated that the temperature at gripping arm tips does not exceed 50degC.

Injection of foreign materials (e.g., DNA, RNAi, sperm, protein, and drug compounds) into individual cells has significant implications in genetics, transgenics, assisted reproduction, and drug discovery. This video presents a microrobotic system for fully automated zebraflsh embryo injection, which overcomes the problems inherent in manual operation, such as human fatigue and large variations in success rates due to poor reproducibility. Based on computer vision and motion control, the automated microrobotic system is capable of immobilizing a large number of zebraflsh embryos into a regular pattern within seconds and injecting 15 embryos (chorion unremoved) per minute with a success rate, survival rate, and phenotypic rate all close to 100%.

This paper presents a semi-automated microrobotic system for adherent cell injection. Different from embryos/oocytes that have a spherical shape and regular morphology, adherent cells are flat with a thickness of a few micrometers and are highly irregular in morphology. Based on computer vision microscopy and motion control, the system coordinately controls a three-degrees-of-freedom microrobot and a precision XY stage. The microrobotic system demonstrates an injection speed of 25 endothelial cells per minute with a survival rate of 96% and a success rate of 82% (n=1012). The system has a high degree of performance consistency. It is immune to operator proficiency variations and from human fatigue, requiring a human operator to select injection destinations through computer mouse clicking as the only operator intervention. The microrobotic adherent cell injection system makes the injection of thousands of adherent cells practical and will enable our testing of intracellular behavior of semiconductive quantum dots (QDs).

This paper presents a micropipette aspiration system and a cell contour visual tracking algorithm for realtime, high-accuracy mechanical characterization of individual cells. The computer vision tracking algorithm measures cell deformation parameters in real time (30Hz) with a resolution down to 0.21 pixel, significantly enhancing the accuracy and efficiency of the micropipette aspiration technique. Representing another advantage over manual measurements in terms of characterization accuracies, the micropipette aspiration system features precise synchronization between cell deformations and applied pressure changes. Experimental results on both solid-like cells (interstitial cells) and liquid-like cells (neutrophils) demonstrate the effectiveness of the system and the visual tracking algorithm. Among several characterized mechanical parameters, the viscoelastic properties of porcine aortic valve interstitial cells were, for the first time, quantified in this study.

This paper presents a new design of a two-axis MEMS (microelectromechanical systems) capacitive force sensor with strict linearity and a new sensor calibration method for micro-sensors. Precise calibration of multi-axis micro force sensors is difficult for several reasons, including the need to apply many known force vectors at precise orientations at the micro force scale, and the risk of damaging the small, fragile MEMS device. In this paper the shape from motion method is introduced for micro force sensors resulting in a rapid and effective calibration technique. Structural-electrostatic coupled field simulations are conducted in order to optimize the sensor design, which is calibrated with the shape from motion method as well as the least squares method for comparison purposes. Calibration results demonstrate that the shape from motion method is an effective, practical, and accurate method for calibrating multiaxis micro force sensors

This work reports on a system consisting of a MEMS (microelectromechanical system) gripper and an ultrasonic manipulator. The gripper is electrostatically actuated and includes an integrated force sensor measuring the gripping force. The device is monolithically fabricated using a silicon-on-insulator (SOI) fabrication process. The resolution of the force sensor is in the sub-micronewton range and, therefore, provides feedback of the forces that dominate the micromanipulation processes. A MEMS ultrasonic device is described which aligns small objects such as biological cells prior to manipulation with the gripper. The concept is demonstrated with polymer spheres, glass spheres and Hela cancer cells, thus providing a useful tool in micro-robotics and biological research

Manipulation and characterization of individual biological cells require cellular forces be precisely measured in real time. This paper presents a computer vision-based cellular force measurement platform that allows for the use a single vision sensor to simultaneously obtain two forms of feedback (i.e., vision and force). A novel silicone elastomer-based cell holding device and a sub-pixel visual tracking algorithm are developed. Deflections of elastic, low-stiffness structures are visually tracked, and material deflections are subsequently transformed into cellular forces. Experimental results demonstrate that the current vision-based force sensing system is capable of performing robust cellular force measurements at a full 30 Hz with a 3.7 muN resolution. Importantly, the vision-based cellular force sensing framework established in this study is not scale or cell line dependent. The device design, visual tracking algorithm, and experimental technique form a powerful framework that permits visually resolving cellular forces in real time with a picoNewton (26 pN) resolution for applications in single cell manipulation and characterization

Linear encoders for nanoscale position sensing based on single carbon nanotube (CNT) arrays are presented. Vertically aligned single multi-walled carbon nanotubes (MWNTs) are realized using a combination of e-beam lithography and plasma-enhanced chemical vapour deposition (PECVD) growth. Electron beam lithography is used to define 50-150 nm nickel catalyst dots at precise locations on a silicon chip. Precise control of the position, density and alignment of the tubes has been achieved. Aligned nanotube arrays with spacing varying from 250 nm to 25 /spl mu/m are realized. Field emission properties of the array are investigated inside a scanning electron microscope (SEM) equipped with a 3-DOF nanorobotic manipulator with nanometer resolution functioning as a scanning anode. With this scanning anode and the single MWNT array, a nano encoder is investigated experimentally. Vertical position is detected by the change in emission current, whereas the horizontal position of the scanning anode is sensed from the emission distribution. A resolution of 98.3 nm in the vertical direction and 38.0 nm (best: 12.9 nm) in the lateral direction has been achieved.

Autofocusing is a fundamental technology for automated biological and biomedical analyses and is indispensable for routine use of microscopes on a large scale. This paper presents a comprehensive comparison study of 18 focus algorithms in which a total of 139,000 microscope images are analyzed. Six samples were used with three observation methods (bright field, phase contrast, an d differential interference contrast (DIC)) under two magnifications (100/spl times/ and 400/spl times/). A ranking methodology is proposed, based on which the 18 focus algorithms are ranked. Image pre-processing is also conducted to extensively reveal the performance and robustness of the focus algorithms. The presented guidelines allow for the selection of the optimal focus algorithm for different microscopy applications.

This paper presents a deformable object tracking algorithm based on the boundary element method (BEM). BEM differs from the finite element method (FEM) in that only the boundary of the object needs to be meshed for BEM. FEM requires that the interior of the object is meshed in addition to its boundary. This feature of BEM makes it attractive for computer vision problems. We present a deformable template that uses BEM to model deformations. This deformable template is registered to an image using an energy minimization approach. The BEM tracking algorithm presented in this paper constraints the tracking results to satisfy the condition of static equilibrium. This increases the robustness of the tracking results and enhances the usefulness of the forces obtained from the tracking procedure. We demonstrate the tracking performance of this algorithm for objects with linear and non-linear elastic properties. In addition, the results of tracking the deformations of a cell are presented.

This paper presents a microrobotic system integrating microscope vision and microforce feedback for characterizing biomembrane mechanical properties. Robust visual tracking of deformable biomembrane contour using physics-based models is described. A multi-axis MEMS-based force sensor is used to determine applied forces on biomembranes and develop a novel biomembrane mechanical model. By visually extracting geometry changes on a biomembrane during loading, geometry changes can be used to estimate applied forces using the biomembrane mechanical model and the determined elastic modulus. Forces on a biomembrane can be visually observed and controlled, thus creating a framework for vision and force assimilated cell manipulation. The experimental results quantitatively describe mouse zona pellucida (ZP) stiffness increase during ZP hardening and provide an understanding of ZP protein structure development, i.e., an increase in the number of cross links of protein ZP1 between ZP2-ZP3 units that is conjectured to be responsible for ZP stiffness increase. Furthermore, the system, technique, and model presented in this paper can be applied to investigating mechanical properties of other biomembranes and other cell types, which has the potential to facilitate many biological studies, such as cell injury and recovery where biomembrane mechanical property changes need to be monitored.

This paper presents design, fabrication, and calibration results of MEMS-based two-axis capacitive force sensors capable of resolving forces up to 490/spl mu/N with a resolution of 0.01 /spl mu/N in x, and up to 900 /spl mu/N with a resolution of 0.24 /spl mu/N in y in the passive mode. Electrostatic microactuators are integrated to enable the force sensors to operate in an actively servoed mode, in which system stiffness is modulated using force compensation, greatly increasing force measurement dynamic ranges. When the microforce sensor is actively servoed, an externally applied force is balanced by the electrostatic forces generated by the electrostatic microactuators within the sensor. The movable parts of the sensor are maintained in the equilibrium position, making the system a regulator system. The force measurement is obtained by interpreting the actuation voltages. Probes of different shapes are integrated with the sensors for micromanipulation. Other types of end-effectors, such as microgrippers and microneedles for different micromanipulation tasks can be integrated by modifying the fabrication sequence. The current application of the force sensors is for providing real-time force feedback during microrobotic cell manipulation.

This paper presents the design, fabrication, modeling, and control of a dual-axis electrostatic microactuation system. To form the 3D structure only three masks are used on silicon-on-insulator wafers using deep reactive ion etching. The bulk micromachined high aspect ratio structure produces large force output, achieving the full motion range with 10.7 V in x and 70.1 V in y. To provide position feedback for high precision manipulation, a capacitive position sensing mechanism, capable of resolving position changes up to 5 /spl mu/m with a resolution of 0.01 /spl mu/m in both x and y is integrated. A nonlinear model inversion technique is proposed for nonlinear electrostatic microactuation system identification and improving system linearity and response. The effectiveness of the technique was verified in experiments. Applications of the system include micromanipulation and microassembly.

Advances in microbiology demonstrate the need for manipulating individual biological cells, such as for cell injection which includes pronuclei injection and intracytoplasmic injection. Conventionally, cell injection has been conducted manually. In this paper, we present a microrobotic system capable of performing automatic embryo pronuclei DNA injection autonomously and semi-autonomously through a hybrid visual servoing control scheme. After injection, the DNA injected embryos were transferred into a pseudopregnant foster female mouse to reproduce transgenic mice for cancer studies. Experimental results show that the injection success rate was 100%. The system setup, hybrid control scheme and other important issues in this application, such as automatic focusing, are discussed.