Early detection and accurate diagnosis can predict the risk of malignant disease transformation, thereby increasing the probability of effective treatment. Identifying mild syndrome with small pathological regions serves as an ominous warning and is fundamental in the early diagnosis of diseases. While deep learning algorithms, particularly convolutional neural networks (CNNs), have shown promise in segmenting medical objects, analyzing small areas in medical images remains challenging. This difficulty arises due to information losses and compression defects from convolutional and pooling operations in CNNs, which become more pronounced as the network deepens, especially for small medical objects. To address these challenges, we propose a novel scale-variant attention-based network (SvANet) for accurately segmenting small-scale objects in medical images. The SvANet consists of scale-variant attention (SvAttn), cross-scale guidance, Monte Carlo attention (MCAttn), and Vision Transformer (ViT), which incorporates cross-scale features and alleviates compression artifacts for enhancing the discrimination of small medical objects. Quantitative experimental results demonstrate the superior performance of SvANet, achieving 96.12%, 96.11%, 89.79%, 84.15%, 80.25%, 73.05%, and 72.58% in mean Dice (mDice) coefficient for segmenting kidney tumors, skin lesions, hepatic tumors, polyps, surgical excision cells, retinal vasculatures, and sperms, which occupy less than 1% of the image areas in KiTS23, ISIC 2018, ATLAS, PolypGen, TissueNet, FIVES, and SpermHealth datasets, respectively.

An unbiased assessment of sperm morphology and motility is crucial for assessing fertility potential and guiding visual feedback for microrobotic manipulation. Automated analysis and selection of optimal sperm are essential for in vitro fertilization treatments, such as robotic intracytoplasmic sperm injection. However, conventional image processing methods face limitations in analyzing small sperm objects under microscopic imaging. While convolutional neural networks (CNNs) have brought promising advancements in microscopic image analysis, previous CNN methods have struggled to accurately differentiate tiny objects. These methods often require staining or fluorescence techniques to enhance visual contrast between sperm and culture medium, leading to clinical impracticality. To address these limitations, we introduce a novel sperm recognition network named the sperm feature-correlated network (SFCNet), for accurate and efficient segmentation and tracking of minute sperm objects. The SFCNet employs innovative modules, including collateral multi-scale convolution, cross-scale feature map guide, atrous spatial pyramid convolution with pooling, lateral attention, and multi-scale tracking proposal, to preserve essential sperm details despite their small size. Experimental results indicate that the SFCNet surpassed the state-of-the-art models designed for segmenting or tracking small objects, achieving up to a 28.39% higher Sorensen-Dice coefficient in segmentation and a 10.33% higher average precision in tracking. Additionally, the SFCNet excelled in sperm morphometric analysis, achieving errors below 15%. Moreover, the SFCNet also secured top-tier performance in sperm motility analysis, acquiring errors below 13% in seven sperm motility parameters.Note to Practitioners—This study is stimulated by the need to analyze the quality of motile sperms and select the optimal one for in vitro fertilization. Existing methods for detecting sperm fall short as they require a relatively high-magnification microscopic image or the usage of stain or fluorescence to increase sperm visualization, which limits the selection process or even makes the sperm clinically unavailable. To overcome these limitations, the present work proposes a new framework based on deep learning, which includes the design of extracting multi-scale sperm features. Experimental results suggest that the proposed method can perform better than existing methods in real-time analysis of multiple motile sperms’ morphology and motility at 20$\times$objective. In the future, there is a high potential for fertility specialists and healthcare workers to apply the presented framework in fertility treatment with higher accuracy and efficiency.

Feedrate scheduling is crucial for improving productivity and accuracy in robotic milling applications. However, due to the nonlinear relationship between the joint space and task space, how to plan a time-optimal feedrate profile with quick analytical solution while ensuring kinematic control up to the snap level remains quite a challenge. To solve these concerns, a snap-bounded and time-optimal feedrate scheduling model is first presented in this article. For accelerating the solving process of the nonlinear model, a synchronous linearization approach is also introduced to help relax the highly nonlinear constraints in both joint space and task space into linear ones. Thereby, the originally complex feedrate scheduling issue is converted to a finite-state convex optimization problem, and an analytical solution to the feedrate profile could be computed efficiently using a straightforward linear programming algorithm. Finally, comparative simulation and experiment are carried out to verify the effectiveness of the proposed method.

Integrating low-rank adaptation (LoRA) with federated learning (FL) has received widespread attention recently, aiming to adapt pretrained foundation models (FMs) to downstream medical tasks via privacy-preserving decentralized training. However, owing to the direct combination of LoRA and FL, current methods generally undergo two problems, i.e., aggregation deviation, and differential privacy (DP) noise amplification effect. To address these problems, we propose a novel privacy-preserving federated finetuning framework called Deviation Eliminating and Noise Regulating (DEeR). Specifically, we firstly theoretically prove that the necessary condition to eliminate aggregation deviation is guaranteeing the equivalence between LoRA parameters of clients. Based on the theoretical insight, a deviation eliminator is designed to utilize alternating minimization algorithm to iteratively optimize the zero-initialized and non-zero-initialized parameter matrices of LoRA, ensuring that aggregation deviation always be zeros during training. Furthermore, we also conduct an in-depth analysis of the noise amplification effect and find that this problem is mainly caused by the "linear relationship" between DP noise and LoRA parameters. To suppress the noise amplification effect, we propose a noise regulator that exploits two regulator factors to decouple relationship between DP and LoRA, thereby achieving robust privacy protection and excellent finetuning performance. Additionally, we perform comprehensive ablated experiments to verify the effectiveness of the deviation eliminator and noise regulator. DEeR shows better performance on public medical datasets in comparison with state-of-the-art approaches. The code is available at https://github.com/CUHK-AIM-Group/DEeR.

This article investigates the prescribed-time formation control in the task space of multirobot systems (MRSs), which is subject to the uncertain nonlinear dynamics and the position requirements. The strategy constructs a cascade system consisting of control and reference layers by connections of coupled prescribed-time control units, which separately guarantee the convergence of coordination errors, accuracy of states, and synchronization between layers. Meanwhile, the transformation from task space to joint space is built based on the pseudo-inverse of the Jacobi matrix, which avoids the singular value problem brought by computing the inverse kinematics. The adaptive control method utilizes parameter estimation to eliminate the inaccuracy problem brought by the pseudo-inverse of Jacobi matrix transformation. Then, this article provides solutions to the formation control and the formation along the trajectory control. Correspondingly, the Lyapunov analysis process proves the system’s stability and the parameter estimation’s boundedness, which confirms the sufficient conditions for realizing the prescribed-time formation control of the MRSs. Finally, this article presents examples of time-varying formation and along-trajectories formation, thereby demonstrating the effect of the controller.

Unbiased measurement of sperm morphometric and motility parameters is essential for assessing fertility potential and guiding visual feedback for microrobotic manipulation. Automated analysis of multiple sperms and selection of an optimal sperm is crucial for in vitro fertilisation treatment such as robotic intracytoplasmic sperm injection. However, conventional image processing methods have limitations in analysing small sperm objects under microscopic imaging. The emergence of convolutional neural networks (CNNs) has offered promising advancements in microscopic image analysis. However, previous CNN methods have struggled to accurately segment tiny objects, requiring staining or fluorescence techniques to enhance visual contrast between sperm and culture medium, leading to clinical impracticality. To address these limitations, we introduce a novel segmentation network named the cross-scale guidance (CSG) network for accurate and efficient segmentation of minute sperm objects. The CSG network employs innovative modules, including collateral multi-scale convolution, cross-scale feature map guide, and multi-scale feature fusion, to preserve essential sperm details despite their small size. Experimental results indicate that the CSG network surpassed the state-of-the-art models designed for small object segmentation, achieving a significant increase up to 18.62% higher mean intersection over union (mIoU). Additionally, the CSG network excelled in sperm morphometric analysis, achieving errors below 20%. Moreover, sperm motility parameters were further derived from the segmentation results for comprehensive sperm fertility analysis.

Cell counting is an essential step in a wide variety of biomedical applications, such as blood examination, semen assessment, and cancer diagnosis. However, microscopic cell counting is conventionally labor-intensive and error-prone for experts, and most of the existing automatic approaches are confined to a specific image type. To address these challenges, we propose a new interactive dual-network framework for automatic and generic cell counting. In this framework, one deep learning model (counter) is trained to regress a density map from a given microscope image. The number of cells in that image can be estimated by performing integration over the regressed density map. Another network (ground truth generator) is employed to dynamically generate suitable ground truth based on the cell samples and the dot annotations to serve as the supervision for training the counter. The interactive process to obtain the optimal model is achieved by jointly training the counter and ground truth generator iteratively. Moreover, we design a hierarchical multi-scale attention-based architecture to act as the counter in the proposed framework. This architecture is crafted to efficiently and effectively process multi-level features, enabling accurate regression of high-quality density maps. Evaluation experiments on three public cell counting datasets demonstrate the superiority of our method.Note to Practitioners—This paper is motivated by the need for advanced healthcare in the deep learning era. As a routine assessment procedure in healthcare settings, cell counting usually suffers from poor accuracy and inefficiency. We provide a solution to ameliorate the situation by developing a deep learning-based framework for automatic cell counting. After being trained in an end-to-end manner, the dual-network system is able to estimate the number of cells from the given microscopic images more accurately than existing methods. Additionally, this method is robust in various scenarios, such as calculating cell populations in suspension and cells in tissues. In the future, the presented pipeline has the potential to be implemented by biomedical practitioners who are non-expert in programming via wrapping it into a graphical user interface.

Accurate and unbiased examinations of skin lesions are critical for the early diagnosis and treatment of skin diseases. Visual features of skin lesions vary significantly because the images are collected from patients with different lesion colours and morphologies by using dissimilar imaging equipment. Recent studies have reported that ensembled convolutional neural networks (CNNs) are practical to classify the images for early diagnosis of skin disorders. However, the practical use of these ensembled CNNs is limited as these networks are heavyweight and inadequate for processing contextual information. Although lightweight networks (e.g., MobileNetV3 and EfficientNet) were developed to achieve parameter reduction for implementing deep neural networks on mobile devices, insufficient depth of feature representation restricts the performance. To address the existing limitations, we develop a new lite and effective neural network, namely HierAttn. The HierAttn applies a novel deep supervision strategy to learn the local and global features by using multi-stage and multi-branch attention mechanisms with only one training loss. The efficacy of HierAttn was evaluated by using the dermoscopy images dataset ISIC2019 and smartphone photos dataset PAD-UFES-20 (PAD2020). The experimental results show that HierAttn achieves the best accuracy and area under the curve (AUC) among the state-of-the-art lightweight networks.

Microscopic cell detection is a challenging task due to significant inter-cell occlusions in dense clusters and diverse cell morphologies. This paper introduces a novel framework designed to enhance automated cell detection. The proposed approach integrates a deep learning model that produces an inverse distance transform-based detection map from the given image, accompanied by a secondary network designed to regress a cell density map from the same input. The inverse distance transform-based map effectively highlights each cell instance in the densely populated areas, while the density map accurately estimates the total cell count in the image. Then, a custom counting-aided cell center extraction strategy leverages the cell count obtained by integrating over the density map to refine the detection process, significantly reducing false responses and thereby boosting overall accuracy. The proposed framework demonstrated superior performance with F-scores of 96.93%, 91.21%, and 92.00% on the VGG, MBM, and ADI datasets, respectively, surpassing existing state-of-the-art methods. It also achieved the lowest distance error, further validating the effectiveness of the proposed approach. These results demonstrate significant potential for automated cell analysis in biomedical applications.

Open set recognition (OSR) aims to accurately classify known diseases and recognize unseen diseases as the unknown class in medical scenarios. However, in existing OSR approaches, gathering data from distributed sites to construct large-scale centralized training datasets usually leads to high privacy and security risk, which could be alleviated elegantly via the popular cross-site training paradigm, federated learning (FL). To this end, we represent the first effort to formulate federated open set recognition (FedOSR), and meanwhile propose a novel Federated Open Set Synthesis (FedOSS) framework to address the core challenge of FedOSR: the unavailability of unknown samples for all anticipated clients during the training phase. The proposed FedOSS framework mainly leverages two modules, i.e., Discrete Unknown Sample Synthesis (DUSS) and Federated Open Space Sampling (FOSS), to generate virtual unknown samples for learning decision boundaries between known and unknown classes. Specifically, DUSS exploits inter-client knowledge inconsistency to recognize known samples near decision boundaries and then pushes them beyond decision boundaries to synthesize discrete virtual unknown samples. FOSS unites these generated unknown samples from different clients to estimate the class-conditional distributions of open data space near decision boundaries and further samples open data, thereby improving the diversity of virtual unknown samples. Additionally, we conduct comprehensive ablation experiments to verify the effectiveness of DUSS and FOSS. FedOSS shows superior performance on public medical datasets in comparison with state-of-the-art approaches. The source code is available at https://github.com/CityU-AIM-Group/FedOSS.

Magnetically actuated miniature robots are limited in their mechanical outputting capability, because the magnetic forces decrease significantly with decreasing robot size and increasing actuating distance. Hence, the output force of these robots can hardly meet the demand for specific biomedical applications (e.g., tissue penetration). This article proposes a tetherless magnetic impact needle robot (MINRob) based on a triple-magnet system with reversible and repeatable magnetic collisions to overcome this constraint on output force. The working procedure of the proposed system is divided into several states, and a mathematical model is developed to predict and optimize the force output. These force values in magnetic impact and penetration are obtained from a customized setup, indicating a ten-fold increase compared with existing miniature robots that only utilize magnetic attractive force. Eventually, the proposed MINRob is integrated with a teleoperation system, enabling remote and precise control of the robot's position and orientation. The triple-magnet system offers promising locomotion patterns and penetration capacity via the notably increased force output, showing great potential in robot-assisted tissue penetration in minimally invasive healthcare.

Previous work utilized three-dimensional (3-D) convolutional neural networks (CNNs) tomodel the spatial appearance and temporal evolution concurrently for sign language recognition (SLR) and exhibited impressive performance. However, there are still challenges for 3-D CNN-based methods. First, motion information plays a more significant role than spatial content in sign language. Therefore, it is still questionable whether to treat space and time equally and model them jointly by heavy 3-D convolutions in a unified approach. Second, because of the interference from the highly redundant information in sign videos, it is still nontrivial to effectively extract discriminative spatiotemporal features related to sign language. In this study, deep R(2+1)D was adopted for separate spatial and temporal modeling and demonstrated that decomposing 3-D convolution filters into independent spatial and temporal convolutions facilitates the optimization process in SLR. A lightweight spatial–temporal–channel attention module, including two submodules called channel–temporal attention and spatial–temporal attention, was proposed to make the network concentrate on the significant information along spatial, temporal, and channel dimensions by combining squeeze and excitation attention with self-attention. By embedding this module into R(2+1)D, superior or comparable results to the state-of-the-art methods on the CSL-500, Jester, and EgoGesture datasets were obtained, which demonstrated the effectiveness of the proposed method.

Uncertainty estimation for point cloud semantic segmentation is to quantify the confidence degree for the predicted label of points, which is essential for decision-making tasks. This paper proposes a neighborhood spatial aggregation based method, NSA-MC dropout, to achieve efficient uncertainty estimation for point cloud semantic segmentation. Unlike the traditional uncertainty estimation method MC dropout de-pending on repeated inferences, our NSA-MC dropout achieves uncertainty estimation through one-time inference. Specifically, a space-dependent method is designed to sample the model many times by performing stochastic forward pass through the model just once, and it approximates the repeated inferences based sampling process in MC dropout. Besides, a neighborhood spatial aggregation module, called NSA, aggregates neighborhood probabilistic outputs for each point and works with space-dependent sampling to establish output distribution. Finally, we propose an uncertainty-aware framework NSA-MC dropout to capture the uncertainty of prediction results efficiently. Experimental results show that our method obtains comparable performance with MC dropout. More significantly, our NSA-MC dropout has little influence on the efficiency of semantic inference. It is much faster than MC dropout, and the inference time does not establish a coupling relation with the sampling times. Our code is available at https://github.com/chaoqi7/Uncertainty_Estimation_PCSS

Accurate position feedback in a wide range is critical for medical microrobotics and robot-assisted examinations, such as colonoscopy, bronchoscopy and capsule endoscopy examination. Among the many modalities of positioning feedback, magnetic tracking is a preferable method due to the unique advantages of free line of sight, free energy storage and untethered connection. However, the field strength of the magnetic source decreases with the third power of the distance, limiting the effectiveness of position feedback at long distances. In order to maintain a consistently high tracking accuracy in a broad area, this paper presents a new dynamic tracking solution by applying a movable sensor array. In this new solution, the tracking accuracy of the magnet is first determined and optimized within a short range. When the target microrobot carrying the magnet exceeds this optimized range, the sensor array is relocated by an external robotic arm to keep the target in the effective tracking range. Moreover, we also propose a multi-point locating algorithm to minimize the varying background noise. Experimental results show that the proposed method increases the range of magnetic tracking and achieves a satisfactory level of tracking accuracy, which demonstrates significant potentials to improve the position feedback of microrobots in medical applications.

Human motion prediction is an increasingly interesting topic in computer vision and robotics. In this paper, we propose a new end-to-end feedforward network, TrajectoryCNN, to predict future poses. Compared with the most existing methods, we introduce a new trajectory space and focus on modeling motion dynamics of the input sequence with coupled spatio-temporal features, dynamic local-global features, and global temporal co-occurrence features in the new space. Specifically, the coupled spatio-temporal features describe the spatial and temporal structural information hidden in a natural human motion sequence, which can be easily mined using CNN by simultaneously covering the spatial and temporal dimensions of the sequence with the convolutional filters. The dynamic local-global features encode different correlations among joint trajectories of human motion (i.e. strong correlations among joint trajectories of one part and weak correlations among joint trajectories of different parts), which can be captured by stacking multiple residual trajectory blocks and incorporating our skeletal representation. The global temporal co-occurrence features represent different importance of different input poses to mine the motion dynamics for predicting future poses, which can be obtained automatically by learning free parameters for each pose with our TrajectoryCNN. Finally, we predict future poses with the captured motion dynamic features in a non-recursive manner. Extensive experiments show that our method achieves state-of-the-art performance on five benchmarks (e.g. Human3.6M, CMU-Mocap, 3DPW, G3D, and FNTU), which demonstrates the effectiveness of our proposed method. The code is available at https://github.com/lily2lab/TrajectoryCNN.git.

Action repetition counting is to estimate the occurrence times of the repetitive motion in one action, which is a relatively new, significant, but challenging problem. To solve this problem, we propose a new method superior to the traditional ways in two aspects, without preprocessing and applicable for arbitrary periodicity actions. Without preprocessing, the proposed model makes our scheme convenient for real applications; processing the arbitrary periodicity action makes our model more suitable for the actual circumstance. In terms of methodology, firstly, we extract action features using ConvNets and then use Principal Component Analysis algorithm to generate the intuitive periodic information from the chaotic high-dimensional features; secondly, we propose an energy-based adaptive feature mode selection scheme to adaptively select proper deep feature mode according to the background of the video; thirdly,we construct the periodic waveform of the action based on the high-energy rules by filtering the irrelevant information. Finally, we detect the peaks to obtain the times of the action repetition. Our work features two-fold: 1) We give a significant insight that features extracted by ConvNets for action recognition can well model the self-similarity periodicity of the repetitive action. 2) A high-energy based periodicity mining rule using features from ConvNets is presented, which can process arbitrary actions without preprocessing. Experimental results show that our method achieves superior or comparable performance on the three benchmark datasets, i.e. YT_Segments, QUVA, and RARV.

In many practical engineering applications, the number of actions that have been finished should be known, particularly for an untrimmed video sequence that includes an event with a series of actions, it is important to know the number of actions that have been finished. In this paper, we termed this process as visual event progress estimation (EPE). However, the research related to this problem is few in the research community. To solve this problem, a visual human action analysis-based framework, namely one-shot simultaneously action detection and identification (SADI)-EPE, is presented in this paper. The visual EPE is modeled as an online one-shot learning-based problem; sliding window and attention-based bag of key poses formulate our framework. Unlike most of the action analysis methods relying on a number of training data of some predefined classes, our method can realize SADI for any event if one sample of the event is given, which makes it feasible for practical applications. At the same time, not only SADI but also the progress estimation of the event can be realized by our algorithm. In terms of methodology, the key pose is defined by an invariant pose descriptor from skeletal data and silhouette data. Moreover, in order to extract representative and discriminative poses from one training sample, we present a new bidirectional kNN-based attention weighted key pose selection method, which can filter the unrelated actions and model different importance of various key poses. In addition, an attention-based multi-modal fusion scheme, which addresses the difficulty of high-dimensional features and few training samples, is proposed to augment the performance of our algorithm. Finally, we propose an evaluation criterion for the estimation problem. Extensive results demonstrated the efficacy of our proposed framework.

Manipulation of motile cells such as bacteria and sperm is required in both cell biology and clinical applications. For immobilizing a motile sperm, the sperm head and tail positions must be accurately tracked, interference of proximal sperms on the target sperm must be tackled, and the orientation of the sperm must be properly aligned with the manipulation tool in order not to damage the sperm head where DNA is contained. Manual operation of sperm immobilization has stringent skill requirements, and both manual operation and existing robotic sperm immobilization suffer from inconsistent success rates and incapability of manipulating sperms swimming in all directions. This paper presents a robotic system for fully automated tracking, orientation control, and immobilization of motile sperms. Algorithms were developed for robustly tracking the sperm head and estimating the sperm tail positions under interfering conditions. A new visual servo control strategy was developed to enable the robotic system to actively adjust sperm orientation for immobilizing a sperm swimming in any direction. Experimental results from robotic immobilization of 400 sperms confirmed that the robotic system achieved a consistent success rate of 94.5 %, independent of sperm velocity or swimming direction.

Magnetic micromanipulation has the advantage of untethered control, high precision, and biocompatibility and has recently undergone great advances. The magnetic micromanipulation task to tackle in this paper is to three dimensionally navigate a 5-μm magnetic bead inside a mouse embryo and accurately apply forces to intraembryonic structures to perform mechanical measurements at multiple locations. Existing technologies are not able to achieve these navigation and measurement goals because of poor magnetic force scaling and/or lacking the capability of applying an accurately controlled force. This paper reports a three-dimensional magnetic tweezer system that enables, for the first time, intraembryonic magnetic navigation and force application. A single magnetic bead was introduced into a mouse embryo via robotic microinjection. The magnetic tweezer system accurately controlled the position of the magnetic bead via visually servoed magnetic control. By moving the magnetic bead with known forces inside the embryo, cytoplasm viscosity was measured, which is eight times the viscosity of water. For performing mechanical measurements on the cellular structures inside the mouse embryo, the system should be capable of applying forces up to 120 pN with a resolution of 4 pN. The results revealed that the middle region is significantly more deformable than the side regions of the inner cell mass.

Magnetic micromanipulation has the advantage of untethered control, high precision, and biocompatibility and has recently undergone great advances. The magnetic micromanipulation task to tackle in this work is to three-dimensionally navigate a 5-micrometer magnetic bead inside a mouse embryo and perform mechanical measurements at multiple locations. Existing technologies are not able to achieve these navigation and measurement goals because of poor magnetic force scaling and/or lacking the capability of applying an accurately controlled force. This paper reports a robotic magnetic tweezer system that enables, for the first time, intra- embryonic magnetic navigation and force application. A single magnetic bead was introduced into a mouse embryo via robotic microinjection. The robotic magnetic tweezer system accurately controls the position of the magnetic bead via visually servoed magnetic control. The system is also capable of applying forces up to 120 pN with a resolution of 1.78 pN for performing mechanical measurements on the cellular structures inside the mouse embryo, revealing that the middle region is more deformable than the side regions of the inner cell mass.

Ampoule injection is a routinely used treatment in hospitals due to its rapid effect after intravenous injection. During manufacturing, tiny foreign particles can be present in the ampoule injection. Therefore, strict inspection must be performed before ampoule injections can be sold for hospital use. In the quality control inspection process, most ampoule enterprises still rely on manual inspection which suffers from inherent inconsistency and unreliability. This paper reports an automated system for inspecting foreign particles within ampoule injections. A custom-designed hardware platform is applied for ampoule transportation, particle agitation, and image capturing and analysis. Constructed trajectories of moving objects within liquid are proposed for use to differentiate foreign particles from air bubbles and random noise. To accurately classify foreign particles, multiple features including particle area, mean gray value, geometric invariant moments, and wavelet packet energy spectrum are used in supervised learning to generate feature vectors. The results show that the proposed algorithm is effective in classifying foreign particles and reducing false positive rates. The automated inspection system inspects over 150 ampoule injections per minute (versus ~ 12 ampoule injections per minute by technologist) with higher accuracy and repeatability. In addition, the automated system is capable of diagnosing impurity types while existing inspection systems are not able to classify detected particles.

Mammalian sperms reorient against fluid flow in the female reproductive tract, known as rheotaxis. Compared to chemotaxis that provides short-distance guidance, rheotaxis provides long-distance guidance for a sperm to find the egg cell. However, only a low number of sperms are capable of rheotaxis and their tail behavior during reorientation is not yet known. We have developed an automated system to manipulate human sperm orientation in fluid flow and quantitatively reveal sperm behavior changes during rheotaxis. The system automatically detects multiple sperms, selects the sperm for analysis, controls fluid flow, and quantifies sperm tail behavior. Sperm head angle is used as feedback to control fluid flow and select reorienting sperms. High accuracy of head angle tracking and automated sperm selection enables the capturing of dynamic sperm turning behavior in a large sample size. Algorithms are developed to track sperm tail skeletons and quantify tail beating amplitude and asymmetry, based on which the first quantitative analysis of sperm tail behavior in rheotaxis is obtained. Experimental results reveal, for the first time, that the sperms that are capable of reorienting against fluid flow beat their tails more asymmetrically than those sperms that are unable to reorient against fluid flow while no significant difference was found in their tail beating amplitudes.

This paper reports the first robotic system for vitrification of mammalian embryos. Vitrification is a technique for preserving oocytes and embryos in clinical IVF (in vitro fertilization). The procedure involves multiple steps of stringently timed pick-and-place operation for processing an oocyte/embryo in vitrification media. In IVF clinics, vitrification is conducted manually by highly skilled embryologists. Processing one oocyte/embryo occupies the embryologist 15–20 minutes, depending on protocols chosen to implement. Due to poor reproducibility and inconsistency across operators, success rates and survival rates also vary significantly. Through collaboration with IVF clinics, we are in process to realize robotic vitrification and aim ultimately to standardize clinical vitrification from manual operation to fully automated robotic operation. Our robotic system is embedded with two contact detection methods to determine the relative Z positions of the vitrification micropipette, embryo, and vitrification straw. A 3D tracking algorithm is developed for visually servoed embryo transfer and real-time monitoring of embryo volume changes during vitrification. Excess medium is automatically removed from around the vitrified embryo on the vitrification straw to achieve a high cooling rate. Tests on mouse embryos demonstrate that the system is capable of performing vitrification with a throughput at least three times that of manual operation and achieved a high survival rate (88.9%) and development rate (93.8%).

Mechanical force is an integral part of tissue morphogenesis and patterning. We have developed an automated micro-aspiration system to investigate how mouse limb bud tissue responds to extrinsic forces in order to understand whether tissue-generated forces can be a part of the mechanism causing oriented cell behaviors observed in mouse limb bud morphogenesis. The system is capable of performing automated micropipette tracking, tissue-tip contact detection, pressure control, and prolonged application of constant pressure. A three-dimensional tissue tracking algorithm is developed based on the processing of time-lapsed confocal Z-stack images. 3D visual feedback from confocal microscopy imaging, for the first time, is used to realize 3D visual servoing to control the micropipette position to compensate for tissue movement. This enables stable force application in a biologically relevant time scale (e.g., 60 minutes) during which cell remodeling occurs. Experimental results demonstrate that micro-aspiration on mouse limb bud is capable of creating tension anisotropy which causes force-responsive cells to dynamically remodel through polarized cell division and rosette resolution.

The Kleihauer-Betke test (KBT) is a widely used method for measuring fetal-maternal hemorrhage (FMH) in maternal care. In hospitals, KBT is performed by a certified technologist to count a minimum of 2,000 fetal and maternal red blood cells (RBCs) on a blood smear. Manual counting is inherently inconsistent and subjective. This paper presents a system for automated counting and distinguishing fetal and maternal RBCs on clinical KB slides. A custom-adapted hardware platform is used for KB slide scanning and image capturing. Spatial-color pixel classification with spectral clustering is proposed to separate overlapping cells. Optimal clustering number and total cell number are obtained through maximizing cluster validity index. To accurately identify fetal RBCs from maternal RBCs, multiple features including cell size, shape, gradient and saturation difference are used in supervised learning to generate feature vectors, to tackle cell color, shape and contrast variations across clinical KB slides. The results show that the automated system is capable of completing the counting of over 60,000 cells (vs. 2,000 by technologists) within 5 minutes (vs. 15 minutes by technologists). The counting results are highly accurate and correlate strongly with those from benchmarking flow cytometry measurement.

Nanomanipulation under scanning electron microscopy (SEM) has been demonstrated as an enabling technique for the manipulation and characterization of nanomaterials. We recently developed nanomanipulation techniques for the extraction and identification of DNA contained within sub-nuclear locations of a single cell nucleus. In nanomanipulation of DNA, a key step is target identification through SEM-fluorescence correlative imaging. Existing image correlation techniques often require fiducial marks and/or manual feature selection or data training, which are unsuitable for DNA nanomanipulation. This paper presents an approach for correlating SEM-fluorescence microscopy images, proven effective in processing images taken under poor SEM imaging conditions imposed by the necessity of preserving DNA's biochemical integrity. The performance of the image correlation approach under different imaging conditions was quantitatively evaluated. Compared to manual correlation by skilled operators, the automated correlation approach demonstrated an order of magnitude higher speed. The SEM-fluorescence correlation approach enables targeted nanomanipulation of sub-cellular structures under SEM.

Most mammalian cells (e.g., cancer cells and cardiomyocytes) adhere to a culturing surface. Compared to robotic injection of suspended cells (e.g., embryos and oocytes), fewer attempts were made to automate the injection of adherent cells due to their smaller size, highly irregular morphology, small thickness (a few micrometers thick), and large variations in thickness across cells. This paper presents a recently developed robotic system for automated microinjection of adherent cells. The system is embedded with several new capabilities: automatically locating micropipette tips; robustly detecting the contact of micropipette tip with cell culturing surface and directly with cell membrane; and precisely compensating for accumulative positioning errors. These new capabilities make it practical to perform adherent cell microinjection truly via computer mouse clicking in front of a computer monitor, on hundreds and thousands of cells per experiment (vs. a few to tens of cells as state-of-the-art). System operation speed, success rate, and cell viability rate were quantitatively evaluated based on robotic microinjection of over 4,000 cells. This paper also reports the use of the new robotic system to perform cell-cell communication studies using large sample sizes. The gap junction function in a cardiac muscle cell line (HL-1 cells), for the first time, was quantified with the system.

Probing nanometer-sized structures to evaluate the performance of integrated circuits (IC) for design verification and manufacturing quality monitoring demands precision nanomanipulation technologies. To minimize electron-induced damage and improve measurement accuracy, scanning electron microscopy (SEM) imaging parameters must be cautiously chosen to ensure low electron energy and dosage. This results in significant image noise and drift. This paper presents automated nanoprobing with a nanomanipulation system inside a standard SEM. We achieved SEM image denoising and drift compensation in real time. This capability is necessary for achieving robust visual tracking and servo control of nanomanipulators for probing nanostructures in automated operation. This capability also proves highly useful to conventional manual operation by rendering real-time SEM images that have little noise and drift. The automated system probed nanostructures on an SEM metrology chip as surrogates of electronic features on IC chips. Success rates in visual tracking and Z-contact detection under various imaging conditions were quantitatively discussed. The experimental results demonstrate the system's capability for automated probing of nanostructures under IC-chip-probing relevant electron microscope imaging conditions.

In robotic micromanipulation, end-effector tips must be first located under microscopy imaging before manipulation is performed. The tip of micromanipulation tools is typically a few micrometers in size and highly delicate. In all existing micromanipulation systems, the process of locating the end-effector tip is conducted by a skilled operator, and the automation of this task has not been attempted. This paper presents a technique to automatically locate end-effector tips. The technique consists of programmed sweeping patterns, motion history image end-effector detection, active contour to estimate end-effector positions, autofocusing and quad-tree search to locate an end-effector tip, and, finally, visual servoing to position the tip to the center of the field of view. Two types of micromanipulation tools (micropipette that represents single-ended tools and microgripper that represents multiended tools) were used in experiments for testing. Quantitative results are reported in the speed and success rate of the autolocating technique, based on over 500 trials. Furthermore, the effect of factors such as imaging mode and image processing parameter selections was also quantitatively discussed. Guidelines are provided for the implementation of the technique in order to achieve high efficiency and success rates.

Nanomanipulation inside electron microscopes enables a multitude of precision applications. The semiconductor industry employs this capability to probe sub-micrometer-sized features to evaluate the performance of integrated circuits (IC) for design/quality monitoring. In electron microscopy imaging, the use of low accelerating voltages and high magnifications, as required for IC nanoprobing tasks, results in significant image noise and drift. This paper presents automated nanoprobing with a nanomanipulation system inside a standard scanning electron microscope (SEM). We achieved SEM image denoising and drift compensation in real time. This capability is necessary for achieving robust visual tracking and servo control of nanoprobes for probing nanostructures in automated operation. This capability also proves highly useful to conventional manual operation by rendering real-time SEM images that have little noise and drift. The automated system probed nanostructures on an SEM metrology chip as surrogates of electronic features on IC chips. Success rates in visual tracking and Z-contact detection under various imaging conditions were quantitatively discussed. The experimental results demonstrate the system's capability for automated probing of nanostructures under IC-chip-probing relevant EM imaging conditions.

Locating end-effector tips is a prerequisite step in micromanipulation. The tip of micromanipulation tools is typically a few micrometers in size and highly delicate. In all existing automated micromanipulation systems, the process of locating the end-effector tip is conducted by a skilled operator, and the automation of this task has not been attempted. This paper presents a technique for automatically locating end-effector tips. The technique consists of programmed sweeping patterns, MHI (motion history image) end-effector detection, active contour for estimating end-effector positions, autofocusing and quad-tree search for locating end-effector tip, and finally visual servoing to position the tip to the center of the field of view. Two types of micromanipulation tools (micropipette representing single-ended tools and microgripper representing multi-ended tools) were used in experiments for testing. Quantitative results were reported in the speed and success rate of the auto-locating technique, based on over 500 trials. Furthermore, the effect of factors such as imaging mode and image processing parameter selections was also quantitatively discussed. Guidelines are provided for the implementation of the technique in order to achieve high efficiency and success rates.