Segmenting individual trophectoderm (TE) cells is essential for developing quantitative metrics to assess the developmental potential of human blastocysts. The elongated shape and circular arrangement of TE cells lead to continuously varying orientations across the image, posing challenges for existing cell instance segmentation methods that assume uniformly oriented cells. As a result, most methods segment the TE as a single region, and the development of quantitative, cell-level metrics predictive of live birth potential remains unexplored. In this work, we propose an instance segmentation model that represents elongate, circularly arranged TE cells using elliptical distance maps, with which superior performance in both segmentation accuracy and metric extraction was achieved, compared with state-of-the-art methods. The extracted metrics, including TE cell number, the mean and standard deviation of cell length, width, area, and mean inter-cell distance, serve as effective predictors of a blastocyst’s live birth potential. When used to predict live birth, these metrics achieved a significantly higher area under the receiver operating characteristic curve (AUC = 0.693) than traditional TE morphological grades (AUC = 0.585). The source code is publicly available at https://github.com/robotVisionHang/TESeg.

Quantitative analysis of cleavage-stage embryos is a critical step for embryo evaluation in in vitro fertilization (IVF) treatment. Deep learning-based instance segmentation methods have emerged as a promising solution. However, existing methods, typically designed for opaque instances in nature scenes, face challenges when applied to segmenting overlapping cells in cleavage-stage embryos. The semi-transparency of the cells in cleavage-stage embryos leads to vague boundaries and complex overlaps between cells, making it difficult for existing models to segment these cells without merging or missing cells. In this work, we propose a novel instance segmentation network for segmenting overlapping, semi-transparent cleavage-stage embryo cells. Specifically, the network introduces two specialized branches that learn complementary representations: an overlap branch for addressing inter-instance overlaps and a boundary branch for dealing with intra-instance shapes. Then, an attention-based feature fusion module is designed to integrate learned overlap and boundary features to enhance the instance representation for the segmentation branch. Finally, an overlap-aware post-processing module leverages the outputs from the overlap and boundary branches to adaptively adjust suppression thresholds, ensuring the preservation of distinct yet highly overlapping cells. Experiments on two cleavage-stage embryo datasets demonstrate that our method outperformed state-of-the-art methods in both segmentation accuracy and recall, highlighting its potential for quantitative embryo evaluation in clinical IVF.

Quantifying sperm flagellar beating behavior (e.g., beating amplitude, frequency, and wavelength) plays a crucial role in biological research, clinical diagnostics, and the design of sperm-inspired microrobots. However, existing computational methods struggle to accurately and efficiently analyze the highly dynamic, complex, and fine structures of sperm flagella, especially when portions of the flagellum become invisible due to three-dimensional out-of-focus beating. This paper proposes an automated high-throughput tool for quantitative analysis of sperm flagellar beating. The core innovation is continuous convolution (CConv), which adaptively captures the irregular, time-varying patterns of sperm flagella while ensuring continuity in segmentation outputs, even in the presence of locally invisible regions caused by out-of-focus motion. CConv can be integrated into various neural network architectures as a plug-and-play module. Extensive experiments demonstrate that integrating CConv consistently improves the accuracy and continuity of flagella segmentation across different networks. Furthermore, utilizing a curvature-based approach, we quantified key flagellar beating parameters, including length, amplitude, frequency, and wavelength. Applying the high-throughput tool on 1200 sperm revealed that sperm from fertile donors had significantly higher flagellar beating frequency than sperm from infertile patients. The proposed automated tool unlocks high-throughput, quantitative analysis of sperm flagellar beating, showing the potential for applications in reproductive biology and engineering research. The codes and datasets will be released at https://github.com/Goldfish-Yu/CConv.

Video object detection (VOD) of motile cells (e.g., bacteria and sperm) under microscopy is challenging due to motion blur, sporadic out-of-focus, and pose variations. Compared with VOD in generic scenes, the lower contrast and smaller color space of microscopy imaging further introduce feature overlap between the foreground objects and the background objects (e.g., impurity cells and contaminants). Transformer-based methods have achieved great success in the VOD of generic scenes by utilizing object queries to model the inner-frame objects and the inter-frame objects. However, the appearance overlap problem in microscopy video frames significantly compromises the inter-frame query aggregation by introducing background features into the object query. To tackle this challenge, this paper reports a static-dynamic query-based VOD network that treats object queries of the current video frame and reference video frames differently. Specifically, a two-stage framework is implemented that first generates high-quality object queries of reference frames with a static Transformer decoder pre-trained on a still image dataset. The network is then trained on a per-frame annotated dataset using a dynamic Transformer decoder to model the object queries of the current frame. A Reference Query Relation Module is further proposed to enhance the reference queries for more effective aggregation with the current query. Experiments on clinically collected biopsied sperm datasets validated the effectiveness of the proposed method.

It remains a formidable challenge to accurately recognize motion intentions of patients thus to control hand exoskeletons according to their volition. Current methods primarily focus on recognition of limited patient’s motion intentions, with the purpose of controlling preconfigured gestures of a hand exoskeleton for grasping objects. These methods exhibit a marked shortfall when encountering scenarios that are unexpected or not designed in advance, such as non-preprogrammed hand movements and object manipulation tasks. To tackle this issue, large language model (LLM) and speech recognition technology are employed in this study to allow the patient to control a hand exoskeleton at will. In particular, two LLMs are tailored to formulate codes of either generating non-preprogrammed gestures or dealing with unencountered objects. Additionally, an incremental learning framework is proposed to enable patients to perform both predefined and non-predefined operation tasks by integrating a natural language parser with the two LLM-based learners. The natural language parser can directly control the hand exoskeleton to perform predefined operations tasks from prestored command set, while the LLM-based learners can incrementally expand the control command set so as to enhance adaptability of the hand exoskeleton to complex activities over daily use. This study is a pioneering work in the field of hand exoskeletons, which will revolutionize the way to control hand exoskeletons. Furthermore, the proposed framework can be easily generalized to any other robots by modifying the prompt of customized LLMs, which provides a new idea to achieve autonomous learning in robotics.Note to Practitioners—The motivation of this article is to tackle the challenge of intention recognition for performing activities of daily living (ADLs) by stroke patients using a multi-degree of freedom hand exoskeleton. Existing methods for intention recognition so far can only be used for several tasks that are predefined in advance, thus none of them allow patients to control the hand exoskeleton completely at will. To surpass this limitation, an LLM-enabled incremental learning framework that integrates a hand exoskeleton controller with Large Language Model (LLM) is proposed and validated in this study. The framework offers patients an intuitive interface via voice interaction and enables patients to perform not only predefined operation tasks by the hand exoskeleton controller but also non-predefined ones that can be learned from the LLM. As a result, the hand exoskeleton controller continues to learn from the LLM, therefore is gradually able to perform all tasks in daily life. This pioneering study paves a new way in building patient-controlled hand exoskeletons with autonomous intelligence that can deal with non-predefined operation tasks in unstructured environments.

Sperm morphology measurement is vital for diagnosing male infertility, which involves quantification of multiple subcellular parts for each sperm. Instance-aware part segmentation networks have been introduced to address this task by automatically identifying individual sperm and segmenting their subcellular parts. However, major limitations of state-of-the-art instance-aware part segmentation networks include: 1) they are time-consuming and computational expensive due to sequential processing and multi-stage frameworks; 2) they perform poorly for densely packed sperm that overlap or cross over one another. To overcome these challenges, this paper proposes 1) integrating instance identification and subcellular part segmentation within a single-stage framework to save inference time and memory usage; 2) dividing a sperm target into simpler components (head and tail) to improve prediction accuracy, followed by a contrastive learning-based matching method to pair the head and tail. Experimental results on our clinically collected human sperm dataset demonstrated that the proposed network not only outperformed state-of-the-art CP-Net (by 3.5% APp vol) but also achieved realtime inference (48.0 frames per second), effectively meeting the clinical requirements for automated parts segmentation of sperm. final part segmentation results. 2) Since the sperm head and tail have simpler shapes, they are detected separately to improve segmentation accuracy. A contrastive learning-based method is then designed to pair head and tail based on similarity of feature embeddings extracted from the proposed instance prediction branch. The proposed method significantly outperformed existing networks, particularly in handling densely packed sperm. The presented method has applicability to analyzing sperm and more broadly other cell types.

Automated morphological measurement of cellular and subcellular structures in live cells is important for evaluating cell functions. Due to their small size and transparent appearance, visualizing cellular and subcellular structures often requires high magnification microscopy and fluorescent staining. However, high magnification microscopy gives a limited field of view, and fluorescent staining alters cell viability and/or activity. Therefore, microscopy image enhancement methods have been developed to predict detailed intracellular structures in live cells. Existing image enhancement networks are mostly CNN-based models lacking global information or Transformer-based models lacking local information. For these purposes, a novel CNN-Transformer combined bilateral U-Net (CTBUnet) is proposed to effectively aggregate both local and global information. Experiments on the collected sperm cell enhancement dataset demonstrate the effectiveness of proposed network for both super-resolution and virtual staining prediction.

Accurate and prompt speed grasp intention recognition is crucial in online human-robot interaction (HRI). However, dynamic grasping relying on complete motion for high recognition accuracy will lead to an unavoidable delay in real-time prediction. To address this issue, we propose a Spatial Gating and Temporal Weighting Early Grasp Prediction (STEGP) method that utilizes incomplete dynamic grasping data from sliding windows to reduce the time delay for reliable robot teleoperation. The proposed method comprises a synergy-based feature extraction module, a spatial gating classification module, and a time-decay weighting fusion prediction module. The spatial-temporal mechanism with gating units effectively classifies sequential movements, achieving performance comparable to that of Transformers but being much easier to implement. Integrating a time-decay weighting frame enables reliable early prediction even with incomplete data. gMLP is chosen for the classification of hand dynamic grasping because of its high accuracy, realizing 93.83% accuracy for 33 grasping categories. The prediction tests demonstrated 85.4% accuracy, with the first 25% grasp completion across 28 subjects. Online robotic teleoperation grasp experiments achieved a 57.4% reduction in time delay and a 93.3% success rate.

Automated positioning systems play a pivotal role in micro-scale cell manipulation. In clinical intracytoplasmic sperm injection (ICSI) for infertility treatment, a motile sperm needs to be immobilized by glass micropipette tapping for subsequent surgical steps. The process requires accurate tracking of the target sperm and precise alignment between the sperm tail and the micropipette. Manual sperm immobilization suffers from inconsistent success rates, and current robotic systems developed for the task fail to comply with the standard clinical setup. Instead of using a motorized micromanipulator as in existing robotic systems, this paper presents an automated and compact three-dimensional (3-D) positioning stage for sperm immobilization that can be seamlessly integrated into standard clinical platforms. To tackle the challenge of accurately tracking the target sperm with degraded detection quality due to the complex 3-D motion of the positioning stage, a multi-stage sperm tracking scheme is designed for detection-to-tracklet association. To prevent physical contact between the sperm head and the micropipette, an adaptive tail-tapping planning strategy based on the sperm head orientation analysis is established. A visual servo controller equipped with a dynamic sperm motion observer is further employed to achieve precise positioning of the target sperm during the immobilization process. Experimental results demonstrated that the proposed system achieved a sperm tracking accuracy of 88.12%, and a sperm positioning accuracy of$2.3~\pm ~1.2~\mu $m. Further experiments revealed the system achieved a success rate of 93.5% and a time cost of 5.5 s for automated sperm immobilization. Note to Practitioners—This work presents an automated positioning system for the robotic immobilization of live human sperm in the intracytoplasmic sperm injection (ICSI) process. Conventional robotic systems developed for sperm immobilization use motorized stages which only provide two degrees of freedom (DOF) and disturb standard clinical setups with bulky sizes and additional equipment. Leveraging the advantages of piezoelectric positioners, a compact 3-D positioning system is developed to perform robotic sperm immobilization in an all-in-one manner. A sperm immobilization tracker is developed to track the target sperm during the immobilization process. The proposed multi-stage data association metric can effectively resist the noisy detection results due to occlusion and out-of-focus blur introduced by the 3-D movement of the positioning stage. Based on the analysis of the sperm head orientation, an adaptive tail-tapping planning strategy is established to avoid the risk of contacting the sperm head where DNA is contained. The developed positioning system can be easily integrated into a standard microscopy operation platform for biological cell manipulation. The proposed tracking scheme is applicable in various microscopy cell analysis scenarios where the detection results are inevitably affected by the degraded image quality.

Fish can adaptively adjust their body kinematics and swimming modes by sensing to realize optimal propulsion. However, most soft robotic fish have an unchangeable swimming mode through simple structure design, making them difficult to adapt to dynamic and complex fluid environments. Here, inspired by the multiple muscle synergy and lateral line sensing function of fish, we developed a soft robotic fish with multiple actuating units and embedded sensing elements. By collaboratively controlling the amplitude and phase of excitation from the multiple flexible actuating units, the soft robotic fish can successfully realize various swimming modes very similar to those of natural fish. Additionally, the embedded flexible sensing elements enable the robotic fish to sense the swimming state and the surrounding fluid environment in real time. The multiple actuation and embedded sensing allow the soft robotic fish to adaptively switch to an optimal swimming mode in a certain fluid environment. The multimode swimming and perception capabilities proposed in this work not only make soft robotic fish more intelligent and adaptable to complex fluid environments, but also contribute to the future implementation of autonomous control capabilities for robotic fish.

Traditional sperm morphology analysis is based on tedious manual annotation. Automated morphology analysis of a high number of sperm requires accurate segmentation of each sperm part and quantitative morphology evaluation. State-of-the-art instance-aware part segmentation networks follow a "detect-then-segment" paradigm. However, due to sperm’s slim shape, their segmentation suffers from large context loss and feature distortion due to bounding box cropping and resizing during ROI Align. Moreover, morphology measurement of sperm tail is demanding because of the long and curved shape and its uneven width. This paper presents automated techniques to measure sperm morphology parameters automatically and quantitatively. A novel attention-based instance-aware part segmentation network is designed to reconstruct lost contexts outside bounding boxes and to fix distorted features, by refining preliminary segmented masks through merging features extracted by feature pyramid network. An automated centerline-based tail morphology measurement method is also proposed, in which an outlier filtering method and endpoint detection algorithm are designed to accurately reconstruct tail endpoints. Experimental results demonstrate that the proposed network outperformed the state-of-the-art top-down RP-R-CNN by 9.2% ${\mathbf{AP}}_{vol}^p$, and the proposed automated tail morphology measurement method achieved high measurement accuracies of 95.34%,96.39%,91.20% for length, width and curvature, respectively.

Automated positioning systems play a pivotal role in micro-scale cell manipulation. In clinical intracytoplasmic sperm injection (ICSI) of in vitro fertilization (IVF) treatment, a motile sperm needs to be immobilized by glass micropipette tapping for subsequent surgical steps. The process requires accurate tracking of the target sperm and precise alignment between the sperm tail and the micropipette. Manual sperm immobilization suffers from inconsistent success rates, and current robotic systems developed for the task fail to comply with the standard clinical setup. Instead of using a motorized micromanipulator as in existing robotic systems, this paper presents an automated, compact three-dimensional positioning stage for sperm immobilization that can be seamlessly integrated into standard clinical platforms. Based on the analysis of the sperm head orientation, an adaptive tail tapping planning strategy is established to avoid the risk of touching the sperm head where DNA is contained. A visual servo controller equipped with a dynamic sperm motion observer is employed to achieve precise tracking and positioning of the target sperm three-dimensionally. Experimental results revealed the system achieved a success rate of 93.5% and a time cost of 5.5 s for automated sperm immobilization.

Instance segmentation of biological cells is important in medical image analysis for identifying and segmenting individual cells, and quantitative measurement of subcellular structures requires further cell-level subcellular part segmentation. Subcellular structure measurements are critical for cell phenotyping and quality analysis. For these purposes, instance-aware part segmentation network is first introduced to distinguish individual cells and segment subcellular structures for each detected cell. This approach is demonstrated on human sperm cells since the World Health Organization has established quantitative standards for sperm quality assessment. Specifically, a novel Cell Parsing Net (CP-Net) is proposed for accurate instance-level cell parsing. An attention-based feature fusion module is designed to alleviate contour misalignments for cells with an irregular shape by using instance masks as spatial cues instead of as strict constraints to differentiate various instances. A coarse-to-fine segmentation module is developed to effectively segment tiny subcellular structures within a cell through hierarchical segmentation from whole to part instead of directly segmenting each cell part. Moreover, a sperm parsing dataset is built including 320 annotated sperm images with five semantic subcellular part labels. Extensive experiments on the collected dataset demonstrate that the proposed CP-Net outperforms state-of-the-art instance-aware part segmentation networks.

Assist-as-needed (AAN) assistance can promote active voluntary participation in rehabilitation and motor function recovery of post-stroke patients. However, different patients have personalized damaged regions and recovery states, causing difficulties to obtain adaptive and customized assistance in robot-assisted rehabilitation. This paper presents a mirror Adaptive Assist-As-Needed (AAAN) scheme, including two modules of Multi-Sensors Fused Estimation (MSFE) and Online Incremental Mirror Adaptation (OIMA), to encourage the subjects to actively participate in rehabilitation. Specifically, the first MSFE module can obtain the needed assistance based on the functional capability of the post-stroke patients via the data fusion of biological and motional signals using Kalman Filter. The second OIMA module fine-tunes the control torques estimated by MSFE to adapt the muscle fatigue and stiffness varieties of the affected limb based on the motion and physiological reference of the mirror healthy limb. The results demonstrate that the AAAN strategy can realize the transparent mode for healthy subjects and promote post-stroke patients to rehabilitate the affected limb with active participation using EMG signals 90.5% similar to those of the mirror healthy limb. The proposed method can be expected to greatly enhance power assistance and rehabilitation outcome of post-stroke patients using exoskeletons by provoking active participation. Note to Practitioners—For robotic rehabilitation, it is crucial to provide suitable assistances that can maximize the participation of post-stroke patients, which can promote the recovery outcome of therapies. The main purpose of this work is to achieve the adaptive assist-as-needed control strategy for upper limb rehabilitation tasks in two steps. Firstly, the elbow joint torques of a post-stroke patient are estimated by data fusion of motion and electromyography (EMG) signals using Kalman Filter, which can make up for the shortcomings of the individual signals, such as poor reliability and low sensitivity. Secondly, the joint motion and EMG signals of the mirror healthy limb are used as the reference to calculate the adaptive needed assistance to rehabilitate the affected limb. The preliminary experiments with healthy and post-stroke subjects demonstrate that this approach can obtain stable motion with the natural physiological states of subjects and enhance active voluntary participation in rehabilitation. In the future study, it will be investigated how to accelerate the adaptation of new patients based on the knowledge of the learned individuals using machine learning methods, such as lifelong learning and incremental learning.

Detecting interchanges in road networks benefit many applications, such as vehicle navigation and map generalization. Traditional approaches use manually defined rules based on geometric, topological, or both properties, and thus can present challenges for structurally complex interchange. To overcome this drawback, we propose a graph-based deep learning approach for interchange detection. First, we model the road network as a graph in which the nodes represent road segments, and the edges represent their connections. The proposed approach computes the shape measures and contextual properties of individual road segments for features characterizing the associated nodes in the graph. Next, a semi-supervised approach uses these features and limited labeled interchanges to train a graph convolutional network that classifies these road segments into an interchange and non-interchange segments. Finally, an adaptive clustering approach groups the detected interchange segments into interchanges. Our experiment with the road networks of Beijing and Wuhan achieved a classification accuracy >95% at a label rate of 10%. Moreover, the interchange detection precision and recall were 79.6 and 75.7% on the Beijing dataset and 80.6 and 74.8% on the Wuhan dataset, respectively, which were 18.3–36.1 and 17.4–19.4% higher than those of the existing approaches based on characteristic node clustering.

It has been clinically proven that exoskeletons are effective self-training rehabilitation or daily living assistance devices for patients with hand dysfunctions. However, exoskeleton-assisted hand exercises with high degrees-of-freedom are considered as challenging tasks because the digit space, especially the thumb, cannot accommodate enough actuators. In this article, we report a tendon-driven soft hand exoskeleton with a hybrid configuration for thumb actuation. The soft hand exoskeleton system uses the least number of actuators to realize full degrees-of-freedom actuation for all digits. It is tested on a stroke patient with hemiplegia and a healthy subject. The experimental results show that the hand exoskeleton could assist the stroke patient to accomplish various training tasks, such as thumb encircling, grasping, pinching, releasing, and writing. It was found that digit trajectories and joint angle changes of the stroke patient were close to those of the healthy subject. Especially, the range of motion of the stroke patient shows significant improvement with the hand exoskeleton assistance compared to that without the hand exoskeleton assistance. The research in this article paves the way to develop fully actuated soft hand exoskeleton that can be eventually integrated with an electroencephalogram or electromyography for self-training rehabilitation or daily living assistance.

Three-dimensional structured light (SL) measurement of highly reflective surface is a challenge faced in industrial metrology. The high dynamic range (HDR) technique provides a solution by fusing images under multiple exposures; however, the process is highly time-consuming. This article reports a new SL-based method to measure parts with highly reflective surfaces from only a single exposure. A new quantitative metric is defined to optimally select camera exposure for capturing input single-exposure images. Different from existing image gradient or entropy-based metrics, the new metric incorporates both intensity modulation and overexposure. A skip pyramid context aggregation network (SP-CAN) is proposed to enhance the single exposure-captured images. Compared with existing image enhancement methods, SP-CAN effectively preserves detailed encoded phase information near edges and corners during enhancement. Experiments with various industrial parts demonstrated that the average time cost of the proposed method was 0.6 s, which was only one tenth of the HDR method (ten exposures), and the two methods achieved similar coverage rates (97.6% versus 98.0%) and measurement accuracy (0.040 mm versus 0.038 mm).

A 3D measurement system consisting of a 3D scanner and an industrial robot (eye-in-hand) is commonly used to scan large object under test (OUT) from multiple fieldof-views (FOVs) for complete measurement. A data stitching process is required to align multiple FOVs into a single coordinate system. Marker-free stitching assisted by robot’s accurate positioning becomes increasingly attractive since it bypasses the cumbersome traditional fiducial marker-based method. Most existing methods directly use initial Denavit-Hartenberg (DH) parameters and hand-eye calibration to calculate the transformations between multiple FOVs. Since accuracy of DH parameters deteriorates over time, such methods suffer from high stitching errors (e.g., 0.2 mm) in long-term routine industrial use. This paper reports a new robot-scanner calibration approach to realize such measurement with low data stitching errors. During long-term continuous measurement, the robot periodically moves towards a 2D standard calibration board to optimize kinematic model’s parameters to maintain a low stitching error. This capability is enabled by several techniques including virtual arm-based robot-scanner kinematic model, trajectory-based robot-world transformation calculation, nonlinear optimization. Experimental results demonstrated a low data stitching error (< 0.1 mm) similar to the cumbersome marker-based method and a lower system downtime (< 60 seconds vs. 10-15 minutes by traditional DH and hand-eye calibration).

Recent years have witnessed a growing interest in localization algorithms for wireless ad hoc networks. In most localization algorithms, increasing the density of anchor nodes is one of the main strategies to improve the localization accuracy in dense networks. In this paper, based on the number of reference nodes, we propose a distributed localization algorithm, i.e., history based multi-node collaborative localization algorithm (HMCL), which provides a potential approach for localization in sparse ad hoc wireless networks. In the proposed HMCL algorithm, we exploit a new motion model to filter the imprecise estimation values based on the historical position information of nodes, which can improve the localization accuracy and reduce the computation overhead and energy consumption. Moreover, we utilize different strategies to achieve the localization of nodes with different priorities measured by the distance information between neighbor nodes. We verify through experiment that the proposed algorithm provides better performance in terms of localization precision and energy consumption. Besides, we also analyze the effect of the number of neighbor nodes, node density and moving speed of nodes on localization precision.