The task performance of mobile manipulators can be significantly enhanced by whole-body control and optimization in complex scenarios. Due to the nonlinear properties of whole-body dynamics and parameter uncertainty, modeling accurate system dynamics is essential in addition to designing an effective control strategy. However, traditional control methods have high computational costs and fail to deal with the parameter errors caused by model linearization. To address these issues, we propose a model predictive control (MPC)-Net, a learning-based approach that facilitates rapid online optimization by combining deep learning with multiple MPCs. Firstly, we develop a parameter identification algorithm based on a deep neural model to estimate the unknown dynamics parameters. Although the control performance of the MPC approach positively correlated with the prediction horizon, a long horizon would result in additional computational costs. Thus, MPC-Net is constructed by combining multiple sub-MPC issues, and the nonlinear coefficients are obtained by using a deep neural network-based optimization framework. Furthermore, MPC-Net generates the solution by combining the outputs of multiple sub-MPC problems using the nonlinear transformation of learned coefficients. Experiments are conducted on a mobile manipulator, which demonstrates the proposed MPC-Net-based optimization control offers fast efficient computation and low tracking error performance.

Segmenting individual trophectoderm (TE) cells is essential for developing quantitative metrics to assess the developmental potential of human blastocysts. The elongated shape and circular arrangement of TE cells lead to continuously varying orientations across the image, posing challenges for existing cell instance segmentation methods that assume uniformly oriented cells. As a result, most methods segment the TE as a single region, and the development of quantitative, cell-level metrics predictive of live birth potential remains unexplored. In this work, we propose an instance segmentation model that represents elongate, circularly arranged TE cells using elliptical distance maps, with which superior performance in both segmentation accuracy and metric extraction was achieved, compared with state-of-the-art methods. The extracted metrics, including TE cell number, the mean and standard deviation of cell length, width, area, and mean inter-cell distance, serve as effective predictors of a blastocyst’s live birth potential. When used to predict live birth, these metrics achieved a significantly higher area under the receiver operating characteristic curve (AUC = 0.693) than traditional TE morphological grades (AUC = 0.585). The source code is publicly available at https://github.com/robotVisionHang/TESeg.

Reinforcement learning methods for manipulator trajectory generation often suffer from low sample efficiency and inadequate exploration. To address this issue, an efficient expert guidance mechanism with dynamic movement primitives is incorporated into reinforcement learning. In the mechanism, the expert model is built to provide suggested actions and guided rewards for the agent. It gradually improves the performance of the agent through a learning strategy of multiple stages, combined with expert demonstration and self-exploration. The proposed method enhances the performance of actor–critic-based reinforcement learning algorithms, which is verified in simulation, and further validated in a sweet pepper harvesting experiment.

Current numerical methods for solving kinematic identification (KI) and inverse kinematics (IK) are limited in accuracy, convergence rates, and robustness, necessitating further enhancement. This paper presents a modified Halley method for solving the KI and IK problems of serial robots based on the product of exponentials formula, achieving quintic convergence. Specifically, a general error model is first established based on exponential coordinates, and KI and IK are reformulated as root-finding problems. Next, the modified Halley method, which we prove to be a fifth-order method and incorporates a damping strategy, is proposed to resolve the singularity issue and enhance robustness. Subsequently, the Jacobian and Hessian matrices required for the proposed method are analytically derived based on the time differential of exponentials. Furthermore, highly simplified explicit formulas for these matrices are presented for the IK problem. Simulations on serial robots with various configurations validate the proposed method's accuracy, convergence rates, and robustness in solving KI and IK problems, as well as its advantages over the state-of-the-art. Additionally, experimental validation of KI on two physical robots further demonstrates the effectiveness of the proposed method. Our custom-written MATLAB and C++ codebases are made publicly available for download.

Point cloud clustering is a promising method for 3D surface defect diagnosis in manufacturing but requires manual clustering parameter selection, reducing usability. This paper proposes an automatic point cloud clustering method to address this issue. It employs a strategy that progresses from coarse to fine. In the coarse searching stage, a K-Nearest Neighbor (KNN) graph analysis technique is developed to recognize potential defective regions in parallel. Moving on to the fine stage of extracting detailed defects, a modified DBSCAN algorithm is proposed, in which the clustering parameters are calculated automatically from the KNN graph analysis results. Experimental results showed that the proposed method achieved cloud clustering with automatically calculated clustering parameters for surface defect diagnosis. The proposed method outperformed the traditional region growing algorithm in accuracy (0.942 vs. 0.680) and processing speed (21500 points/sec vs. 8740 points/sec) without requiring manual intervention.Note to Practitioners—This paper presents a method for diagnosing defects on automobile and flat steel surfaces. Current 3D point cloud techniques for surface defect diagnosis require manual parameter adjustments, reducing usability. This paper proposes an automatic method without manual intervention. The proposed method uses a coarse-to-fine strategy. The 3D point cloud is divided into sub-blocks to locate potential defects, and a clustering algorithm then extracts detailed defects with automatically determined parameters. We mathematically characterize changes in point density caused by surface defects and show how these features can be used for clustering parameter calculation. Experimental results demonstrate the method’s efficiency on flat as well as some curved surfaces, but it has yet to be evaluated on complex structures. Future work will aim to broaden its application to include a more extensive variety of surfaces and integrate it with robotic vision systems.

Instance segmentation of biological cells is important in medical image analysis for identifying and segmenting individual cells, and quantitative measurement of subcellular structures requires further cell-level subcellular part segmentation. Subcellular structure measurements are critical for cell phenotyping and quality analysis. For these purposes, instance-aware part segmentation network is first introduced to distinguish individual cells and segment subcellular structures for each detected cell. This approach is demonstrated on human sperm cells since the World Health Organization has established quantitative standards for sperm quality assessment. Specifically, a novel Cell Parsing Net (CP-Net) is proposed for accurate instance-level cell parsing. An attention-based feature fusion module is designed to alleviate contour misalignments for cells with an irregular shape by using instance masks as spatial cues instead of as strict constraints to differentiate various instances. A coarse-to-fine segmentation module is developed to effectively segment tiny subcellular structures within a cell through hierarchical segmentation from whole to part instead of directly segmenting each cell part. Moreover, a sperm parsing dataset is built including 320 annotated sperm images with five semantic subcellular part labels. Extensive experiments on the collected dataset demonstrate that the proposed CP-Net outperforms state-of-the-art instance-aware part segmentation networks.

Automated manipulation of deformable objects is challenging due to the object’s deformation behavior. Different from still deformable objects such as wires and cloth, biological organisms such as sperm and worms are both deformable and motile, requiring the control of both deformation and motion. This paper reports automated orientation control of live sperm, as an example of motile deformable cells. Robotic manipulation of human sperm was performed by using a glass micropipette, which is a standard clinical tool, to rotate individual motile sperm. Sperm rotation must be performed before immobilization, as required in clinical cell surgery for infertility treatment. To control tail deformation during sperm rotation, a path planner was designed based on kinematic analysis and manipulation point update. To deal with the intrinsic motion of a motile sperm, a motorized stage was controlled to compensate for sperm swimming motion, and an observer was designed to decouple sperm orientation from its wiggling motion. A sliding mode controller was designed to cope with stiffness variances along the sperm tail and among different sperm. Deep neural networks were developed for robust sperm tail detection, and Kalman filter was used to predict tail motion. Experimental results demonstrated that automated sperm manipulation achieved an orientation error of 0.8° and operation time of 6.8 s, both significantly less than those of manual operation. The designed observer was effective to reduce sperm orientation error by reducing the disturbance from sperm wiggling motion. The developed sliding mode controller outperformed the PID controller in operation time, reducing the time of oocyte exposure to the ambient environment.Note to Practitioners—This work tackled the challenge of rotating a fast-swimming and deformable sperm in clinical cell surgeries. Automated manipulation of deformable objects has wide applications in industrial and service settings such as manipulating wires and folding cloth. However, the intrinsic motion of a motile sperm and the lack of a rotational degree of freedom in standard micromanipulators pose difficulties to automated sperm manipulation. In this paper, we propose automation techniques for sperm orientation control. For sperm tail detection, deep learning was used to handle the variances of shape and length among different sperm. A path planning strategy and a controller were designed to achieve automated rotation of motile sperm, with its deformation and motion both controlled. The developed methods can be generalized to the manipulation of other deformable objects such as wires, cables and cloth. These objects exhibit significant variance of mechanical properties, and calibration is often time-consuming. The designed controller can be used to manipulate deformable objects with robustness to varied mechanical parameters. Path planning was designed by updating the manipulation point based on the object’s deformation behavior, and is suitable in manipulation where constraints are imposed such as the object’s strain.

Three-dimensional structured light (SL) measurement of highly reflective surface is a challenge faced in industrial metrology. The high dynamic range (HDR) technique provides a solution by fusing images under multiple exposures; however, the process is highly time-consuming. This article reports a new SL-based method to measure parts with highly reflective surfaces from only a single exposure. A new quantitative metric is defined to optimally select camera exposure for capturing input single-exposure images. Different from existing image gradient or entropy-based metrics, the new metric incorporates both intensity modulation and overexposure. A skip pyramid context aggregation network (SP-CAN) is proposed to enhance the single exposure-captured images. Compared with existing image enhancement methods, SP-CAN effectively preserves detailed encoded phase information near edges and corners during enhancement. Experiments with various industrial parts demonstrated that the average time cost of the proposed method was 0.6 s, which was only one tenth of the HDR method (ten exposures), and the two methods achieved similar coverage rates (97.6% versus 98.0%) and measurement accuracy (0.040 mm versus 0.038 mm).

Multirobot systems have shown great potential in dealing with complicated tasks that are impossible for a single robot to achieve. One essential problem encountered in cooperatively working of the multirobot systems is the unknown initial transformation relationships from hand to eye, base to base, and flange to tool. In this article, the problem of multicoordinates calibration for a dual-robot system is formulated to a matrix equation AXB = YCZ. A novel approach for simultaneously solving the unknowns in equation AXB = YCZ is proposed, which is composed of a closed form method based on the Kronecker product and an iterative method which converts the calculation of a nonlinear problem to an optimization problem of a strictly convex function. The closed form method is used to quickly obtain an initial estimation for the iterative method to improve the efficiency and accuracy of iteration. In addition, a series of conditions on the solvability of the problem are proposed to guide the operators to select appropriate robot attitudes during the calibration process. To show the feasibility and superiority of the proposed iterative method, two other calibration methods are chosen to be compared to the proposed method through simulation and practical experiments. The comparison results verify the superiority of the proposed method in accuracy, efficiency, and stability.

A 3D measurement system consisting of a 3D scanner and an industrial robot (eye-in-hand) is commonly used to scan large object under test (OUT) from multiple fieldof-views (FOVs) for complete measurement. A data stitching process is required to align multiple FOVs into a single coordinate system. Marker-free stitching assisted by robot’s accurate positioning becomes increasingly attractive since it bypasses the cumbersome traditional fiducial marker-based method. Most existing methods directly use initial Denavit-Hartenberg (DH) parameters and hand-eye calibration to calculate the transformations between multiple FOVs. Since accuracy of DH parameters deteriorates over time, such methods suffer from high stitching errors (e.g., 0.2 mm) in long-term routine industrial use. This paper reports a new robot-scanner calibration approach to realize such measurement with low data stitching errors. During long-term continuous measurement, the robot periodically moves towards a 2D standard calibration board to optimize kinematic model’s parameters to maintain a low stitching error. This capability is enabled by several techniques including virtual arm-based robot-scanner kinematic model, trajectory-based robot-world transformation calculation, nonlinear optimization. Experimental results demonstrated a low data stitching error (< 0.1 mm) similar to the cumbersome marker-based method and a lower system downtime (< 60 seconds vs. 10-15 minutes by traditional DH and hand-eye calibration).

ABSTRACTTo-date few research has successfully integrated big data from multiple sources to characterize urban mixed-use buildings. In this paper, we introduce a probabilistic model to integrate multi-source and geospatial big data (social network data, taxi trajectories, Points of Interest and remote sensing images) to characterize urban mixed-use buildings. The usefulness of our model is demonstrated with a case study of the Tianhe District in megacity Guangzhou, China. The model predicted building functions at 85% accuracy based on ground truth data from field surveys. We further explored the spatial patterns of the identified building functions. Most mixed-use buildings are located along major streets. Our proposed model can identify mixed-use buildings in a city; information is useful for planning evaluation and urban policymaking.

David M. Mark published his first journal article in 1970. Since then, he has written or coauthored more than 220 publications over a period of 40 years as of 28 May 2012. Based on data from Web of Science (WoS) and Google Scholar, Mark’s publications have been cited over 7410 times by researchers in more than 80 countries or regions as of 28 May 2012, when this paper was first prepared. The geographic extent of Mark’s scholarly influence is truly global. An examination of his 20 most cited articles reveals that his work in diverse areas as digital elevation models, geomorphology, geographic cognition, and ontology of the geospatial domain enjoyed a lasting impact worldwide.