Computer-aided surgical simulation is a critical component of orthognathic surgical planning, where accurately simulating face-bone shape transformations is significant. The traditional biomechanical simulation methods are limited by their computational time consumption levels, labor-intensive data processing strategies and low accuracy. Recently, deep learning-based simulation methods have been proposed to view this problem as a point-to-point transformation between skeletal and facial point clouds. However, these approaches cannot process large-scale points, have limited receptive fields that lead to noisy points, and employ complex preprocessing and postprocessing operations based on registration. These shortcomings limit the performance and widespread applicability of such methods. Therefore, we propose a Transformer-based coarse-to-fine point movement network (TCFNet) to learn unique, complicated correspondences at the patch and point levels for dense face-bone point cloud transformations. This end-to-end framework adopts a Transformer-based network and a local information aggregation network (LIA-Net) in the first and second stages, respectively, which reinforce each other to generate precise point movement paths. LIA-Net can effectively compensate for the neighborhood precision loss of the Transformer-based network by modeling local geometric structures (edges, orientations and relative position features). The previous global features are employed to guide the local displacement using a gated recurrent unit. Inspired by deformable medical image registration, we propose an auxiliary loss that can utilize expert knowledge for reconstructing critical organs. Our framework is an unsupervised algorithm, and this loss is optional. Compared with the existing state-of-the-art (SOTA) methods on gathered datasets, TCFNet achieves outstanding evaluation metrics and visualization results. The code is available at https://github.com/Runshi-Zhang/TCFNet.

Dual-arm robots (e.g., humanoid robots) possess substantial potential for executing collaborative tasks in universal scenarios. However, the workspaces of the individual arms frequently overlap, rendering self-collision avoidance crucial for maintaining safe robotic operations. Existing motion planning-based methods demonstrate inadequate real-time efficacy, and learning-based distance proxy methods are subject to elevated false positive rates. To address these challenges, we present a novel minimum distance prediction neural network for reactive collision avoidance of dual-arm robots, which considers the continuous motion of the robotic arms and the interrelation of joint configurations. The temporal joint configurations are encoded and divided into historical and current features. A state-space model is utilized to capture the temporal dependency of historical features. A self-attention mechanism is employed to model the hidden relationships among current features. The integration of historical and current features via a cross-attention mechanism followed by a gated fusion module allows for precise prediction of the minimum distance between the dual arms. Simulations and real-world experiments, including human-robot interaction and autonomous grasping tasks, were conducted using two redundant robotic arms. The proposed method achieves an average error of 1.804 cm in minimum distance prediction. In dual-arm autonomous grasping experiments, an average error of 1.471 cm is attained. Our approach has improved accuracy by 38.08% over the state-of-the-art methods. No collisions occurred throughout all real-world experiments. This method holds promise for extensive applications of dual-arm robots. The code is accessible at https://github.com/XuejinLuo/SelfCollision.

Complicated image registration is a key issue in medical image analysis, and deep learning-based methods have achieved better results than traditional methods. The methods include ConvNet-based and Transformer-based methods. Although ConvNets can effectively utilize local information to reduce redundancy via small neighborhood convolution, the limited receptive field results in the inability to capture global dependencies. Transformers can establish long-distance dependencies via a self-attention mechanism; however, the intense calculation of the relationships among all tokens leads to high redundancy. We propose a novel unsupervised image registration method named the unified Transformer and superresolution (UTSRMorph) network, which can enhance feature representation learning in the encoder and generate detailed displacement fields in the decoder to overcome these problems. We first propose a fusion attention block to integrate the advantages of ConvNets and Transformers, which inserts a ConvNet-based channel attention module into a multihead self-attention module. The overlapping attention block, a novel cross-attention method, uses overlapping windows to obtain abundant correlations with match information of a pair of images. Then, the blocks are flexibly stacked into a new powerful encoder. The decoder generation process of a high-resolution deformation displacement field from low-resolution features is considered as a superresolution process. Specifically, the superresolution module was employed to replace interpolation upsampling, which can overcome feature degradation. UTSRMorph was compared to state-of-the-art registration methods in the 3D brain MR (OASIS, IXI) and MR-CT datasets (abdomen, craniomaxillofacial). The qualitative and quantitative results indicate that UTSRMorph achieves relatively better performance. The code and datasets are publicly available at https://github.com/Runshi-Zhang/UTSRMorph.

Osteotomy holds a pivotal position among the fundamental procedures in craniomaxillofacial (CMF) surgery. However, there are inherent challenges and risks associated with ensuring the recuperation of occlusion, safeguarding the facial nerves and blood vessels, as well as preserving facial aesthetics. In this study, a non-invasive image-to-patient registration method for navigation/robotic CMF surgery based on intraoperative freehand ultrasound (US) 3D reconstruction is proposed. Building upon this, a CMF osteotomy robotic system with compliant human-robot interaction and osteotomy trajectory planning was devised. In the freehand US 3D reconstruction and registration experiments, the registration errors for human volunteers and phantoms were consistently less than 1 mm. In robot osteotomy experiments based on the resulting registration, the average osteotomy error was below 1.5 mm. The proposed US 3D reconstruction based registration method is non-invasive and radiation-free, and shows the promising accuracy which is suitable for CMF robotic or navigation systems.

Craniomaxillofacial (CMF) surgery always relies on accurate preoperative planning to assist surgeons, and automatically generating bone structures and digitizing landmarks for CMF preoperative planning is crucial. Since the soft and hard tissues of the CMF regions possess complicated attachment, segmenting the CMF bones and detecting the CMF landmarks are challenging problems. In this study, we proposed a semantic segmentation network to segment the maxilla, mandible, zygoma, zygomatic arch, and frontal bones. Then, we obtained the minimum bounding box around the CMF bones. After cropping, we used the top-down heatmap landmark detection network, similar to the segmentation module, to identify 18 CMF landmarks from the cropping patch. In addition, an unbiased heatmap encoding method was proposed to generate actual landmark coordinates in the heatmap. To overcome quantization effects in the heatmap-based landmark detection networks, the distribution-prior coordinate representation of medical landmarks (DCRML) was proposed to utilize the prior distribution of the encoding heatmap, approximating the accurate landmark coordinates in heatmap decoding by Taylor's theorem. The encoding and decoding method can easily contribute to other existing landmark detection frameworks based on heatmaps; consequently, these approaches can readily benefit without changing model structure. We used prior segmentation knowledge to enhance the semantic information around the landmarks, increasing landmark detection accuracy. The proposed framework was evaluated by 100 healthy persons and 86 patients from multicenter cooperation. The mean Dice score of our proposed segmentation network achieved over 88 %; in particular, the mandible accuracy was approximately 95%. The mean error of landmarks was 1.84 ±1.32 mm.