Precision orthodontics relies heavily on timely precision measurement during an orthodontic treatment. However, the measurement done by orthodontists currently (i.e., calipers) cannot provide a fast and precise measurement. The study introduces a novel real-time automatic image-based system for precision tooth measurement, characterized by its high accuracy and efficiency. The video input of the dentures from a head-mounted camera was taken one frame at a time. Since 70% of orthodontal treatment is done using fixed brackets, we used an object detection algorithm to locate the position of the brackets within the frame. A novel multi-object tracking algorithm was developed to track and update the positions based on the long short-time memory information collected on the previous frames. A measurement module then measures and updates the distance between each bracket and saves it making sure the measurements are available even when the bracket is not visible in the current frame. The real-time tracking and measurement improved the efficiency and accuracy of tooth measurement. In clinical cases, the teeth were recognized with 97% accuracy and tracked without any loss within a video input of 60 FPS. This measurement achieves a resolution on the order of 0.01 mm, with a data error less than 50 μm. This real-time automatic tooth measurement proved to be efficient and easy to use compared to existing methods while improving accuracy. It can be used in aiding precision orthodontics and in the process of automating dental treatments.

Implicit scene representations have recently shown promising results in photo-realistic 3D reconstruction and view synthesis based on calibrated views. However, their applications face several challenges, including unknown camera pose, boundary ambiguity, and observation noise. This paper proposes a novel online scene representation method that simultaneously learns to represent the target scene and estimates the camera poses from an RGB-D stream. An implicit scene representation function built with scale-encoded cascaded grids is proposed to represent scenes online from incremental observations. This implicit function is optimized in a reparameterized domain that provides defined boundaries. In this reparameterized domain, the cascaded grids are progressively distilled under geometric and photometric supervision to improve their model capacity and geometry accuracy. A radiance field deblurring module based on the physical imaging process is further proposed to restore a photo-realistic reconstruction against camera motion blur, which is the main component of the observation noise. The proposed method can produce sharp and photo-realistic representations of scenes under various shooting conditions without known camera poses. Experiments on multiple datasets have demonstrated the effectiveness of the proposed method in improving view synthesis and camera tracking results for online scene representation tasks.

Achieving accurate face reconstruction with geometry details from a single-view images is an important task for orthodontics. Although the 3D Morphable Model (3DMM) based methods provide an effective framework, the low-dimensional linear space is insufficient to cover geometric details. In this paper, we propose a hybrid shape deformation representation with multi-branch supervision for detail prediction. In orthodontic scenarios, shape deformation can be considered as the aggregation of intuitive appearance component and ambiguous geometry component, which is involved in frontal shape and depth correction respectively. Hence orthogonal decomposition is employed to decompose the shape deformation into frontal-plane position offset and depth offset. Frontal-plane position offset is represented in an explicit-local-dependent manner based on grid deformation while depth offset is represented in an implicit-local-dependent manner based on dense prediction. To facilitate orthodontic-based evaluation, we construct an orthodontic-specific dataset and design a novel metric to involve the relative position dependency between regions of interest. Experimentally, we demonstrate outstanding performance of face reconstruction on FaceScape, MICC Florence and orthodontic-specific dataset with both quantitative and qualitive evaluation.

Establishing reliable correspondences between two views is one of the most important components of various vision tasks. This paper proposes a novel sparse-to-local-dense (S2LD) matching method to conduct fully differentiable correspondence estimation with the prior from epipolar geometry. The sparse-to-local-dense matching asymmetrically establishes correspondences with consistent sub-pixel coordinates while reducing the computation of matching. The salient features are explicitly located, and the description is conditioned on both views with the global receptive field provided by the attention mechanism. The correspondences are progressively established in multiple levels to reduce the underlying re-projection error. We further propose a 3D noise-aware regularizer with differentiable triangulation. Additional guidance from 3D space is encoded by the regularizer in training to handle the supervision noise caused by the errors in camera poses and depth maps. The proposed method demonstrates outstanding matching accuracy and geometric estimation capability on multiple datasets and tasks.

3D reconstruction and understanding from monocular camera is a key issue in computer vision. Recent learning-based approaches, especially multi-task learning, significantly achieve the performance of the related tasks. However a few works still have limitation in drawing loss-spatial-aware information. In this paper, we propose a novel Joint-confidence-guided network (JCNet) to simultaneously predict depth, semantic labels, surface normal, and joint confidence map for corresponding loss functions. In details, we design a Joint Confidence Fusion and Refinement (JCFR) module to achieve multi-task feature fusion in the unified independent space, which can also absorb the geometric-semantic structure feature in the joint confidence map. We use confidence-guided uncertainty generated by the joint confidence map to supervise the multi-task prediction across the spatial and channel dimensions. To alleviate the training attention imbalance among different loss functions or spatial regions, the Stochastic Trust Mechanism (STM) is designed to stochastically modify the elements of joint confidence map in the training phase. Finally, we design a calibrating operation to alternately optimize the joint confidence branch and the other parts of JCNet to avoid overfiting. The proposed methods achieve state-of-the-art performance in both geometric-semantic prediction and uncertainty estimation on NYU-Depth V2 and Cityscapes.

Since the tumor moves with the patient's breathing movement in clinical surgery, the real-time prediction of respiratory movement is required to improve the efficacy of radiotherapy. Some RNN-based respiratory management methods have been proposed for this purpose. However, these existing RNN-based methods often suffer from the degradation of generalization performance for a long-term window (such as 600 ms) because of the structural consistency constraints. In this paper, we propose an innovative Long Short-term Transformer (LSTformer) for long-term real-time accurate respiratory prediction. Specifically, a novel Long-term Information Enhancement module (LIE) is proposed to solve the performance degradation under a long window by increasing the long-term memory of latent variables. A lightweight Transformer Encoder (LTE) is proposed to satisfy the real-time requirement via simplifying the architecture and limiting the number of layers. In addition, we propose an application-oriented data augmentation strategy to generalize our LSTformer to practical application scenarios, especially robotic radiotherapy. Extensive experiments on our augmented dataset and publicly available dataset demonstrate the state-of-the-art performance of our method on the premise of satisfying the real-time demand.

In this paper, we propose a disparity refinement method that directly refines the winner-take-all (WTA) disparity map by exploring its statistical significance. According to the primary steps of the segment-based stereo matching, the reference image is over-segmented into superpixels and a disparity plane is fitted for each superpixel by an improved random sample consensus (RANSAC). We design a two-layer optimization to refine the disparity plane. In the global optimization, mean disparities of superpixels are estimated by Markov random field (MRF) inference, and then, a 3D neighborhood system is derived from the mean disparities for occlusion handling. In the local optimization, a probability model exploiting Bayesian inference and Bayesian prediction is adopted and achieves second-order smoothness implicitly among 3D neighbors. The two-layer optimization is a pure disparity refinement method because no correlation information between stereo image pairs is demanded during the refinement. Experimental results on the Middlebury and KITTI datasets demonstrate that the proposed method can perform accurate stereo matching with a faster speed and handle the occlusion effectively. It can be indicated that the "matching cost computation + disparity refinement" framework is a possible solution to produce accurate disparity map at low computational cost.

Three-dimensional (3D) models of tooth-alveolar bone complex are needed in treatment planning and simulation for computer-aided orthodontics. Tooth and alveolar bone segmentation from computed tomography (CT) images is a fundamental step in reconstructing their models. Due to less application of alveolar bone in conventional orthodontic treatment which may cause undesired side effects, the previous studies mainly focused on tooth segmentation and reconstruction, and did not consider the alveolar bone. In this study, we proposed a method to implement both tooth and alveolar bone segmentation from dental CT images for reconstructing their 3D models. First, the proposed method extracted the connected region of tooth and alveolar bone from CT images using a global convex level set model. Then, individual tooth and alveolar bone are separated from the connected region based on Radon transform and a local level set model. The experimental results showed that the proposed method could successfully complete both the tooth and alveolar bone segmentation from CT images, and outperformed the state of the art tooth segmentation methods in terms of accuracy. This suggests that the proposed method can be used in reconstructing the 3D models of tooth-alveolar bone complex for precise treatment.

This paper studies the problem of image-based leader-follower formation control for mobile robots, where the controller is designed independently of the leader's motion. An adaptive control scheme, which is suitable for both omnidirectional and perspective cameras, is proposed. The proposed approach avoids the need for accurate calibration of the extrinsic parameters of the omnidirectional camera as well as the intrinsic and extrinsic parameters of perspective camera. Additionally, the coefficients of the plane where the feature point moves relative to the camera frame can be uncertain. These uncertain constant parameters are estimated using an adaptive estimator. Uniform Semi-global Practical Asymptotic Stability (USPAS) of the system is shown using the Lyapunov approach. Experimental results are presented to demonstrate the effectiveness of the proposed control scheme.

Customized archwires are demanded in the lingual orthodontic treatment for patients suffering from malocclusion. Traditionally, these archwires could only be bent by experienced orthodontists manually. This pattern requires a specialized skills training and occupies long charside time, but still cannot ensure the accuracy of appliances. Therefore, a novel robotic system was developed for automatic and accurate preparation in our study. First, the implementation of hardware system was designed. Second, a modular and ROS-integrated control system was developed to control automatic bending. Third, an adaptive sampling-based bending planner with collision checker in a time-varying environment was established and realized in control system architecture. Preliminary validation of the developed robot system and its control system have been conducted both in simulation and physical robotic system. Experimental results have shown that the developed robotic system with its ROS-integrated control system was able to accomplish automatic and accurate orthodontic archwire preparation.

Tooth segmentation from dental computed tomography (CT) images with metal artifacts is challenging as metal artifacts make some of the crown boundaries unrecognizable. This letter proposes a semiautomatic method for crown segmentation from CT images with metal artifacts. A user manually selects a starting slice and initializes this slice. Then crown contours are segmented automatically from volumetric CT images slice by slice. In the segmentation of each slice, the Radon transform is used to extract a line to separate neighboring crowns into independent ones. A statistical shape prior-based level set model is then applied to segment each crown from the mesial or distal side of the line. The proposed method was tested on 15 set of volumetric images. Experimental results validated that it is effective to extract crown contours from CT images with metal artifacts.

In clinics, customized archwires are demanded for lingual orthodontic treatment. However, only very experienced orthodontists can handle the manual appliance preparation. This pattern not only occupies lots of the orthodontist's labor time, but also can not ensure the accuracy of the appliances. Therefore, a robotic system was developed for automatic and accurate orthodontic archwire bending in our study. First, a method for customized archwire parameterization was developed. Second, an adaptive sampling-based bending planner with collision checker in time-varying environment was designed. Finally, a bending control strategy was used to eliminate the springback effect of the archwires and bending point shift during the bending process. A self-developed simulation platform based on Robot Operating System with MoveIt was used for preliminary validation of the proposed method. Physical experiments for multi-functional orthodontic bends on the robotic system were conducted as well. The results have shown that the developed robotic system using the proposed planning and control method was able to accomplish the automatic and accurate orthodontic archwire bending.