Abstract In order to align the virtual and real content precisely through augmented reality devices, especially in optical see-through head-mounted displays (OST-HMD), it is necessary to calibrate the device before using it. However, most existing methods estimated the parameters via 3D-2D correspondences based on the 2D alignment, which is cumbersome, time-consuming, theoretically complex, and results in insufficient robustness. To alleviate this issue, in this paper, we propose an efficient and simple calibration method based on the principle of directly calculating the projection transformation between virtual space and the real world via 3D-3D alignment. The proposed method merely needs to record the motion trajectory of the cube-marker in the real and virtual world, and then calculate the transformation matrix between the virtual space and the real world by aligning the two trajectories in the observed view. There are two advantages associated with the proposed method. First, the operation is simple. Theoretically, the user only needs to perform four alignment operations for calibration without changing the rotation variation. Second, the trajectory can be easily distributed throughout the entire observation view, resulting in more robust calibration results. To validate the effectiveness of the proposed method, we conducted extensive experiments on our self-built optical see-through head-mounted display (OST-HMD) device. The experimental results show that the proposed method can achieve better calibration results than other calibration methods.

The proliferation of single-cell multimodal sequencing technologies has enabled us to understand cellular heterogeneity with multiple views, providing novel and actionable biological insights into the disease-driving mechanisms. Here, we propose a comprehensive end-to-end single-cell multimodal analysis framework named Deep Parametric Inference (DPI). DPI transforms single-cell multimodal data into a multimodal parameter space by inferring individual modal parameters. Analysis of cord blood mononuclear cells (CBMC) reveals that the multimodal parameter space can characterize the heterogeneity of cells more comprehensively than individual modalities. Furthermore, comparisons with the state-of-the-art methods on multiple datasets show that DPI has superior performance. Additionally, DPI can reference and query cell types without batch effects. As a result, DPI can successfully analyze the progression of COVID-19 disease in peripheral blood mononuclear cells (PBMC). Notably, we further propose a cell state vector field and analyze the transformation pattern of bone marrow cells (BMC) states. In conclusion, DPI is a powerful single-cell multimodal analysis framework that can provide new biological insights into biomedical researchers. The python packages, datasets and user-friendly manuals of DPI are freely available at https://github.com/studentiz/dpi.

High-accuracy, low-latency gaze tracking is becoming one of the indispensable features in augmented reality (AR) head-mounted devices (HMDs). Researchers have proposed different approaches to predict gaze positions from eye images. However, since only the eye modality is focused, these appearance-based algorithms are still struggle to trade off the accuracy and running speed in HMDs. In this paper, we propose a lightweight multi-modal network (HE-Tracker) to regress gaze positions. By fusing head-movement features with eye features, HE-Tracker achieves comparable accuracy (3.655° in all subjects) and $27 \times$ speedup (48 fps in the specialized AR HMD) compared to the state-of-the-art gaze tracking algorithm. We further demonstrate that when applying our head-eye coordination strategy to other baseline models, all these models achieve at least 6.36% performance improvement without a pronounced effect on running speed. Moreover, we construct HE-Gaze, the first multi-modal dataset with eye images and head-movement data for near-eye gaze tracking. This dataset is currently made of 757,360 frames and 15 persons, providing an opportunity to foster research in multi-modal gaze tracking approaches. Our dataset is available at DOWNLOAD LINK1.

Abstract The multi‐camera calibration is an essential step for many spatially aware applications, such as robotic navigation, augmented reality, and 3D human pose estimation. Traditional calibration methods use off‐the‐shelf checkerboards or triangles as the known world coordinate system and their corresponding corners are set as control points, which heavily depends on specific calibration patterns and is not suitable for calibration pattern‐denied environments. In this paper, an automatic calibration method is proposed to calibrate the multi‐camera system without the aid of a known calibration pattern. The key idea of the proposed method is that the authors consider the human body, which is always available, as the counterpart of the calibration pattern. The authors’ approach starts with binocular camera calibration, in which the extrinsic and intrinsic parameters are calculated in order and followed by a joint optimisation. With the results of each pair of binocular camera calibration, the multi‐camera system calibration is carried out in three steps: (i) parameters initialisation, (ii) extrinsic parameters optimisation, and (iii) jointly optimising intrinsic and extrinsic parameters. Since the authors’ approach does not require additional calibration patterns except for one visible person, it is flexible and easy to be implemented. Real experiments are conducted in different scenes, camera angles, and camera settings. Human pose estimation with the multi‐camera system is additionally performed for exhaustive experiments. The experimental results demonstrate that the authors’ method shows superior performance than the traditional method with the aid of a specific calibration pattern.

This paper proposes a new decoupling control for permanent magnet synchronous motor (PMSM) in which both the support vector machine generalized inverse (SVMGI) and the internal model control (IMC) are employed. By introducing the SVMGI systems into the original PMSM system, a new pseudo-linear system is developed. Moreover, based on the characteristics of the pseudo-linear system, the IMC theory is utilized to design extra controllers to improve the robustness of the whole system. Consequently, the proposed control strategy incorporates the merits of both the SVMGI and the IMC. The utility of the proposed control strategy is demonstrated by both the simulation and the experiments at various operations. The results validate successful decoupling of the original system, strong robustness to load disturbance and good dynamic performance.

Target localization system is one of the basic aspects for mobile robot navigation. One of the most popular navigation way is the navigation based on visual. But, this way is limited by the light and the field of view so that it can't accurately detect the targets in special environments. To resolve the above problem of target localization way based on visual, this paper has studied the sound source target localization system. The key technologies of this system have been discussed. They are the structure of the microphone array, the multi acoustic source separation in dynamic environment, sound source localization method in mixed noise environment, the time delay estimation method in the mixed-noise environment, multi-sensor information fusion, and so on. Finally, an auditory system based on microphone array was found to locate the sound source target of 3-D space and some emulational experiments validate the correctness of this auditory system and the accuracy and realtime of the sound source localization.