Either RGB images or inertial signals have been used for the task of motion capture (mocap), but combining them together is a new and interesting topic. We believe that the combination is complementary and able to solve the inherent difficulties of using one modality input, including occlusions, extreme lighting/texture, and out-of-view for visual mocap and global drifts for inertial mocap. To this end, we propose a method that fuses monocular images and sparse IMUs for real-time human motion capture. Our method contains a dual coordinate strategy to fully explore the IMU signals with different goals in motion capture. To be specific, besides one branch transforming the IMU signals to the camera coordinate system to combine with the image information, there is another branch to learn from the IMU signals in the body root coordinate system to better estimate body poses. Furthermore, a hidden state feedback mechanism is proposed for both two branches to compensate for their own drawbacks in extreme input cases. Thus our method can easily switch between the two kinds of signals or combine them in different cases to achieve a robust mocap. Quantitative and qualitative results demonstrate that by delicately designing the fusion method, our technique significantly outperforms the state-of-the-art vision, IMU, and combined methods on both global orientation and local pose estimation. Our codes are available for research at https://shaohua-pan.github.io/robustcap-page/.

Vibration signals are generally utilized for machinery fault diagnosis to perform timely maintenance and then reduce losses. Thus, the feature extraction on one-dimensional vibration signals often determines accuracy of those fault diagnosis models. These typical deep neural networks (DNNs), e.g., convolutional neural networks (CNNs), perform well in feature learning and have been applied in machine fault diagnosis. However, the supervised learning of CNN often requires a large amount of labeled images and thus limits its wide applications. In this article, a new DNN, one-dimensional residual convolutional autoencoder (1-DRCAE), is proposed for learning features from vibration signals directly in an unsupervised-learning way. First, 1-D convolutional autoencoder is proposed in 1-DRCAE for feature extraction. Second, a deconvolution operation is developed as decoder of 1-DRCAE to reconstruct the filtered signals. Third, residual learning is employed in 1-DRCAE to perform feature learning on 1-D vibration signals. The results show that 1-DRCAE has good signal denoising and feature extraction performance on vibration signals. It performs better on feature extraction than the typical DNNs, e.g., deep belief network, stacked autoencoders, and 1-D CNN.