Arm posture tracking is essential for many applications, such as gesture recognition, fitness training, and motion-based controls. Smartwatches with Inertial Measurement Unit (IMU) sensors (i.e., accelerometer, gyroscope, and magnetometer) provide a convenient way to track the orientation and location of the wrist. Existing orientation estimations are based on predefined data fusion methods that do not consider the variations in the data quality of different IMU sensors. Existing location estimations rely on the estimated orientation results. A small orientation estimation error may cause high inaccuracy in location estimation. Moreover, these location estimation algorithms, e.g., Hidden Markov Model and Particle Filters, cannot provide real-time tracking on commercial mobile devices due to high computation overhead. This paper presents RTAT, a Real-Time Arm Tracking system that tackles the above limitations in a data-driven way. RTAT estimates both orientation and location simultaneously using a multitask learning neural network. It also incorporates a unique attention layer and a dedicated loss function to learn the dynamic relationship among IMU sensors. RTAT supports real-time tracking by performing model inference on smartphones. Finally, to train RTAT's neural network, we develop an easy-to-use labeled data collection system that uses a low-cost virtual reality system to provide orientation and location labels for the smartwatch. Extensive experiments show RTAT significantly outperforms existing state-of-the-art solutions in both accuracy and latency.

In this demo, we present RTAT, a real-time arm tracking system that tracks both orientation and location of a smartwatch simultaneously by a multitask learning neural network. We incorporate an attention layer and design a dedicated loss for the multitask neural network to learn the dynamic relationships among Inertial Measurement Unit (IMU) sensors. RTAT supports real-time tracking by performing deep learning inference on a smartphone. Finally, to train RTAT, we develop an easy-to-use labeled data collection system that uses a low-cost virtual reality system to measure the ground truth orientation and location of the smartwatch. Extensive experiments show RTAT outperforms significantly the state-of-the-art solutions in inference accuracy and latency.