mmWave-based activity recognition technology has attracted widespread attention as it provides the ability of device-free, ubiquitous and accurate sensing. Recognition of human activities intrinsically demands to be real-time and continuous, but the state of the arts is still far limited with the capacity in this regard. The main obstacle lies in activity sequence segmentation, i.e., locating the boundaries between consecutive activities in an activity sequence. This is a daunting task, due to the unclear activity boundaries and the variable activity duration. In this paper, we proposeZuMa, the first mmWave-based approach to real-time continuous activity recognition. When resorting to a machine learning model for activity recognition, our insight is that the recognition confidence of the recognition model is highly correlated to the accuracy of activity sequence segmentation, so that the former can be utilized as a feedback metric to finely adjust the segmentation boundaries. Based on this insight,ZuMais a coarse-to-fine grained approach, which includes the fast coarse-grained activity chunk extraction and the find-grained explicit segmentation adjustment and recognition. We have implementedZuMawith the commercial mmWave radar and evaluated its performance under various settings. The results demonstrate thatZuMaachieves an average recognition error of 12.67%, which is 65.08% and 71.87% lower than that of the two baseline methods. The average recognition delay ofZuMais only 1.86 s.

Gaze tracking is a useful human-to-computer interface, which plays an increasingly important role in a range of mobile applications. Gaze calibration is an indispensable component of gaze tracking, which transforms the eye coordinates to the screen coordinates. The existing approaches of gaze tracking either have limited accuracy or require the user's cooperation in calibration and in turn hurt the quality of experience. We in this paper propose vGaze, continuous gaze tracking with implicit saliency-aware calibration on mobile devices. The design of vGaze stems from our insight on the temporal and spatial dependent relation between the visual saliency and the user's gaze. vGaze is implemented as a light-weight software that identifies video frames with “useful” saliency information, sensing the user's head movement, performs opportunistic calibration using only those “useful” frames, and leverages historical information for accelerating saliency detection. We implement vGaze on a commercial mobile device and evaluate its performance in various scenarios. The results show that vGaze can work at real time with video playback applications. The average error of gaze tracking is 1.51cm (2.884$^{\circ }$) which decreases to 0.99cm (1.891$^{\circ }$) with historical information and 0.57cm (1.089$^{\circ }$) with an indicator.

Gaze tracking is a useful human-to-computer interface, which plays an increasingly important role in a range of mobile applications. Gaze calibration is an indispensable component of gaze tracking, which transforms the eye coordinates to the screen coordinates. The existing approaches of gaze tracking either have limited accuracy or require the user's cooperation in calibration and in turn hurt the quality of experience. We in this paper propose vGaze, implicit saliency-aware calibration for continuous gaze tracking on mobile devices. The design of vGaze stems from our insight on the temporal and spatial dependent relation between the visual saliency and the user's gaze. vGaze is implemented as a light-weight software that identifies video frames with "useful" saliency information, sensing the user's head movement, and performs opportunistic calibration using only those "useful" frames. We implement vGaze on a commercial mobile device and evaluate its performance in various scenarios. The results show that vGaze can work at real time with video playback applications. The average error of gaze tracking is 1.51cm.

Consumer Virtual Reality (VR) has been widely used in various application areas, such as entertainment and medicine. In spite of the superb immersion experience, to enable high-quality VR on untethered mobile devices remains an extremely challenging task. The high bandwidth demands of VR streaming generally overburden a conventional wireless connection, which affects the user experience and in turn limits the usability of VR in practice. In this paper, we propose FoVR, attention-based hierarchical VR streaming through bandwidth-limited wireless networks. The design of FoVR stems from the insight that human's vision is hierarchical, so that different areas in the field of view (FoV) can be served with VR content of different qualities. By exploiting the gaze tracking capacity of the VR devices, FoVR is able to accurately predict the user's attention so that the streaming of hierarchical VR can be appropriately scheduled. In this way, FoVR significantly reduces the bandwidth cost and computing cost while keeping high quality of user experience. We implement FoVR on a commercial VR device and evaluate its performance in various scenarios. The experiment results show that FoVR reduces the bandwidth cost by 88.9% and 76.2%, respectively compared to the original VR streaming and the state-of-the-art approach.