In this work, we propose an end-to-end system that provides both hardware and software support for real-time gesture recognition. We apply a convolutional neural network over 3D rotation data of finger joints rather than over vision-based data, in order to extract high-level intentions (features) users are trying to convey. A pair of customized motion capturing gloves are designed with inertial measurement unit (IMU) sensors to obtain gestural datasets for network training and real-time recognition. A network reduction strategy has been developed to appropriately reduce a network's complexity in both depth and width dimensions while maintaining a high recognition accuracy with the classification model produced by the network. The classification model is able to classify new data samples by scanning a real-time stream of joint rotations during the use of the gloves. Our evaluation results expose the relationships between the network reduction hyperparameters and the change of recognition accuracy. Based on the evaluation, we are able to determine an appropriate version of the light network and achieve 98% accuracy.

In a situation where a large and chaotic collection of digital images must be manually sorted or categorized, there are two challenges: (1) unnatural actions during a prolonged human-computer interaction and (2) limited display space for image browsing. An immersive 3D interface is prototyped, where a person sorts a large collection of digital images with his or her bare hands in a virtual environment, and performs hand motions matching characteristics of sorting gestures in the real world. The virtual reality environment provides extra levels of immersion for displaying images.