This paper presents a novel approach for performing intuitive 3D gesture-based interaction using depth data acquired by Kinect. Unlike current depth-based systems that focus only on classical gesture recognition problem, we also consider 3D gesture pose estimation for creating immersive gestural interaction. In this paper, we formulate gesture-based interaction system as a combination of two separate problems, gesture recognition and gesture pose estimation. We focus on the second problem and propose a direct method for recovering hand motion parameters. Based on the range images, a new version of optical flow constraint equation is derived, which can be utilized to directly estimate 3D hand motion without any need of imposing other constraints. Our experiments illustrate that the proposed approach performs properly in real-time with high accuracy. As a proof of concept, we demonstrate the system performance in 3D object manipulation. This application is intended to explore the system capabilities in real-time biomedical applications. Eventually, system usability test is conducted to evaluate the learn ability, user experience and interaction quality in 3D interaction in comparison to 2D touch-screen interaction.

This paper proposes an intuitive wireless sensor/actuator based communication network for human animal interaction for a digital zoo. In order to enhance effective observation and control over wild life, we have built a wireless sensor network. 25 video transmitting nodes are installed for animal behavior observation and experimental vibrotactile collars have been designed for effective control in an animal park.

We have developed a head gesture controlled electric wheelchair system to aid persons with severe disabilities. Realtime range information obtained from a stereo camera is used to locate and segment the face images of the user from the sensed video. We use an Isomap based nonlinear manifold learning map of facial textures for head pose estimation. Our system is a non-contact vision system, making it much more convenient to use. The user is only required to gesture his/her head to command the wheelchair. To overcome problems with a non responding system, it is necessary to notify the user of the exact system state while the system is in use. In this paper, we explore the use of vibrotactile rendering of head gestures as feedback. Three different feedback systems are developed and tested, audio stimuli, vibrotactile stimuli and audio plus vibrotactile stimuli. We have performed user tests to study the usability of these three display methods. The usability studies show that the method using both audio plus vibrotactile response outperforms the other methods (i.e. audio stimuli, vibrotactile stimuli response).

Vibration offers many potential benefits for the use of mobile phones. In this paper, we propose a new method of rendering live football game on mobile phones using vibration. A mobile phone is “synchronized” with the ball in the real field. By holding the phone, users are able to experience dynamic movements of the ball, to know attacking directions and which team is leading the attack. The usability test of our system shows that vibrotactile display is suitable for rendering live football information on mobile phones by adopting designed coding schemes with a right training process.

To enhance their social interactive ability, we study how to provide the sight-impaired with reliable, "on-line" emotion information. The technical challenge we address here is how to render rich facial expressions in an intuitive way. We demonstrate that manifold of facial expressions is a compact and natural way to characterize human emotions. To compute manifold of facial expressions the standard locally linear embedding (LLE) algorithm is extended to handle the problem of real-time coding of new videos. Experimental results show that using manifold of facial expressions for vibrotactile rendering is very encouraging.