To cultivate professional sports referees, we develop a sports referee training system, which can recognize whether a trainee wearing the Myo armband makes correct judging signals while watching a prerecorded professional game. The system has to correctly recognize a set of gestures related to official referee's signals (ORSs) and another set of gestures used to intuitively interact with the system. These two gesture sets involve both large motion and subtle motion gestures, and the existing sensor-based methods using handcrafted features do not work well on recognizing all kinds of these gestures. In this work, deep belief networks (DBNs) are utilized to learn more representative features for hand gesture recognition, and selective handcrafted features are combined with the DBN features to achieve more robust recognition results. Moreover, a hierarchical recognition scheme is designed to first recognize the input gesture as a large or subtle motion gesture, and the corresponding classifiers for large motion gestures and subtle motion gestures are further used to obtain the final recognition result. Moreover, the Myo armband consists of eight-channel surface electromyography (sEMG) sensors and an inertial measurement unit (IMU), and these heterogeneous signals can be fused to achieve better recognition accuracy. We take basketball as an example to validate the proposed training system, and the experimental results show that the proposed hierarchical scheme considering DBN features of multimodality data outperforms other methods.

Tactic learning in virtual reality (VR) has been proven to be effective for basketball training. Endowed with the ability of generating virtual defenders in real time according to the movement of virtual offenders controlled by the user, a VR basketball training system can bring more immersive and realistic experiences for the trainee. In this article, an autoregressive generative model for instantly producing basketball defensive trajectory is introduced. We further focus on the issue of preserving the diversity of the generated trajectories. A differentiable sampling mechanism is adopted to learn the continuous Gaussian distribution of player position. Moreover, several heuristic loss functions based on the domain knowledge of basketball are designed to make the generated trajectories assemble real situations in basketball games. We compare the proposed method with the state-of-the-art works in terms of both objective and subjective manners. The objective manner compares the average position, velocity, and acceleration of the generated defensive trajectories with the real ones to evaluate the fidelity of the results. In addition, more high-level aspects such as the empty space for offender and the defensive pressure of the generated trajectory are also considered in the objective evaluation. As for the subjective manner, visual comparison questionnaires on the proposed and other methods are thoroughly conducted. The experimental results show that the proposed method can achieve better performance than previous basketball defensive trajectory generation works in terms of different evaluation metrics.

In this article, a VR-based basketball training system comprising a standalone VR device and a tablet is proposed. The system is intended to improve the ability of players to understand offensive tactics and practice these tactics correctly. We compare the training effectiveness of various degrees of immersion, including a conventional basketball tactic board, a 2D monitor, and virtual reality. A multi-camera-based human tracking system was designed and built around a real-world basketball court to record and analyze the running trajectory of each player during tactical execution. The accuracy of the running path and hesitation time at each tactical step were evaluated for each participant. Furthermore, we assessed several subjective measurements, including simulator sickness, presence, and sport imagery ability, to conduct a more comprehensive exploration of the feasibility of the proposed VR framework for basketball tactics training. The results indicate that the proposed system is useful for learning complex tactics. Furthermore, high VR immersion training improves athletes' abilities with regards to strategic imagery.

Introducing olfactory display in the virtual reality (VR) system brings the immersive experience to new heights. However, it is intractable to simulate olfactory features (such as the intensity and the direction) with multiple levels. Visual stimuli have been proved to dominate human perception among multiple sensors in virtual environments. If visual stimuli can be used to guide the olfactory sense in VR, the design of the olfactory display can be simpler but still able to provide olfactory experience with more diversity. To understand the visual-olfactory effect on different olfactory characteristics, a portable olfactory display that can control the intensity and direction of odors was developed. An experimental study was conducted to investigate cross-modal human perception, i.e. how the visually virtual odor representation in VR influences human perception of real odor produced by the proposed olfactory display. The results showed that the perception of odor intensity and directionality can be modulated by visually virtual odor representation.

Learning basketball tactic via virtual reality environment requires real-time feedback to improve the realism and interactivity. For example, the virtual defender should move immediately according to the player's movement. In this paper, we proposed an autoregressive generative model for basketball defensive trajectory generation. To learn the continuous Gaussian distribution of player position, we adopt a differentiable sampling process to sample the candidate location with a standard deviation loss, which can preserve the diversity of the trajectories. Furthermore, we design several additional loss functions based on the domain knowledge of basketball to make the generated trajectories match the real situation in basketball games. The experimental results show that the proposed method can achieve better performance than previous works in terms of different evaluation metrics.

We propose a robust basketball player tracking framework for multi-cameras which have high portion of overlapping with each other and are set at human height. A novel detection grouping method is proposed to more correctly merge the projected detection results. Instead of using linear motion assumption to predict the human motion, we applied a regional consistency assumption to calculate the motion affinity. Further-more, we design a one-to-one clustering method to associate the most matching tracklets together using correlation values between tracklets and generate final trajectory results. Since there is no public labeled overlapping cross-cameras basketball dataset, we collected our own dataset, MISBasketball, and labeled the ground truth to evaluate the proposed tracking framework.

Decision-making is an essential part in basketball offenses. In this paper, we proposed a basketball offensive decision-making VR training system. During the training, the trainee can intuitively interact with the system by wearing a motion capture suit and be trained in different virtual defensive scenarios designed by professional coaches. The system will recognize the offensive action performed by the user and provide correct suggestions when he/she makes a bad offensive decision. We compared the effectiveness of the training protocols by using conventional tactics board and the proposed VR system. Furthermore, we investigated the influence of using prerecorded 360-degree panorama video and computer simulated virtual content to create immersive training environment.

In this paper, we present a basketball tactic training framework with the aid of virtual reality (VR) technology to improve the effectiveness and experience of tactic learning. Our proposal is composed of 1) a wireless VR interaction system with motion capture devices which is applicable in the fast movement basketball running scenario; 2) a computing server that generates three-dimensional virtual players, defenders, and advantageous tactics guide. By the assistance of our VR training system, the user can vividly experience how the tactics are executed by viewing from the a specific player's viewing direction. Moreover, the basketball tactic movement guidance and virtual defenders are rendered in our VR system to make the users feel like playing in a real basketball game, which improves the efficiency and effectiveness of tactics training.

Harmonizing the style of all the furniture placed within a constrained space/scene has been regarded as one of the most important tasks in interior design. Most previous style analysis works measure the style similarity or compatibility of the objects based on predefined geometric features extracted from 3D models. However, “style” is a high-level semantic concept, which is difficult to be described explicitly by handcrafted geometric features. Deep neural network has been claimed to have more powerful ability to mimic the perception of human visual cortex. Therefore, in this work we utilize Triplet Convolutional Neural Network (Triplet CNN) to analyze style compatibility between 3D furniture models of different classes (e.g., a table and a lamp). It should be noted that analyzing the style compatibility between two or more furniture of different classes is quite difficult, as the given furniture may have distinctive structures or geometric elements. We conducted experiments based on a collected dataset containing 420 textured 3D furniture models. A group of raters were recruited from Amazon Mechanical Turk (AMT) to evaluate the comparative suitability of paired models within the dataset. The experimental results reveal that the proposed furniture style compatibility method based on deep learning is better than the state-of-the-art method and can be used for furniture recommendation.

In this demo, we present an immersive virtual reality (VR) system which integrates multimodal interaction sensors (i.e., smartphone, Kinect v2, and Myo armband) and streaming technology to improve the VR experience. The integrated system solves the common problems in most VR systems: (1) the very limited playing area due to transmission cable between computer and display/interaction devices, and (2) non-intuitive way of controlling virtual objects. We use Unreal Engine 4 to develop an immersive VR game with 6 interactive levels to demonstrate the feasibility of our system. In the game, the user not only can freely walk within a large playing area surrounded by multiple Kinect sensors but also select the virtual objects to grab and throw with the Myo armband. The experiment shows that our idea is workable for VR experience.