Redirected walking (RDW) and omnidirectional treadmill (ODT) are two effective solutions to the natural locomotion interface in virtual reality. ODT fully compresses the physical space and can be used as the integration carrier of all kinds of devices. However, the user experience varies in different directions of ODT, and the premise of interaction between users and integrated devices is a good match between virtual and real objects. RDW technology uses visual cues to guide the user's location in physical space. Based on this principle, combining RDW technology with ODT to guide the user's walking direction through visual cues can effectively improve user experience on ODT and make full use of various devices integrated on ODT. This paper explores the novel prospects of combining RDW technology with ODT and formally puts forward the concept of O-RDW (ODT-based RDW). Two baseline algorithms, i.e., OS2MD (ODT-based steer to multi-direction), and OS2MT (ODT-based steer to multi-target), are proposed to combine the merits of both RDW and ODT. With the help of the simulation environment, this paper quantitatively analyzes the applicable scenarios of the two algorithms and the influence of several main factors on the performance. Based on the conclusions of the simulation experiments, the two O-RDW algorithms are successfully applied in the practical application case of multi-target haptic feedback. Combined with the user study, the practicability and effectiveness of O-RDW technology in practical use are further verified.

Simultaneous localization and mapping (SLAM) is a key technology that provides user equipment (UE) tracking and environment mapping services, enabling the deep integration of sensing and communication. The millimeter-wave (mmWave) communication, with its larger bandwidths and antenna arrays, inherently facilitates more accurate delay and angle measurements than sub-6 GHz communication, thereby providing opportunities for SLAM. However, none of the existing works have realized the SLAM function under the 5G new radio (NR) standard due to specification and hardware constraints. In this study, we investigate how 5G mmWave communication systems can achieve situational awareness without changing the transceiver architecture and 5G NR standard. We implement 28-GHz mmWave transceivers that deploy OFDM-based 5G NR waveform with 160-MHz channel bandwidth, and we realize beam management following the 5G NR. Furthermore, we develop an efficient successive cancellation-based angle extraction approach to obtain angles of arrival and departure from the reference signal received power measurements. On the basis of angle measurements, we propose an angle-only SLAM algorithm to track UE and map features in the radio environment. Thorough experiments and ray tracing-based computer simulations verify that the proposed angle-based SLAM can achieve submeter-level localization and mapping accuracy with a single base station and without the requirement of strict time synchronization. Our experiments also reveal many propagation properties critical to the success of SLAM in 5G mmWave communication systems.

In this study, a graph regularized algorithm for early expression detection (EED), called GraphEED, is proposed. EED is aimed at detecting the specified expression in the early stage of a video. Existing EED detectors fail to explicitly exploit the local geometrical structure of the data distribution, which may affect the prediction performance significantly. According to manifold learning, the data in real-world applications are likely to reside on a low-dimensional submanifold embedded in the high-dimensional ambient space. The proposed graph Laplacian consists of two parts: 1) a k -nearest neighbor graph is first constructed to encode the geometrical information under the manifold assumption and 2) the entire expressions are regarded as the must-link constraints since they all contain the complete duration information and it is shown that this can also be formulated as a graph regularization. GraphEED is to have a detection function representing these graph structures. Even with the inclusion of the graph Laplacian, the proposed GraphEED has the same computational complexity as that of the max-margin EED, which is a well-known learning-based EED, but the detection performance has been largely improved. To further make the model appropriate in large-scale applications, with the technique of online learning, the proposed GraphEED is extended to the so-called online GraphEED (OGraphEED). In OGraphEED, the buffering technique is employed to make the optimization practical by reducing the computation and storage cost. Extensive experiments on three video-based datasets have demonstrated the superiority of the proposed methods in terms of both effectiveness and efficiency.

Electroluminescence imaging becomes a very useful technique to automatically detect defects for solar cells since it can provide high resolution electroluminescence images. However, few methods explicitly consider the visual characteristics of the defects and the noises in solar cells. In this article, a global pairwise similarity and concatenated saliency guided neural network is proposed by fully considering the observed visual characteristics in electroluminescence solar cell images. The proposed network exploits a global pairwise similarity module and a concatenated saliency module to refine the features extracted by the convolutional neural network. The global pairwise similarity module aims to refine the features of an image pixel by modeling long-range dependencies. The concatenated saliency module is exploited to suppress the background and decouple different salient regions to better represent the features of an image. Extensive experiments based on five different baselines, i.e., VGG16, ResNet56, ResNet50, DenseNet40, and GoogleNet, prove that the proposed method significantly outperforms the baseline models and show that both the global similarity module and the concatenated saliency module can help detect defective solar cells in electroluminescence images.

The natural locomotion interface is critical to the development of many VR applications. For household VR applications, there are two basic requirements: natural immersive experience and minimized space occupation. The existing locomotion strategies generally do not simultaneously satisfy these two requirements well. This article presents a novel omnidirectional treadmill (ODT) system named Hex-Core-MK1 (HCMK1). By implementing two kinds of mirror-symmetrical spiral rollers to generate the omnidirectional velocity field, this proposed system is capable of providing real walking experiences with a full-degree of freedom in an area as small as 1.76 m$^{2}$, while delivering great advantages over several existing ODT systems in terms of weight, volume, latency and dynamic performance. Compared with the sizes of Infinadeck and HCP, the two best motor-driven ODTs so far, the 8 cm height of HCMK1 is only 20% of Infinadeck and 50% of HCP. In addition, HCMK1 is a lightweight device weighing only 110 kg, which provides possibilities for further expanding VR scenarios, such as terrain simulation. The system latency of HCMK1 is only 9ms. The experiments show that HCMK1 can deliver a starting acceleration of 16.00 m/s$^{2}$and a braking acceleration of 30.00 m/s$^{2}$.

Estimating the kernel mean in a reproducing kernel Hilbert space is central to many kernel-based learning algorithms. Given a finite sample, an empirical average is used as a standard estimation of the target kernel mean. Prior works have shown that better estimators can be constructed by shrinkage methods. In this work, we propose to corrupt data examples with noise from known distributions and present a new kernel mean estimator, called the marginalized kernel mean estimator, which estimates kernel mean under the corrupted distributions. Theoretically, we justify that the marginalized kernel mean estimator introduces implicit regularization in kernel mean estimation. Empirically, on a variety of tasks, we show that the marginalized kernel mean estimator is sample-efficient and obtains much lower estimation errors than the existing estimators.

Multiview learning has received substantial attention over the past decade due to its powerful capacity in integrating various types of information. Conventional unsupervised multiview dimension reduction (UMDR) methods are usually conducted in an offline manner and may fail in many real-world applications, where data arrive sequentially and the data distribution changes periodically. Moreover, satisfying the requirements of high memory consumption and expensive retraining of the time cost in large-scale scenarios are difficult. To remedy these drawbacks, we propose an online UMDR (OUMDR) framework. OUMDR aims to seek a low-dimensional and informative consensus representation for streaming multiview data. View-specific weights are also learned in this article to reflect the contributions of different views to the final consensus presentation. A specific model called OUMDR-E is developed by introducing the exclusive group LASSO (EG-LASSO) to explore the intraview and interview correlations. Then, we develop an efficient iterative algorithm with limited memory and time cost requirements for optimization, where the convergence of each update is theoretically guaranteed. We evaluate the proposed approach in video-based expression recognition applications. The experimental results demonstrate the superiority of our approach in terms of both effectiveness and efficiency.

Epileptic seizure prediction is one of the most used therapeutic adjuvant strategies for drug-resistant epilepsy. Conventional methods usually adopt handcrafted features and manual parameter setting. The over-reliance on the expertise of specialists may lead to weak exploitation of features and low popularization of clinical application. This paper proposes a novel parameterless patient-specific method based on Fourier Neural Network (FNN), where the Fourier transform and backpropagation learning are synthesized to make the predictor more efficient and practical. The employment of FNN is the first attempt in the field of seizure prediction due to its automatic extraction of immanent spectra in epileptic signals. Despite the self-adaptive superiority of FNN, we introduce Convolutional Neural Network (CNN) to further improve its search capability in high-dimensional feature spaces. The study also develops a multi-layer module to estimate spectral power ratios of raw recordings, which optimizes the prediction by enhancing feature diversity. Based on these modules, this paper proposes a two-channel deep neural network: Fourier Ratio Convolutional Neural Network (FRCNN). To demonstrate the reliability of the model, we explain the mathematical meaning of hidden-layer neurons in FRCNN theoretically. This approach is evaluated on both intracranial and scalp EEG datasets. It shows that the predictor achieved a sensitivity of 91.2% and a false prediction rate (FPR) of 0.06[Formula: see text]h[Formula: see text] across intracranial subjects and a sensitivity of 85.4% and an FPR of 0.14[Formula: see text]h[Formula: see text] over scalp subjects. The results indicate that FRCNN enables the convenience of epilepsy treatments while preserving a high degree of precision. In the end, a detailed comparison with the previous methods demonstrates that FRCNN has achieved higher performance and generalization ability.

Locomotion is one of the most important problems in virtual reality. Real walking experience is the key to immersively explore the virtual world. Several strategies have been proposed to solve the problem, but most are not suitable to solve the locomotion problem in Room-Scale VR. The omnidirectional treadmill is an effective way to provide a natural walking experience within the Room-Scale VR. This paper proposes a novel omnidirectional treadmill named HEX-CORE-PROTOTYPE (HCP). The principle of synthesis and decomposition of velocity is applied to form an omnidirectional velocity field. Our system could provide a full degree of freedom and real walking experience in place. Compared to the current best system, the height of HCP is only 40% of it. The application shows the effectiveness of our system to solve the locomotion problem in Room-Scale VR.

As a key component of human-computer intelligent interaction and many real-world applications, the real-time property of facial expression recognition is especially important. However, the recognition result of conventional video-based approaches can not be given until the entire video is finished. In this letter, we deal with early expression detection, which aims to identify the expression as early as possible before its ending. This is a relatively new and challenging problem. Max-margin early event detector (MMED) is a well-known framework, which can make early detection. However, the linearity restricts its applications. We thus introduce kernel learning to model the nonlinear structure of complex data distribution. Moreover, the model is further reformulated in an online setting to address the streaming videos. The high retraining cost and large memory requirement of MMED are thus significantly reduced. In addition, we employ AlexNet architecture to make further comparison with mid-level features. Experiments on two popular video-based expression datasets demonstrate both the effectiveness and efficiency of the proposed method.

The multilayer perceptrons (MLPs) are widely used in many fields, however, singularities in the parameter space may seriously influence the learning dynamics of MLPs and cause strange learning behaviors. Given that the singularities are the subspaces of the parameter space where the Fisher information matrix (FIM) degenerates, the FIM plays a key role in the study of the singular learning dynamics of the MLPs. In this paper, we obtain the analytical form of the FIM for unipolar activation function-based MLPs where the input subjects to the Gaussian distribution with general covariance matrix and the unipolar error function is chosen as the activation function. Then three simulation experiments are taken to verify the validity of the obtained results.

Video-based facial expression recognition has received substantial attention over the past decade, while early expression detection (EED) is still a relatively new and challenging problem. The goal of EED is to identify an expression as quickly as possible after the expression starts and before it ends. This timely ability has many potential applications, ranging from human-computer interaction to security. The max-margin early event detector (MMED) is a well-known ranking model for early event detection. It can achieve competitive EED performance but suffers from several critical limitations: 1) MMED lacks flexibility in extracting useful information for segment comparison, which leads to poor performance in exploring the ranking relation between segment pairs; 2) the training process is slow due to the large number of constraints, and the memory requirement is also usually hard to satisfy; and 3) MMED is linear in nature, and hence may not be appropriate for data in a nonlinear feature space. To overcome these limitations, we propose an online multi-instance learning (MIL) framework for EED. In particular, the MIL technique is first introduced to generalize MMED, resulting in the proposed MIL-based EED (MIED), which is more general and flexible than MMED, since various instance construction and combination strategies can be adopted. To accelerate the training process, we reformulate MIED in the online setting and develop online multi-instance learning framework for EED (OMIED). To further exploit the nonlinear structure of the data distribution, we incorporate the kernel methods in OMIED, which results in the proposed online kernel multi-instance learning for early expression detection. Experiments on two popular and one challenging video-based expression data sets demonstrate both the efficiency and effectiveness of the proposed methods.

The existence of singularities often affects the learning dynamics in feedforward neural networks. In this paper, based on theoretical analysis results, we numerically analyze the learning dynamics of radial basis function (RBF) networks near singularities to understand to what extent singularities influence the learning dynamics. First, we show the explicit expression of the Fisher information matrix for RBF networks. Second, we demonstrate through numerical simulations that the singularities have a significant impact on the learning dynamics of RBF networks. Our results show that overlap singularities mainly have influence on the low dimensional RBF networks and elimination singularities have a more significant impact to the learning processes than overlap singularities in both low and high dimensional RBF networks, whereas the plateau phenomena are mainly caused by the elimination singularities. The results can also be the foundation to investigate the singular learning dynamics in deep feedforward neural networks.

Video-based facial expression recognition (FER) has recently received increased attention as a result of its widespread application. Using only one type of feature to describe facial expression in video sequences is often inadequate, because the information available is very complex. With the emergence of different features to represent different properties of facial expressions in videos, an appropriate combination of these features becomes an important, yet challenging, problem. Considering that the dimensionality of these features is usually high, we thus introduce multiview dimension reduction (MVDR) into video-based FER. In MVDR, it is critical to explore the relationships between and within different feature views. To achieve this goal, we propose a novel framework of MVDR by enforcing joint structured sparsity at both inter- and intraview levels. In this way, correlations on and between the feature spaces of different views tend to be well-exploited. In addition, a transformation matrix is learned for each view to discover the patterns contained in the original features, so that the different views are comparable in finding a common representation. The model can be not only performed in an unsupervised manner, but also easily extended to a semisupervised setting by incorporating some domain knowledge. An alternating algorithm is developed for problem optimization, and each subproblem can be efficiently solved. Experiments on two challenging video-based FER datasets demonstrate the effectiveness of the proposed framework.

Reliable and accurate short-term wind speed forecasting is of great importance for secure power system operations. In this study, a novel two-step method to construct a multi-model ensemble, which consists of linear regression, multi-layer perceptrons and support vector machines, is proposed. The ensemble members first compete with each other in a number of training rounds, and the one with the best forecasting accuracy in each round is recorded. Then, after all the training rounds, the occurrence frequency of each member is calculated and used as the weight to form the final multi-model ensemble. The effectiveness of the proposed multi-model ensemble has been assessed on the real datasets collected from three wind farms in China. The experimental results indicate that the proposed ensemble is capable of providing better performance than the single predictive models composing it.

Radial basis function (RBF) networks are one of the most widely used models for function approximation and classification. There are many strange behaviors in the learning process of RBF networks, such as slow learning speed and the existence of the plateaus. The natural gradient learning method can overcome these disadvantages effectively. It can accelerate the dynamics of learning and avoid plateaus. In this letter, we assume that the probability density function (pdf) of the input and the activation function are gaussian. First, we introduce natural gradient learning to the RBF networks and give the explicit forms of the Fisher information matrix and its inverse. Second, since it is difficult to calculate the Fisher information matrix and its inverse when the numbers of the hidden units and the dimensions of the input are large, we introduce the adaptive method to the natural gradient learning algorithms. Finally, we give an explicit form of the adaptive natural gradient learning algorithm and compare it to the conventional gradient descent method. Simulations show that the proposed adaptive natural gradient method, which can avoid the plateaus effectively, has a good performance when RBF networks are used for nonlinear functions approximation.

We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the universal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depending on the stability of the singularity and the initial parameters of learning. The results presented in this letter are universally applicable to a wide class of hierarchical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate work.

It has long been observed that strange behaviors happen in the gradient learning process of neural networks including multilayer perceptrons (MLPs) and RBF networks because of the singularities arisen from the symmetric structure in these models. The learning behaviors nearby are crucially dependant on the stability of the singularity. For RBF networks, this paper analyzes the stability by investigating the eigenvalues of the Hessian matrix on the overlap singularities. We show that the overlap singularity is a partially stable critical line, and there is only one nonzero eigenvalue on the singularity. The influence of the teacher parameters and initial conditions on eigenvalues is also discussed.

It has been found that strange behaviours will happen because of the singularity in the parameter space (or neuro-manifold) of hierarchical models such as feed-forward neural networks, and the learning dynamics of multilayer perceptrons near the singularity has been well discussed. In this paper, the online learning dynamics near the singularity is investigated for Radial Basis Function (RBF) Neural Networks with all its unit centers, widths and output weights being continuously modified using standard gradient descent algorithm. Results show that in the case of the teacher is on the singularity, if we initiate the learning process near the singularity, then the final parameter values of hidden units are dependant on their initial values: if two hidden units are initialized with similar unit centres and widths, they will overlap; otherwise, one of the hidden units will eliminate.