Brain decoding currently faces significant challenges in individual differences, modality alignment, and high-dimensional embeddings. To address individual differences, researchers often use source subject data, which leads to issues such as privacy leakage and heavy data storage burdens. In modality alignment, current works focus on aligning the softmax probability distribution but neglect the alignment of marginal probability distributions, resulting in modality misalignment. Additionally, images and text are aligned separately with fMRI without considering the complex interplay between images and text, leading to poor image reconstruction. Finally, the enormous dimensionality of CLIP embeddings causes significant computational costs. Although the dimensionality of CLIP embeddings can be reduced by ignoring the number of patches obtained from images and the number of tokens acquired from text, this comes at the cost of a significant drop in model performance, creating a dilemma. To overcome these limitations, we propose a source-free domain adaptation-based brain decoding framework. Firstly, we apply source-free domain adaptation, which only acquires the source model without accessing source data during target model adaptation, to brain decoding to address cross-subject variations, privacy concerns, and the heavy burden of data storage. Secondly, we employ maximum mean discrepancy (MMD) to align the marginal probability distributions between embeddings of different modalities. Moreover, to accommodate the complex interplay between image and text, we concatenate the embeddings of image and text and then use singular value decomposition (SVD) to obtain a new embedding. What’s more, to achieve better image generation quality, we employ the Wasserstein distance (WD) to align the probability distributions of new embeddings. Finally, in the target model adaptation phase of source-free domain adaptation, we employ low-rank adaptation (LoRA) to reduce the high expense of tuning the target model. Sufficient experiments demonstrate our work outperforms state-of-the-art methods for brain decoding tasks.

Node classification is a fundamental task in hypergraph learning. Existing methods generally assume that there are a few labeled nodes given in advance. However, in a newly formed hypergraph, collecting label information is challenging and costly in practice. Besides, current approaches mainly exploit the local consistency relationship, i.e., direct neighborhood information, while ignoring the high-order consistency relationship, i.e., high-order proximity information, limiting the discrimination of the latent representations. To address these issues, we propose leveraging knowledge from an auxiliary well-labeled hypergraph (source hypergraph) to assist the learning tasks in the target hypergraph, thus studying the cross-hypergraph node classification problem. Specifically, we propose a model, namely Local and High-order Consistency Coding and Adaptation (LHCCA), which learns both discriminative and transferable node representations. On the one hand, for each hypergraph, by exploiting the local and high-order consistency relationships, LHCCA obtains two kinds of representations, which are then coded by an attention mechanism to achieve a unified representation. On the other hand, the coded source and target node representations are enforced adversarial domain adaptation and contrastive learning to discover transferable features for adaptation. Furthermore, we derive theoretical analyses to establish desirable properties of the proposed model. Extensive experiments on several real-world datasets are conducted, and the promising results demonstrate the effectiveness of the proposed model.

Electroencephalogram (EEG)-based motor imagery (MI) signals are frequently used in brain-computer interfaces (BCIs) due to their wide applications in the rehabilitation field. However, cross-subject variations often result in a model trained on one participant failing when applied to another. Additionally, privacy concerns regarding sensitive health and mental information in EEG-based MI signals further complicate the situation. Source-free domain adaptation aims to address these cross-subject variations by transferring knowledge from a source domain (i.e., a previous participant) to a target domain (i.e., a new participant) without accessing sensitive source data. However, source-free unsupervised domain adaptation models often face issues with incorrect pseudo-labels, which can lead to unstable and ineffective adaptation. To address this, we propose a source-free semi-supervised domain adaptation algorithm for EEG-based MI signal classification. This algorithm tackles noise accumulation caused by incorrect pseudo-labels while effectively handling data distribution variations and privacy concerns, similar to source-free unsupervised domain adaptation models. Specifically, we train the classifier head using only a limited amount of labeled target data to prevent noise accumulation, and generate pseudo-labels for the unlabeled target data. Furthermore, we introduce an independent self-training head that learns better representations using the generated pseudo-labels, mitigating overfitting caused by the limited labeled target data. Additionally, we design an adversarial head that plays a minimax game to extract more discriminative feature representations from the unlabeled target data. Extensive experiments on three benchmark datasets, compared with eighteen state-of-the-art SFDA methods, demonstrate the superiority of our approach.

For electroencephalogram (EEG)-based emotion recognition, various EEG features are extracted from frequency, time, and time-frequency domains for modeling. Nevertheless, there is no a standard subset of EEG features widely accepted in this research field, giving rise to the challenge of curse dimensionality. To cope with the challenge, many EEG feature selection (FS) methods have been put forward based on information theory. Generally, these methods suffer from the issue of delivering a suboptimal result with heuristic search, and they are also inefficient in balancing the influence of different terms like feature relevance and feature redundancy. Based on this, we present a new fuzzy information-theoretic based optimization approach to attain the goal. To be specific, fuzzy mutual information is unitized to evaluate EEG features from the relevance and redundancy perspective, and data structure information is captured to exploit feature manifold simultaneously. Then, a unified optimization framework is designed to take all of them into consideration, thereby inducing a globally optimal result of EEG FS. Extensive empirical studies on three EEG emotional datasets reveal that our method is able to found out a discriminative and non-redundant feature subset from different domains, and therefore achieves the performance improvement of emotion recognition.

Semi-supervised multi-label learning (SSMLL) involves learning a multi-label classifier from a small set of labeled data and a large set of unlabeled data. Label enhancement (LE), accounting for the relative importance of labels, has been effective in improving the performance of supervised multi-label learning models. Nevertheless, generating a robust SSMLL model with LE based on incomplete label information remains challenging. In this paper, we pioneer the idea of applying LE to SSMLL. First, we design a kNN aggregation-based method, aiming to assign pseudo-labels to unlabeled data and perform the LE process by aggregating label information from neighboring instances. Leveraging the topological structure of the feature space is an effective LE approach for training. However, LE, decoupled from the training process, lacks the dynamic feedback of the training model. To improve this, we incorporate a label propagation mechanism that iteratively optimizes the LE process with the guidance of the available label information. Moreover, we consider local label correlations according to local linear embedding to further enhance the generalization ability of the learning model. Extensive experiments demonstrate that the proposed approach can effectively recover latent label information, resulting in significant performance improvement in SSMLL.

In recommendation systems, cold-start user recommendation is a challenging problem, where precise recommendations are required for users who have not appeared before. Several existing cold-start user recommendation models adopt domain adaptation to extract information from auxiliary source domains to assist the recommendations on the target domain. In this article, we propose that the cold-start user recommendation problem can be formulated by the heterogeneous domain adaption approach. We determine a transformation of user features, e.g., user social relations and historical interactions between warm users and their interested items, into a latent space so that the loss function is set by user feature reconstruction and by feature and distribution matching in the heterogeneous domains. The resulting optimization problem can be solved by matrix eigendecomposition, and the cold-start users’ preferences can thus be obtained. We also extend the proposed model using neural networks. We perform extensive experiments on several real-world datasets, and the results in terms of Precision, Recall, NDCG, and Hit Rate verify the effectiveness of the proposed model.

Graph-structured data, where nodes exhibit either pair-wise or high-order relations, are ubiquitous and essential in graph learning. Despite the great achievement made by existing graph learning models, these models use the direct information (edges or hyperedges) from graphs and do not adopt the underlying indirect information (hidden pair-wise or high-order relations). To address this issue, in this paper, we propose a general framework named Simplicial Complex Neural (SCN) network, in which we construct a simplicial complex based on the direct and indirect graph information from a graph so that all information can be employed in the complex network learning. Specifically, we learn representations of simplices by aggregating and integrating information from all the simplices together via layer-by-layer simplicial complex propagation. In consequence, the representations of nodes, edges, and other high-order simplices are obtained simultaneously and can be used for learning purposes. By making use of block matrix properties, we derive the theoretical bound of the simplicial complex filter learnt by the propagation and establish the generalization error bound of the proposed simplicial complex network. We perform extensive experiments on node (0-simplex), edge (1-simplex), and triangle (2-simplex) classifications, and promising results demonstrate the performance of the proposed method is better than that of existing graph and hypergraph network approaches.

Information theoretical-based methods have attracted a great attention in recent years and gained promising results for multilabel feature selection (MLFS). Nevertheless, most of the existing methods consider a heuristic way to the grid search of important features, and they may also suffer from the issue of fully utilizing labeling information. Thus, they are probable to deliver a suboptimal result with heavy computational burden. In this article, we propose a general optimization framework global relevance and redundancy optimization (GRRO) to solve the learning problem. The main technical contribution in GRRO is a formulation for MLFS while feature relevance, label relevance (i.e., label correlation), and feature redundancy are taken into account, which can avoid repetitive entropy calculations to obtain a global optimal solution efficiently. To further improve the efficiency, we extend GRRO to filter out inessential labels and features, thus facilitating fast MLFS. We call the extension as GRROfast, in which the key insights are twofold: 1) promising labels and related relevant features are investigated to reduce ineffective calculations in terms of features, even labels and 2) the framework of GRRO is reconstructed to generate the optimal result with an ensemble. Moreover, our proposed algorithms have an excellent mechanism for exploiting the inherent properties of multilabel data; specifically, we provide a formulation to enhance the proposal with label-specific features. Extensive experiments clearly reveal the effectiveness and efficiency of our proposed algorithms.

In recommendation systems, the cold-start issue is a long-standing problem where no historical interaction records are given for certain users or items. Under this circumstance, recommendations for new users or new items become challenging. To address this problem, most existing approaches seek to discover a latent common space for users and items. However, these methods require a strong assumption that a shared space exists where the distributions of users and items are identical, which may limit the recommendation performance. In this article, we propose a novel model called Feature Matching Machine (FMM) to learn latent informative user and item representations. Different from previous methods, for warm users (or items), FMM learns two kinds of latent features, i.e., one is constructed by a hypergraph auto-encoder based on historical interactions between users and items, and the other is built by a multi-layer perceptron based on users (or items). Subsequently, FMM matches these two latent feature representations so as to discover the relationships across users (or items) and cold-start items (or users). We conduct extensive experiments on several real-world datasets and compare the proposed method with well-known baseline methods. Promising results demonstrate the effectiveness and efficiency of the proposed model.

In recent years, voice-interaction-based control systems have attracted considerable attention for industrial control systems implementing Industrial Internet of Things (IIoT) technologies. The development of automated semantic understanding relates to the industrial Internet equipment used to realize remote voice control as well as to its intelligent management and control. In these emerging voice-interaction-enabled industrial central control systems, sorting technologies are considered critical. For complex user questions, the level of satisfaction regarding the answers given by such systems tends to be low. Driven by these challenges and opportunities, the optimization of conventional retrieval-based question answering through deep learning methods has become popular. In this study, we propose three deep semantic sorting models based on deep learning, including a multilayer convolutional matching sorting model for single documents and two interactive pairwise bidirectional encoder representations from transformers (BERT) sorting models for document pairs. Two main network architectures are proposed to model document pairs, named Pairwise-Twin-BERT and Pairwise-Triple-BERT. Experimental results indicate that proposed models performed better than state-of-the-art methods based on text matching in a candidate document sorting task.

Class imbalance is a common issue in the community of machine learning and data mining. The class-imbalance distribution can make most classical classification algorithms neglect the significance of the minority class and tend toward the majority class. In this article, we propose a label enhancement method to solve the class-imbalance problem in a graph manner, which estimates the numerical label and trains the inductive model simultaneously. It gives a new perspective on the class-imbalance learning based on the numerical label rather than the original logical label. We also present an iterative optimization algorithm and analyze the computation complexity and its convergence. To demonstrate the superiority of the proposed method, several single-label and multilabel datasets are applied in the experiments. The experimental results show that the proposed method achieves a promising performance and outperforms some state-of-the-art single-label and multilabel class-imbalance learning methods.

This article presents a novel model named Adversarial Auto-encoder Domain Adaptation to handle the recommendation problem under cold-start settings. Specifically, we divide the hypergraph into two hypergraphs, i.e., a positive hypergraph and a negative one. Below, we adopt the cold-start user recommendation for illustration. After achieving positive and negative hypergraphs, we apply hypergraph auto-encoders to them to obtain positive and negative embeddings of warm users and items. Additionally, we employ a multi-layer perceptron to get warm and cold-start user embeddings called regular embeddings. Subsequently, for warm users, we assign positive and negative pseudo-labels to their positive and negative embeddings, respectively, and treat their positive and regular embeddings as the source and target domain data, respectively. Then, we develop a matching discriminator to jointly minimize the classification loss of the positive and negative warm user embeddings and the distribution gap between the positive and regular warm user embeddings. In this way, warm users’ positive and regular embeddings are connected. Since the positive hypergraph maintains the relations between positive warm user and item embeddings, and the regular warm and cold-start user embeddings follow a similar distribution, the regular cold-start user embedding and positive item embedding are bridged to discover their relationship. The proposed model can be easily extended to handle the cold-start item recommendation by changing inputs. We perform extensive experiments on real-world datasets for both cold-start user and cold-start item recommendations. Promising results in terms of precision, recall, normalized discounted cumulative gain, and hit rate verify the effectiveness of the proposed method.

Recommendation systems provide personalized service to users and aim at suggesting to them items that they may prefer. There is an increasing requirement of next-item recommendation systems to infer a user's next favor item based on his/her historical selection of items. In this article, we study the next-item recommendation under the cold-start situation, where the users in the system share no interaction with the new items. Specifically, we seek to address the problem from the perspective of zero-shot learning (ZSL), which classifies samples whose classes are unseen during training. To this end, we crystallize the relationship and setting from ZSL to cold-start next-item recommendation, and further propose a novel model called User-Item Matching and Auto-encoders (UIMA) which learns the latent embeddings for both users and items by exploiting user historical preferences and item attributes. Concretely, UIMA consists of three components, i.e., two auto-encoders for learning user and item embeddings and a matching network to explore the relationship between the learned user and item embeddings. We perform experiments on several cold-start next-item recommendation datasets, including movies, music, and bookmarks. Promising results demonstrate the effectiveness of the proposed method for cold-start next-item recommendation.

Provision of the perception of pseudo-weight through tracking offsets in virtual reality (VR) allows users to estimate the weight of virtual objects. However, the contribution of the user's real-time perception to such illusions is unknown. Here, we focus on this issue using a brain-computer interface (BCI), through which the user's perception of the weight of virtual objects can be detected in real-time and used to adjust the tracking offset in a closed loop. We first trained a computational model with electroencephalography (EEG) data by asking users to imagine lifting a heavy or a light ball. With this model, the user's perception of the object weight could be detected through the BCI in real-time to adjust the tracking offset, thereby enabling further generation of a more realistic visual sensation. Then, we evaluated the effects of the BCI tracking offset on the perception of the weights of three virtual objects used to simulate real objects, namely, tennis, billiard, and bowling balls. Our results showed that the BCI tracking offset could assist participants in generating perceived weights for virtual objects in VR. We further showed that our approach can provide weight perception through real-time adjustment of the tracking offset, which might be useful for new virtual objects that appear suddenly in the virtual environment. Additionally, most participants (78%) preferred this BCI tracking offset system for weight perception. This article provides the first quantification of weight perception for a virtual object with a BCI that can be used to adjust the tracking offset in real-time for pseudo-weight perception in VR.

Deep learning-based auto-driving systems are vulnerable to adversarial examples attacks which may result in wrong decision making and accidents. An adversarial example can fool the well trained neural networks by adding barely imperceptible perturbations to clean data. In this paper, we explore the mechanism of adversarial examples and adversarial robustness from the perspective of statistical mechanics, and propose an statistical mechanics-based interpretation model of adversarial robustness. The state transition caused by adversarial training based on the theory of fluctuation dissipation disequilibrium in statistical mechanics is formally constructed. Besides, we fully study the adversarial example attacks and training process on system robustness, including the influence of different training processes on network robustness. Our work is helpful to understand and explain the adversarial examples problems and improve the robustness of deep learning-based auto-driving systems.

Physical practice with one hand results in performance gains of the other (un-practiced) hand in a unilateral motor task. Yet how it induces performance gains of interlimb coordination in the bimanual movements between trained limb and the opposite, untrained limb is unclear. The present study designed a game-like interactive system for physical practice, in which an avatar’s hands could be controlled itself or by the subject during a bimanual movement task in an immersive virtual reality environment. Participants practiced with the bimanual task by simultaneously drawing non-symmetric three-sided squares (e.g., U and C) to learn limb coordination with the following training strategies: (1) performing and seeing a bimanual task (BH-BH); (2) performing a unimanual task with right hand and seeing a bimanual action (RH-BH); (3) not performing a task but seeing a bimanual action (noH-BH); (4) performing and seeing a unimanual task (RH-RH). We found that the learning performance was better after BH-BH and RH-BH compared with other training strategies. In addition, we examined the effects of virtual hand representations on the learning performance after RH-BH. We found that the performance after training was increased with the realism level of virtual hands. These findings suggest that the proposed approach of RH-BH with realistic virtual hand would result in transfer of coordination skill to the unpracticed hand, which puts forward a new approach for learning and rehabilitation of coordination skill in patients with unilateral motor deficit in immersive environments.

Despite rapid advances in the study of brain–computer interfaces (BCIs) in recent decades, two fundamental challenges, namely, improvement of target detection performance and multidimensional control, continue to be major barriers for further development and applications. In this paper, we review the recent progress in multimodal BCIs (also called hybrid BCIs), which may provide potential solutions for addressing these challenges. In particular, improved target detection can be achieved by developing multimodal BCIs that utilize multiple brain patterns, multimodal signals, or multisensory stimuli. Furthermore, multidimensional object control can be accomplished by generating multiple control signals from different brain patterns or signal modalities. Here, we highlight several representative multimodal BCI systems by analyzing their paradigm designs, detection/control methods, and experimental results. To demonstrate their practicality, we report several initial clinical applications of these multimodal BCI systems, including awareness evaluation/detection in patients with disorder of consciousness (DOC). As an evolving research area, the study of multimodal BCIs is increasingly requiring more synergetic efforts from multiple disciplines for the exploration of the underlying brain mechanisms, the design of new effective paradigms and means of neurofeedback, and the expansion of the clinical applications of these systems.