Deep learning-based methods have achieved remarkable success in brain-computer interfaces (BCIs). However, its inherent assumption of independent and identically distributed (i.i.d.) data renders it vulnerable to out-of-distribution (OOD) scenarios. To address this limitation, the present study proposed a causality-driven convolutional manifold attention network (CD-CMAN) that learned invariant representations from electroencephalogram (EEG) signals to enhance OOD generalization. The framework began with a spatiotemporal convolution module to extract rich temporal and spatial features. Guided by the defined structural causal model and leveraging the strengths of Riemannian geometry and deep learning, dual latent encoders with manifold attention units were crafted to explicitly separate spatiotemporal feature maps into semantic and variation latent factors. A reconstruction module with a dedicated loss was implemented to ensure these factors retaining informative, while the Hilbert-Schmidt independence criterion (HSIC) was introduced to enforce their statistical independence. Further, a variational information bottleneck and gradient reversal layer were incorporated to compress and disentangle the semantic and variation factors. Evaluations on two public datasets under both subject-dependent and subject-independent settings demonstrated that CD-CMAN consistently outperforms comparative baselines. These findings suggest that the proposed model could provide a new solution for the practical application of BCI technology.

Vision Graph Neural Network (ViG) is designed to recognize different objects through graph-level processing. However, ViG constructs graphs with appearance-level neighbors and neglects the category semantic. The oversight results in the unintentional connection of patches that belong to different objects, thus affecting the distinctiveness of categories in multi-label medical image learning. Since the pixel-level annotations for images are not easily available, category-aware graphs can not be directly built. To solve this problem, we consider localizing category-specific regions using Class Activation Maps (CAMs), an effective way to highlight regions belonging to each category without requiring manual annotations. Specifically, we propose a CAM-interacted Vision GNN (CiV-GNN), in which category-aware graphs are formed to perform intra-category graph processing. CIV-GNN includes a Class-activated Patch Division (CAPD) module, which introduces CAMs as guidance for category-aware graph building. Furthermore, we develop a Multi-graph Interactive Processing (MIP) module to model the relations between category-aware graphs, promoting inter-category interaction learning. Experimental results show that CiV-GNN performs well in surgical tool localization and multi-label medical image classification. Specifically, for m2cai16-localization, CiV-GNN exhibits a 1.43% and 7.02% improvement in mAP50 and mAP50-95, respectively, compared to YOLOv8.

The burgeoning field of video-text retrieval has witnessed significant advancements with the advent of deep learning. However, understanding and matching textual descriptions and video data remains a formidable challenge due to the large information gap across textual and video modalities. As observed, the caption of a video is commonly under-described, lacking expressions of minor characters or local details. Some recent advances have attempted to leverage multimodal Large Language Model (mLLM) to bridge the comprehension gap. However, mLLMs’ potential in enhancing video-text retrieval (VTR) is understudied. This paper aims to fill this research vacancy, analyzing the practical significance and model preferences for utilizing mLLMs in VTR enhancement, as well as investigating the effective integration of mLLM-derived information into the retrieval learning. Based on our analytical insights, we innovatively propose treating mLLM as caption supplements rather than substitutes to bridge the expression gap across modalities. To achieve better cross-modal alignment, we systematically generate diverse variations of videos to construct an elastic visual space. By treating mLLM-supplemented captions as out-of-space points, cross-modal representation learning is accomplished through the optimization of a conical-like representation space. Our model achieves state-of-the-art results on various benchmarks, including MSR-VTT, MSVD, and DiDeMo, and analytical experiments suggest appropriate prompt proposals and indicate our method’s robustness to different mLLMs.

Text-to-video retrieval (T2VR) aims to identify the most semantically relevant video based on a text query. Text queries typically involve diverse visual elements and events in video, making it non-trivial to learn a robust video feature representation for different queries. An abundance of spatial information make the model overwhelmed by redundancy and noisy and struggle to focus on linchpin visual elements. Additionally, without effective guidance, models grapple with connecting temporal information across different frames. In this paper, we introduce a Spatial-Temporal Pooling (STP) method to cohesively capture and unify the inherent spatio-temporal context within videos. For spatial information, STP leverages spatial tags such as entities, scenes, and text as attention prompts, steering the model toward salient visual elements while mitigating the impact of redundancies and noise. For temporal information, STP adopts video narratives summarized in captions as temporal prompts to enhance the model’s perception of events. Experimental results show that our approach has achieved improvements on the MSRVTT(1.9%), MSVD(1.2%), and VATEX(0.7%) datasets compared to the SOTA methods.

The neural activity patterns of localized brain regions are crucial for recognizing brain intentions. However, existing electroencephalogram (EEG) decoding models, especially those based on deep learning, predominantly focus on global spatial features, neglecting valuable local information, potentially leading to suboptimal performance. Therefore, this study proposed a dynamic hierarchical convolutional attention network (DH-CAN) that comprehensively learned discriminative information from both global and local spatial domains, as well as from time-frequency domains in EEG signals. Specifically, a multiscale convolutional block was designed to dynamically capture time-frequency information. The channels of EEG signals were mapped to different brain regions based on motor imagery neural activity patterns. The spatial features, both global and local, were then hierarchically extracted to fully exploit the discriminative information. Furthermore, regional connectivity was established using a graph attention network, incorporating it into the local spatial features. Particularly, this study shared network parameters between symmetrical brain regions to better capture asymmetrical motor imagery patterns. Finally, the learned multilevel features were integrated through a high-level fusion layer. Extensive experimental results on two datasets demonstrated that the proposed model performed excellently across multiple evaluation metrics, exceeding existing benchmark methods. These findings suggested that the proposed model offered a novel perspective for EEG decoding research.

This study investigates the trajectory optimization and high-precision motion control of excavators based on an independent metering hydraulic system. Considering both operational efficiency and motion smoothness, we propose a motion control method for excavator manipulators based on time-energy-jerk integrated optimal trajectory planning. The nondominated sorting genetic algorithm II (NSGA-II) algorithm is used to optimize interpolated trajectory based on five-time B-splines in the joint space. To ensure that excavators can accurately execute the planned optimal trajectory, the corresponding arms must be controlled with high precision. The oil inlet flow and the oil return pressure controllers are designed based on the independent metering hydraulic system. The flow controller is designed based on time-logarithmic barrier Lyapunov function to determine the virtual control rate and uses the Levant filter for filtering. The corresponding error transformations are employed to avoid the problem of the explosion of complexity in the traditional backstepping controller designs while ensuring that transient behavior of system tracking errors remains within specified boundaries. The uncertain components and nonlinear functions in the manipulator system are approximated by neural network (NN). Additionally, the pressure controller is used to keep the oil return pressure low to reduce system’s energy consumption. Finally, comparative simulations are conducted to verify the superiority of the proposed controller.

K-means is a fundamental clustering algorithm widely used in both academic and industrial applications. Its popularity can be attributed to its simplicity and efficiency. Studies show the equivalence of K-means to principal component analysis, non-negative matrix factorization, and spectral clustering. However, these studies focus on standard K-means with squared euclidean distance. In this review paper, we unify the available approaches in generalizing K-means to solve challenging and complex problems. We show that these generalizations can be seen from four aspects: data representation, distance measure, label assignment, and centroid updating. As concrete applications of transforming problems into modified K-means formulation, we review the following applications: iterative subspace projection and clustering, consensus clustering, constrained clustering, domain adaptation, and outlier detection.

Endobronchial ultrasound (EBUS) elastography videos have shown great potential to supplement intrathoracic lymph node diagnosis. However, it is laborious and subjective for the specialists to select the representative frames from the tedious videos and make a diagnosis, and there lacks a framework for automatic representative frame selection and diagnosis. To this end, we propose a novel deep learning framework that achieves reliable diagnosis by explicitly selecting sparse representative frames and guaranteeing the invariance of diagnostic results to the permutations of video frames. Specifically, we develop a differentiable sparse graph attention mechanism that jointly considers frame-level features and the interactions across frames to select sparse representative frames and exclude disturbed frames. Furthermore, instead of adopting deep learning-based frame-level features, we introduce the normalized color histogram that considers the domain knowledge of EBUS elastography images and achieves superior performance. To our best knowledge, the proposed framework is the first to simultaneously achieve automatic representative frame selection and diagnosis with EBUS elastography videos. Experimental results demonstrate that it achieves an average accuracy of 81.29% and area under the receiver operating characteristic curve (AUC) of 0.8749 on the collected dataset of 727 EBUS elastography videos, which is comparable to the performance of the expert-based clinical methods based on manually-selected representative frames.

Name ambiguity issue that results from the similarity of many common Chinese names. With the development of artificial intelligence, the disambiguation model based on machine learning has achieved better disambiguation effects and has been widely used in various universities. However, continually improving the disambiguation effect remains a major challenge. Smart campuses based on the Internet of Things are developing rapidly, and a large number of discretely distributed web tables that omit data values exist. However, the usable attributes of the disambiguation model are limited. To overcome these challenges, this study proposes a name disambiguation model of web tables from data integration (NDWT) in smart campuses. The model first recognises the label mapping in a webpage table using four types of label matchers and then designs the instance comparator based on the obtained label mapping. The web tables are integrated according to the instance mapping relationship, and two datasets, one before (BWT) and the other after (A WT) integration, are obtained. Relevant features are subsequently extracted from these two datasets and trained. Finally, the NDWT model is used for disambiguation experiments. Comparative experiments, condu-cted using seven different types of ML models, show that the NDWT model improves significantly after the integration of web tables; in particular, the pairwise F1 of the K-means model increases by 43.23%. The pairwise F1 of the remaining models increases by approximately 10%. The experimental evaluation proves the feasibility of the NDWT model proposed in this study. Confirming that it can achieve a higher distribution quality compared to conventional name disambiguation methods.

Generative Adversarial Networks (GAN) have many potential medical imaging applications, including data augmentation, domain adaptation, and model explanation. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models either cannot scale to high-resolution or are prone to patchy artifacts. In this work, we propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. During training, we adopt a hierarchical structure that simultaneously generates a low-resolution version of the image and a randomly selected sub-volume of the high-resolution image. The hierarchical design has two advantages: First, the memory demand for training on high-resolution images is amortized among sub-volumes. Furthermore, anchoring the high-resolution sub-volumes to a single low-resolution image ensures anatomical consistency between sub-volumes. During inference, our model can directly generate full high-resolution images. We also incorporate an encoder with a similar hierarchical structure into the model to extract features from the images. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation. We also demonstrate clinical applications of the proposed model in data augmentation and clinical-relevant feature extraction.

Extracting hidden phenotypes is essential in medical data analysis because it facilitates disease subtyping, diagnosis, and understanding of disease etiology. Since the hidden phenotype is usually a low-dimensional representation that comprehensively describes the disease, we require a dimensionality reduction method that captures as much disease-relevant information as possible. However, most unsupervised or self-supervised methods cannot achieve the goal because they learn a holistic representation containing both disease-relevant and disease-irrelevant information. Supervised methods can capture information that is predictive to the target clinical variable only, but the learned representation is usually not generalizable for the various aspects of the disease. Hence, we develop a dimensionality-reduction approach to extract Disease Relevant Features (DRFs) based on information theory. We propose to use clinical variables that weakly define the disease as so-called anchors. We derive a formulation that makes the DRF predictive of the anchors while forcing the remaining representation to be irrelevant to the anchors via adversarial regularization. We apply our method to a large-scale study of Chronic Obstructive Pulmonary Disease (COPD). Our experiment shows: (1) Learned DRFs are as predictive as the original representation in predicting the anchors, although it is in a significantly lower dimension. (2) Compared to supervised representation, the learned DRFs are more predictive to other relevant disease metrics that are not used during the training. (3) The learned DRFs are related to non-imaging biological measurements such as gene expressions, suggesting the DRFs include information related to the underlying biology of the disease.

Recently, researches related to unsupervised disentanglement learning with deep generative models have gained substantial popularity. However, without introducing supervision, there is no guarantee that the factors of interest can be successfully recovered (Locatello et al. 2018). Motivated by a real-world problem, we propose a setting where the user introduces weak supervision by providing similarities between instances based on a factor to be disentangled. The similarity is provided as either a binary (yes/no) or real-valued label describing whether a pair of instances are similar or not. We propose a new method for weakly supervised disentanglement of latent variables within the framework of Variational Autoencoder. Experimental results demonstrate that utilizing weak supervision improves the performance of the disentanglement method substantially.

The majority of state-of-the-art deep learning methods are discriminative approaches, which model the conditional distribution of labels given inputs features. The success of such approaches heavily depends on high-quality labeled instances, which are not easy to obtain, especially as the number of candidate classes increases. In this paper, we study the complementary learning problem. Unlike ordinary labels, complementary labels are easy to obtain because an annotator only needs to provide a yes/no answer to a randomly chosen candidate class for each instance. We propose a generative-discriminative complementary learning method that estimates the ordinary labels by modeling both the conditional (discriminative) and instance (generative) distributions. Our method, we call Complementary Conditional GAN (CCGAN), improves the accuracy of predicting ordinary labels and is able to generate high-quality instances in spite of weak supervision. In addition to the extensive empirical studies, we also theoretically show that our model can retrieve the true conditional distribution from the complementarily-labeled data.

Climate prediction is a very challenging problem. Many institutes around the world try to predict climate variables by building climate models called General Circulation Models (GCMs), which are based on mathematical equations that describe the physical processes. The prediction abilities of different GCMs may vary dramatically across different regions and time. Motivated by the need of identifying which GCMs are more useful for a particular region and time, we introduce a clustering model combining Dirichlet Process (DP) mixture of sparse linear regression with Markov Random Fields (MRFs). This model incorporates DP to automatically determine the number of clusters, imposes MRF constraints to guarantee spatio-temporal smoothness, and selects a subset of GCMs that are useful for prediction within each spatio-temporal cluster with a spike-and-slab prior. We derive an effective Gibbs sampling method for this model. Experimental results are provided for both synthetic and real-world climate data.

Crowdclustering is a practical way to incorporate domain knowledge into clustering, by combining opinions from multiple domain experts. Existing crowdclustering methods analyze binary pairwise similarity labels. However, in some applications, experts might provide partition labels. If we convert partition labels into pairwise similarity, then it would be difficult to understand the relationships between clustering solutions from different experts. In this paper, we propose a crowdclustering model that directly analyzes partition labels. The proposed model adopts a novel approach based on a modified multinomial logistic regression model, which simultaneously learns the number of clusters and determines hyper-planes that partition samples into clusters. The proposed model also learns a mapping between the latent clusters and expert labels, revealing the agreements and disagreements between experts. Experiments on benchmark data demonstrate that the proposed model simultaneously learns the number of clusters and discovers the clustering structure. An experiment on disease subtyping problem illustrates that the proposed model helps us understand the agreement and disagreement between experts.

Utilizing expert input often improves clustering performance. However in a knowledge discovery problem, ground truth is unknown even to an expert. Thus, instead of one expert, we solicit the opinion from multiple experts. The key question motivating this work is: which experts should be assigned higher weights when there is disagreement on whether to put a pair of samples in the same group? To model the uncertainty in constraints from different experts, we build a probabilistic model for pairwise constraints through jointly modeling each expert’s accuracy and the mapping from features to latent cluster assignments. After learning our probabilistic discriminative clustering model and accuracies of different experts, 1) samples that were not annotated by any expert can be clustered using the discriminative clustering model; and 2) experts with higher accuracies are automatically assigned higher weights in determining the latent cluster assignments. Experimental results on UCI benchmark datasets and a real-world disease subtyping dataset demonstrate that our proposed approach outperforms competing alternatives, including semi-crowdsourced clustering, semi-supervised clustering with constraints from majority voting, and consensus clustering.

Expert input can improve clustering performance. In today’s collaborative environment, the availability of crowdsourced multiple expert input is becoming common. Given multiple experts’ inputs, most existing approaches can only discover one clustering structure. However, data is multi-faced by nature and can be clustered in different ways (also known as views). In an exploratory analysis problem where ground truth is not known, different experts may have diverse views on how to cluster data. In this paper, we address the problem on how to automatically discover multiple ways to cluster data given potentially diverse inputs from multiple uncertain experts. We propose a novel Bayesian probabilistic model that automatically learns the multiple expert views and the clustering structure associated with each view. The benefits of learning the experts’ views include 1) enabling the discovery of multiple diverse clustering structures, and 2) improving the quality of clustering solution in each view by assigning higher weights to experts with higher confidence. In our approach, the expert views, multiple clustering structures and expert confidences are jointly learned via variational inference. Experimental results on synthetic datasets, benchmark datasets and a real-world disease subtyping problem show that our proposed approach outperforms competing baselines, including meta clustering, semi-supervised clustering, semi-crowdsourced clustering and consensus clustering.

Clustering is a challenging problem because given the same data set, it can be grouped in multiple different ways. Which of these clustering solutions is interesting depends on its domain application. Thus, incorporating domain expert input often improves clustering performance. However, most existing semi-supervised clustering techniques can only incorporate instance-level constraints (a few labels or must-link/cannot-link constraints), which domain experts may not be comfortable providing in knowledge discovery problems because categories are not known. Fortunately, domain experts often have an idea regarding properties that clustering solutions should have in order to be useful in domain application based on domain relevant scores. In this paper, we provide a framework for jointly optimizing the usefulness and quality of a clustering solution. Experiments on a synthetic data, a benchmark data, and a real-world disease subtyping problem demonstrate the usefulness of our proposed approach.

We introduce a novel Bayesian nonparametric model that uses the concept of disease trajectories for disease subtype identification. Although our model is general, we demonstrate that by treating fractions of tissue patterns derived from medical images as compositional data, our model can be applied to study distinct progression trends between population subgroups. Specifically, we apply our algorithm to quantitative emphysema measurements obtained from chest CT scans in the COPDGene Study and show several distinct progression patterns. As emphysema is one of the major components of chronic obstructive pulmonary disease (COPD), the third leading cause of death in the United States [1], an improved definition of emphysema and COPD subtypes is of great interest. We investigate several models with our algorithm, and show that one with age , pack years (a measure of cigarette exposure), and smoking status as predictors gives the best compromise between estimated predictive performance and model complexity. This model identified nine subtypes which showed significant associations to seven single nucleotide polymorphisms (SNPs) known to associate with COPD. Additionally, this model gives better predictive accuracy than multiple, multivariate ordinary least squares regression as demonstrated in a five-fold cross validation analysis. We view our subtyping algorithm as a contribution that can be applied to bridge the gap between CT-level assessment of tissue composition to population-level analysis of compositional trends that vary between disease subtypes.

Clustering is a technique that is usually applied as a tool for exploratory data analysis. Because of the exploratory nature of this task, it would be beneficial if a clustering method generates interpretable results, and allows incorporating domain knowledge. This motivates us to develop a probabilistic discriminative model that learns a rectangular decision rule for each cluster, we call Discriminative Rectangle Mixture (DReaM) model. DReaM gives interpretable clustering results, because the rectangular decision rules discovered explicitly illustrate how one cluster is defined and differs from other clusters. It also facilitates us to take advantage of existing rules because we can choose informative prior distributions for the rectangular rules. Moreover, DReaM allows that the features for generating rules do not have to be the same as the features for discovering cluster structure. We approximate the distribution for the rules discovered via variational inference. Experimental results demonstrate that DReaM gives more interpretable clustering results, and yet its performance is comparable to existing clustering methods when solving traditional clustering. Furthermore, in real applications, DReaM is able to effectively take advantage of domain knowledge, and to generate reasonable clustering results.

Clustering and ranking have been successfully applied independently to homogeneous information networks, containing only one type of objects. However, real-world information networks are oftentimes heterogeneous, containing multiple types of objects and links. Recent research has shown that clustering and ranking can actually mutually enhance each other, and several techniques have been developed to integrate clustering and ranking together on a heterogeneous information network. To the best our knowledge, however, all of such techniques assume the network follows a certain schema. In this paper, we propose a probabilistic generative model that simultaneously achieves clustering and ranking on a heterogeneous network that can follow arbitrary schema, where the edges from different types are sampled from a Poisson distribution with the parameters determined by the ranking scores of the nodes in each cluster. A variational Bayesian inference method is proposed to learn these parameters, which can be used to output ranking and clusters simultaneously. Our method is evaluated on both synthetic and real-world networks extracted from the DBLP and YELP data. Experimental results show that our method outperforms the state-of-the-art baselines.