Irregularly sampled time series (ISTS), characterized by non-uniform time intervals with natural missingness, are prevalent in real-world applications. Existing approaches for ISTS modeling primarily rely on observed values to impute unobserved ones or infer latent dynamics. However, these methods overlook a critical source of learning signal: the reconstruction error inherently produced during model training. Such error implicitly reflects how well a model captures the underlying data structure and can serve as an informative proxy for unobserved values. To exploit this insight, we propose iTimER, a simple yet effective self-supervised pre-training framework for ISTS representation learning. iTimER models the distribution of reconstruction errors over observed values and generates pseudo-observations for unobserved timestamps through a mixup strategy between sampled errors and the last available observations. This transforms unobserved timestamps into noise-aware training targets, enabling meaningful reconstruction signals. A Wasserstein metric aligns reconstruction error distributions between observed and pseudo-observed regions, while a contrastive learning objective enhances the discriminability of learned representations. Extensive experiments on classification, interpolation, and forecasting tasks demonstrate that iTimER consistently outperforms state-of-the-art methods under the ISTS setting.

Open-Set Domain Generalization (OSDG) aims to generalize over unseen target domains containing open classes, and the core challenge lies in identifying unknown samples never encountered during training. Recently, CLIP has exhibited impressive performance in OSDG, while it still falls into the dilemma between structural risk of known classes and open space risk from unknown classes, and easily suffers from over-confidence, especially when distinguishing known-like unknown samples. To this end, we propose a Semantic-enhanced CLIP (SeeCLIP) framework that leverages fine-grained semantics to boost unknown detection, so as to accommodate both risks and enable precise discrimination among categories. In SeeCLIP, we propose a semantic-aware prompt enhancement module to extract fine-grained key semantic features, and establish a fine-grained vision-language alignment. Duplex contrastive learning is proposed for prompt learning, which jointly optimizes duplex losses such that the unknown prompt is similar to known prompts, yet exhibits key semantic differences. We also design a semantic-guided diffusion module to enable nuanced capture in generation. By injecting perturbed key semantics into a diffusion model as control conditions, it generates the closest unknowns or pseudo-open samples with high similarity yet low belongingness to known classes. We formulate a generalization bound for OSDG, and show that SeeCLIP can achieve a lower generalization risk. Extensive experiments on benchmark datasets validate the superiority of SeeCLIP, it outperforms the SOTA methods by nearly 3% on accuracy and 5% on H-index, respectively.

Long-tail recognition remains challenging for pre-trained foundation models like CLIP, which often suffer from performance degradation under imbalanced data. This stems not only from the overfitting/underfitting issues during fine-tuning but, more fundamentally, from the inherent bias inherited from the long-tail distribution of their massive pre-training datasets. To address this, we propose HGLTR (Hierarchy-Guided Long-Tail Recognition), a novel framework that calibrates pre-trained models by injecting objective class hierarchy knowledge. We argue that the semantic proximity defined by a hierarchy provides a robust, data-independent prior to counteract model bias. Our method is specifically designed for vision-language models' dual-modality architecture. At the feature level, we align image embeddings with a hierarchy-guided text similarity structure. At the classifier level, we employ a distillation loss to regularize predictions using soft labels derived from the hierarchy. This dual-level injection effectively transfers knowledge from head to tail classes. Experiments on ImageNet-LT, Places-LT, and iNaturalist 2018 demonstrate that HGLTR achieves state-of-the-art performance, particularly in tail-classes accuracy, highlighting the importance of leveraging structural priors to calibrate foundation models for real-world data.

Recent studies have demonstrated that semi-supervised learning (SSL) is highly vulnerable to backdoor attacks, where adversaries can manipulate up to 90% of model predictions through just a tiny fraction of poisoned training data. Despite the widespread adoption of SSL in safety-critical applications, effective defenses against such attacks remain limited. In this paper, we present a comprehensive defense framework designed to protect SSL against sophisticated backdoor attacks. Our work begins with a systematic analysis of backdoor mechanisms in SSL from two critical perspectives: (1) how attackers establish persistent correlations between triggers and target classes; (2) how triggers are introduced and resist removal at the data level. Our investigation reveals that, unlike supervised learning, SSL backdoor attacks (1) uniquely exploit pseudo-labeling mechanisms to establish stronger trigger-target correlations, and (2) demonstrate remarkable resilience at the data level, with triggers potentially appearing in any frequency band (low, medium, or high). Based on these insights, we introduce Backdoor Invalidator (BI), a defense framework that integrates three novel techniques: complementary learning, trigger mix-up, and dual domain filtering, which collectively obstruct, dilute, and filter the influence of backdoor attacks in both feature learning and data processing. Through extensive evaluation against state-of-the-art attacks, BI significantly reduces the average attack success rate while maintaining comparable accuracy on clean data. We also provide theoretical guarantees for BI’s generalization capability and demonstrate its practical deployability as a plug-in component. The code of this work is available at https://github.com/wxr99/Backdoor Invalidator4SSL.

The open set known class bias is conventionally viewed as a fatal problem i.e., the models trained solely on known classes tend to fit unknown classes to known classes with high confidence in inference. Thus existing methods, without exception make a choice in two manners: most methods opt for eliminating the known class bias as much as possible with tireless efforts, while others circumvent the known class bias by employing a reconstruction method. However, in this paper, we challenge the two widely accepted approaches and present a novel proposition: the so-called harmful known class bias for most methods is, exactly conversely, beneficial for the reconstruction-based method and thus such known class bias can serve as a positive-incentive to the Open set recognition (OSR) models from a reconstruction perspective. Along this line, we propose the Bias Enhanced Reconstruction Learning (BERL) framework to enhance the known class bias respectively from the class level, model level and sample level. Specifically, at the class level, a specific representation is constructed in a supervised contrastive manner to avoid overgeneralization, while a diffusion model is employed by injecting the class prior to guide the biased reconstruction at the model level. Additionally, we leverage the advantages of the diffusion model to design a self-adaptive strategy, enabling effective sample-level biased sampling based on the information bottleneck theory. Experiments on various benchmarks demonstrate the effectiveness and performance superiority of the proposed method.

Existing learning-based visual retouching primarily focuses on improving image quality through end-to-end objective mapping between input and retouched images. However, these approaches often overlook two critical aspects: the progressive nature of image retouching and the subjective aesthetic preferences, resulting in suboptimal visual outcomes. To address this, we introduce Automatic Aesthetic Image Retouching via active reinforcement learning (A$^{3}$3RL) to enhance the visualization experience in two sub-tasks: color enhancement and composition optimization, which are formulated as a unified Markov Decision Process in the proposed A$^{3}$3RL framework. In our approach, each pixel functions as an autonomous agent that determines optimal actions based on aesthetic guidance, engaging in online exploration through immediate pixel-wise and channel-wise feedback from the aesthetic environment. By leveraging a pretrained image aesthetic model, our method ensures that the A$^{3}$3RL process aligns with human aesthetic preferences and adheres to subjective aesthetic principles. The framework integrates pixel-level retouching actions with image-level operations to achieve optimal image sequences through progressive iterations. Extensive experiments demonstrate that our method effectively recalibrates image aesthetics across multiple dimensions: low-level quality metrics (PSNR, SSIM), visual perception (LPIPS), and subjective visual experience (human survey). The results demonstrate high consistency with expert-retouched ground-truth images.

In recent Open-set Recognition (OSR) community, a prevailing belief is that enhancing the discriminative boundaries of closed-set classes can improve the robustness of Deep Neural Networks (DNNs) against open data during testing. Typical studies validate this *implicitly* by empirical evidence, without a formalized understanding of *how DNNs help the closed-set features obtain more discriminative boundaries?* For this, we provide an answer from the Neural Collapse (NC) perspective: DNNs align the closed-set with a *Simplex Equiangular Tight Frame* (ETF) structure that has geometric and mathematical interpretability. Regrettably, although NC naturally occurs in DNNs, we discover that typical studies cannot guarantee the features being learned to strictly align with the ETF. Thus, we introduce a novel concept, Fixed ETF Template (FiT), which holds an ideal structure associated with closed-set classes. To force class means and classifier vectors to align with FiT, we further design a Dual ETF (DEF) loss involving two components. Specifically, *F*-DEF loss is designed to align class means with FiT strictly, yielding optimal inter-class separability. Meanwhile, we extend a dual form to classifier vectors, termed *C*-DEF loss, which guides class means and classifier vectors to satisfy self-duality. Our theoretical analysis proves the validity of the proposed approach, and extensive experiments demonstrate that DEF achieves comparable or superior results with reduced computational resources on standard OSR benchmarks.

Irregularly sampled multivariate time series (ISMTS) are prevalent in reality. Due to their non-uniform intervals between successive observations and varying sampling rates among series, the channel-independent (CI) strategy, which has been demonstrated more desirable for complete multivariate time series forecasting in recent studies, has failed. This failure can be further attributed to the sampling sparsity, which provides insufficient information for effective CI learning, thereby reducing its capacity. When we resort to the channel-dependent (CD) strategy, even higher capacity cannot mitigate the potential loss of diversity in learning similar embedding patterns across different channels. We find that existing work considers CI and CD strategies to be mutually exclusive, primarily because they apply these strategies to the global channel. However, we hold the view that channel strategies do not necessarily have to be used globally. Instead, by appropriately applying them locally and globally, we can create an opportunity to take full advantage of both strategies. This leads us to introduce the Channel Harmony ISMTS Transformer (TimeCHEAT), which utilizes the CD strategy locally and the CI strategy globally. Specifically, we segment the ISMTS into sub-series level patches. Locally, the CD strategy aggregates information within each patch for time embedding learning, maximizing the use of relevant observations while reducing long-range irrelevant interference. Here, we enhance generality by transforming embedding learning into an edge weight prediction task using bipartite graphs, eliminating the need for special prior knowledge. Globally, the CI strategy is applied across patches, allowing the Transformer to learn individualized attention patterns for each channel. Experimental results indicate our proposed TimeCHEAT demonstrates competitive state-of-the-art performance across three mainstream tasks including classification, forecasting and interpolation.

Long-tailed (LT) data distribution is common in multi-label image classification (MLC) and can significantly impact the performance of classification models. One reason is the challenge of learning unbiased instance representations (i.e. features) for imbalanced datasets. Additionally, the co-occurrence of head/tail classes within the same instance, along with complex label dependencies, introduces further challenges. In this work, we delve into this problem through the lens of neural collapse (NC). NC refers to a phenomenon where the last-layer features and classifier of a deep neural network model exhibit a simplex Equiangular Tight Frame (ETF) structure during its terminal training phase. This structure creates an optimal linearly separable state. However, this phenomenon typically occurs in balanced datasets but rarely applies to the typical imbalanced problem. To induce NC properties under Long-tailed multi-label classification (LT-MLC) conditions, we propose an approach named MLC-NC, which aims to learn high-quality data representations and improve the model’s generalization ability. Specifically, MLC-NC accounts for the fact that different labels correspond to different feature parts located in images. MLC-NC extracts class-wise features from each instance through a cross-attention mechanism. To guide the features toward the ETF structure, we introduce visual-semantic feature alignment with a fixed ETF structured label embedding, which helps to learn evenly distributed class centers. To reduce within-class feature variation, we introduce collapse calibration within a lower-dimensional feature space. To mitigate classification bias, we concatenate features and feed them into a binarized fixed ETF classifier. As an orthogonal approach to existing methods, MLC-NC can be seamlessly integrated into various frameworks. Extensive experiments on widely-used benchmarks demonstrate the effectiveness of our method.

Graph neural networks (GNNs) are widely used for node classification tasks, but when encountering distribution shifts due to environmental change in real-world scenarios, they tend to learn unstable correlations between features and labels. To overcome this dilemma, a powerful class of approaches views the environment as the root cause of those unstable correlations, thereby their key focus is to infer the environment involved, enabling the model to avoid capturing environment-sensitive correlations. However, their inferences rely solely on the single-level information from one low-hop ego-graph, neglecting both global information and multi-granularity information in local ego-graphs with different hops. Although applying deeper GNNs on the high-hop ego-graph could capture global information, it will bring the side effect of over-smoothing node representations. To tackle these issues, we propose a novel Multi-Level Environment Inference model named MLEI, which effectively broadens the horizon of training GNNs under node-level distribution shifts. Specifically, MLEI first leverages a linear graph transformer to surpass the scope of ego-graph, efficiently enabling high-level global environment inference. This global environment is in turn used as an overview to assist layer-by-layer environment inference on local multi-hop ego-graphs. Finally, we combine the environment from global and local views and utilize the designed objective function to capture stable predictive patterns. Extensive experiments on real-world datasets demonstrate that our model achieves satisfactory performance compared with the state-of-the-art methods under various distribution shifts.

Minimax optimization recently is widely applied in many machine learning tasks such as generative adversarial networks, robust learning and reinforcement learning. In the paper, we study a class of nonconvex-nonconcave minimax optimization with nonsmooth regularization, where the objective function is possibly nonconvex on primal variable $x$, and it is nonconcave and satisfies the Polyak-Lojasiewicz (PL) condition on dual variable $y$. Moreover, we propose a class of enhanced momentum-based gradient descent ascent methods (i.e., MSGDA and AdaMSGDA) to solve these stochastic nonconvex-PL minimax problems. In particular, our AdaMSGDA algorithm can use various adaptive learning rates in updating the variables $x$ and $y$ without relying on any specifical types. Theoretically, we prove that our methods have the best known sample complexity of $\tilde{O}(\epsilon^{-3})$ only requiring one sample at each loop in finding an $\epsilon$-stationary solution. Some numerical experiments on PL-game and Wasserstein-GAN demonstrate the efficiency of our proposed methods.

Existing few-shot open-set recognition (FSOR) methods primarily employ the meta-learning mechanism, in which each meta-task randomly selects a small subset of base classes as knowns, and samples an equal number of classes from the remaining base classes as pseudo-unknowns. While effective, these methods potentially face two critical weaknesses: i) Class-identity overlapping: The same classes are designated as knowns in one meta-task but may be considered as pseudo-unknowns in another, leading to conflicts across meta-tasks and consequently degrading the model’s performance; ii) Narrow pseudo-unknown utilization: Each meta-task selects only a limited number of base classes as pseudo-unknowns rather than providing a broader view of more available base classes. Fundamentally, these issues arise from the myopia of the meta-learners in existing methods, as they lack a broad view on all available base classes. To this end, we strategically propose a novel Hyperopia Distillation Enhancement framework (HDE) for FSOR, which encourages the meta-learner to observe a broader view of pseudo-unknown classes without too worrying about the class-imbalance issue, while effectively mitigating the class-identity overlapping problem. The key to HDE lies in its dual hyperopia distillation mechanism, which enhances the meta-learner by hyperopically distilling available full-view inter-class relationships and more pseudo-unknown knowledge. Extensive experiments verify the effectiveness of our HDE.

Open-set semi-supervised learning (OSSL) aims to train such a model that can classify inliers from known categories into correct classes while detecting unseen categories as outliers, via leveraging limited labeled data and abundant unlabeled training data. Existing methods typically employ a multi-head structure (i.e., classifier and detector) and optimize each head separately. In contrast, we propose a unified OSSL approach, PSUMatch, which introduces an additional dataset to simultaneously enhance classification performance and outliers detection. Specifically, we first construct a dataset called Progressive Semantic Universum (PSU) which is not belonging to any labeled classes, by combining pseudo-labeled inliers from unlabeled data with labeled data as the training progresses. Then, we treat PSU as negative samples to conduct contrastive learning for better feature representations, enabling it to serve multiple heads simultaneously. In learning, we introduce a soft contrastive loss to progressively refine pseudo-labels while mitigating the effects of outliers. Extensive experiments demonstrate the effectiveness of our approach.

Multi-Label Online Continual Learning (MOCL) requires models to learn continuously from endless multi-label data streams, facing complex challenges including persistent catastrophic forgetting, potential missing labels, and uncontrollable imbalanced class distributions. While existing MOCL methods attempt to address these challenges through various techniques, \textit{they all overlook label-specific region identifying and feature learning} - a fundamental solution rooted in multi-label learning but challenging to achieve in the online setting with incremental and partial supervision. To this end, we first leverage the inherent structural information of input data to evaluate and verify the innate localization capability of different pre-trained models. Then, we propose CUTER (CUT-out-and-Experience-Replay), a simple yet versatile strategy that provides fine-grained supervision signals by further identifying, strengthening and cutting out label-specific regions for efficient experience replay. It not only enables models to simultaneously address catastrophic forgetting, missing labels, and class imbalance challenges, but also serves as an orthogonal solution that seamlessly integrates with existing approaches. Extensive experiments on multiple multi-label image benchmarks demonstrate the superiority of our proposed method. The code is available at \href{https://github.com/wxr99/Cut-Replay}{https://github.com/wxr99/Cut-Replay}

Using unlabeled wild data containing both in-distribution (ID) and out-of-distribution (OOD) data to improve the safety and reliability of models has recently received increasing attention. Existing methods either design customized losses for labeled ID and unlabeled wild data then perform joint optimization, or first filter out OOD data from the latter then learn an OOD detector. While achieving varying degrees of success, two potential issues remain: (i) Labeled ID data typically dominates the learning of models, inevitably making models tend to fit OOD data as IDs; (ii) The selection of thresholds for identifying OOD data in unlabeled wild data usually faces dilemma due to the unavailability of pure OOD samples. To address these issues, we propose a novel loss-difference OOD detection framework (LoD) by intentionally label-noisifying unlabeled wild data. Such operations not only enable labeled ID data and OOD data in unlabeled wild data to jointly dominate the models' learning but also ensure the distinguishability of the losses between ID and OOD samples in unlabeled wild data, allowing the classic clustering technique (e.g., K-means) to filter these OOD samples without requiring thresholds any longer. We also provide theoretical foundation for LoD's viability, and extensive experiments verify its superiority.

In single Domain Generalization (single-DG), data scarcity in the single source domain hampers the learning for invariant features, leading to overfitting over source domain and poor generalization to unseen target domains. Existing single-DG methods primarily augment the source domain by adversarial generation. However, there are still two key challenges. i) With simple feature perturbation to confuse the classifier, it may generate unnatural samples with semantic ambiguity or distortion. ii) It is still difficult to cover the sufficient shift in a real domain by generating indistinguishable samples from source data, thus the learning model is inescapable from overfitting to the single source domain. To this end, we turn to augment the domain prompt, considering that text prompt perturbation is easier to generate and generalize. Then the source domain is expanded with the guidance of augmented text prompts, which are learnable with both semantic consistency and style diversity. Specifically, we propose a ProMpt-driven Expansion and Alignment (ProMEA) method for single-DG, in which a Domain Prompt Expansion module is first developed to expand the single source domain with frequency features of augmented text prompts, in which the amplitude spectrum predominantly harbors the domain style information. With source prompts, a Domain Prompt Alignment module is further designed in inference for adapting target samples to the expanded source domains, in order to reduce the domain discrepancy. Finally, empirically results over single-DG benchmarks demonstrate the superiority of our proposal.

The need to generalize the pre-trained deep learning models to unknown test-time data distributions has spurred research into test-time adaptation (TTA). Existing studies have mainly focused on closed-set TTA with only covariate shifts, while largely overlooking open-set TTA that involves semantic shifts, i.e., unknown open-set classes. However, addressing adaptation to unknown classes is crucial for open-world safety-critical applications such as autonomous driving. In this paper, we emphasize that accurate identification of the open-set samples is rather challenging in TTA. The entanglement of semantic shift and covariate shift mutually confuse the network’s discriminative capability. This co-interference further exacerbates considering the single-pass data nature and low latency requirements. With this under standing, we propose Dual-mode Matching and Prompt-based Open Set Adaptation (DM-POSA) for open-set TTA to enhance discriminative feature learning and unknown classes distinguishment with minimal time cost. DM-POSA identifies open-set samples via dual-mode matching strategies, including model-parameter-based and feature space-based matching. It also optimizes the model with a random pairing discrepancy loss, enhancing the distributional difference between open-set and closed-set samples, thus improving the model’s ability to recognize unknown categories. Extensive experiments show the superiority of DM-POSA over state-of-the-art baselines on both closed-set class adaptation and open-set class detection.

Consistency and complementarity are two key ingredients for boosting multi-view clustering (MVC). Recently with the introduction of popular contrastive learning, the consistency learning of views has been further enhanced in MVC, leading to promising performance. However, by contrast, the complementarity has not received sufficient attention except just in the feature facet, where the Hilbert Schmidt Independence Criterion term or the independent encoder-decoder network is usually adopted to capture view-specific information. This motivates us to reconsider the complementarity learning of views comprehensively from multiple facets including the feature-, view-label- and contrast- facets, while maintaining the view consistency. We empirically find that all the facets contribute to the complementarity learning, especially the view-label facet, which is usually neglected by existing methods. Based on this, a simple yet effectiveMultifacetComplementarity learning framework forMulti-ViewClustering (MCMVC) is naturally developed, which fuses multifacet complementarity information, especially explicitly embedding the view-label information. To our best knowledge, it is the first time to use view-labels explicitly to guide the complementarity learning of views. Compared with the SOTA baselines, MCMVC achieves remarkable improvements, e.g., by average margins over 5.00% and 7.00% respectively in complete and incomplete MVC settings on Caltech101-20 in terms of three evaluation metrics.

Evaluating the performance of low-light image enhancement (LLE) is highly subjective, thus making integrating human preferences into LLE a necessity. Existing methods fail to consider this and present a series of potentially valid heuristic criteria for training LLE models. In this paper, we propose a new paradigm, i.e., aesthetics-guided low-light image enhancement (ALL-E), which introduces aesthetic preferences to LLE and motivates training in a reinforcement learning framework with an aesthetic reward. Each pixel, functioning as an agent, refines itself by recursive actions. We further present ALL-E+, an extended version of ALL-E, which casts a two-stage aesthetics-guided enhancement and denoising. ALL-E+ achieves low-light enhancement and denoising compensation sequentially in a unified framework, resulting in significant improvements in both subjective visual experience and objective evaluation. Extensive experiments show that integrating aesthetic preferences can further improve the visual experience of enhanced images. Our results on various benchmarks also demonstrate the superiority of our method over state-of-the-art methods.

When taking images against strong light sources, the resulting images often contain heterogeneous flare artifacts. These artifacts can significantly affect image visual quality and downstream computer vision tasks. While collecting real data pairs of flare-corrupted/flare-free images for training flare removal models is challenging, current methods utilize the direct-add approach to synthesize training data. However, these methods do not consider automatic exposure and tone mapping in the image signal processing pipeline (ISP), leading to the limited generalization capability of deep model training using such data. Besides, existing light source recovery methods hardly recover multiple light sources due to the different sizes, shapes, and illuminance of various light sources. In this paper, we propose a solution to improve the performance of lens flare removal by revisiting the ISP, remodeling the principle of automatic exposure in the synthesis pipeline, and designing a more reliable light source recovery strategy. The new pipeline approaches realistic imaging by discriminating the local and global illumination through a convex combination, avoiding global illumination shifting and local over-saturation. Moreover, the current deep models are only generalized to specific devices due to the diversity of cameras' ISPs. To achieve better generalization on different devices, we formulate the generalization problem as an adversarial training problem and embed an adversarial curve learning (ACL) paradigm in the synthesis pipeline to gain better performance. For recovering multiple light sources, our strategy convexly averages the input and output of the neural network based on illuminance levels, thereby avoiding the need for a hard threshold in identifying light sources. We also contribute a new flare removal testing dataset containing the flare-corrupted images captured by fifteen types of consumer electronics. The dataset facilitates the verification of the generalization capability of flare removal methods. Extensive experiments show that our solution can effectively improve the performance of lens flare removal and push the frontier toward more general situations.

In recent years, image segmentation has made significant progress, but acquiring annotated data is still a considerable challenge, especially in remote sensing imagery (RSI). The complex structure and inter-category confusion of RSI increase the time-consuming and cost of pixel-level annotation, and noisy annotations inevitably appear. This paper proposes a boundary-aware knowledge distillation method (BAKD) to handle noisy annotations by evaluating their uncertainty. BAKD consists of two core strategies: Predictive Confidence Evaluation (PCE) and Boundary-annotated Reliability Evaluation (BRE). The predictive confidence jointly decided by the teacher and student networks reflects the annotation’s uncertainty. The boundary-annotated reliability directly measures the annotation’s uncertainty based on the distance from the annotation to the semantic boundary. Leveraging these two types of uncertainty information, BAKD assigns each sample a comprehensive boundary-aware weight to identify samples with potential noisy annotations. This alleviates the impact of noisy annotation on the model’s training and improves its generalization performance. Experimental results show that BAKD achieves competitive semantic segmentation performance on the Potsdam and Vaihingen benchmarks compared with the state-of-the-art KD methods. In addition, BAKD can be easily integrated into semantic segmentation methods based on KD, extending their applicability in handling noisy annotations. Codes are available at https://github.com/sunyueue/BAKD.git.

Neural Collapse (NC) presents an elegant geometric structure that enables individual activations (features), class means and classifier (weights) vectors to reachoptimalinter-class separability during the terminal phase of training on abalanceddataset. Once shifted to imbalanced classification, such an optimal structure of NC can be readily destroyed by the notoriousminority collapse, where the classifier vectors corresponding to the minority classes are squeezed. In response, existing works mainly optimize classifiers in an effort to recover NC. However, we discover that this squeezing phenomenon is not only confined to classifier vectors but also occurs with class means. Consequently, reconstructing NC solely at the classifier aspect may be futile, as the class means remain compressed, leading to the violation of inherentself-dualityin NC (i.e., class means and classifier vectors converge mutually) and incidentally, an unsatisfactory collapse of individual activations towards the corresponding class means. To shake off these dilemmas, we present a unifiedAll-aroundNeuralCollapse framework (AllNC), aiming to comprehensively restore NC across multiple aspects including individual activations, class means and classifier vectors. We thoroughly analyze its effectiveness and verify its performance on multiple benchmark datasets as state-of-the-art in both balanced and imbalanced settings.

Unsupervised domain adaptation (UDA) promotes target learning via a single -directional transfer from label-rich source domain to unlabeled target, while its reverse adaption from target to source has not been jointly considered yet. In real teaching practice, a teacher helps students learn and also gets promotion from students, and such a virtuous cycle inspires us to explore dual -directional transfer between domains. In fact, target pseudo-labels predicted by source commonly involve noise due to model bias; moreover, source domain usually contains innate noise, which inevitably aggravates target noise, leading to noise amplification. Transfer from target to source exploits target knowledge to rectify the adaptation, consequently enables better source transfer, and exploits a virtuous transfer circle. To this end, we propose a dual-correction-adaptation network (DualCAN), in which adaptation and correction cycle between domains, such that learning in both domains can be boosted gradually. To the best of our knowledge, this is the first naive attempt of dual-directional adaptation. Empirical results validate DualCAN with remarkable performance gains, particularly for extreme noisy tasks (e.g., approximately +10% on of Office-31 with 40% label corruption).

The graph-structured learning (GSL) aims to assist graph neural networks (GNNs) to yield effective node embeddings for downstream tasks, especially in scenarios with the absence of structures or the existence of unreliable edges. Most GSL models are built on i.i.d. assumption across training and testing data. However, this assumption can be violated, where testing data contain out-of-distribution (OOD) samples. Consequently, those models are limited in generalization, which will lead to a poor structure. On the other hand, while they have made great progress, additional optimized parameters are required due to their implementation with parametric models. To tackle the above problems, we propose a novel generalizable and nonparametric structure learning framework named GNS, which can be easily and effectively applied to various tasks. GNS neither relies on i.i.d. assumption nor even involves any parameters being optimized, instead to find an appropriate similarity between nodes and an associated threshold to establish desirable structures. Specifically, we first incorporate the candidate neighbor distributions for nodes to refine the similarity. Then, we introduce an adaptive threshold discovery method inspired by Fisher's criterion to determine final structures. Extensive experiments demonstrate that GNS excels not only in OOD scenarios but also in the general classification and regression prediction tasks.

Domain generalization aims to learn a well-performed classifier on multiple source domains for unseen target domains under domain shift. Domain-invariant representation (DIR) is an intuitive approach and has been of great concern. In practice, since the targets are variant and agnostic, only a few sources are not sufficient to reflect the entire domain population, leading to biased DIR. Derived from PAC-Bayes framework, we provide a novel generalization bound involving the number of domains sampled from the environment (N) and the radius of the Wasserstein ball centred on the target (r), which have rarely been considered before. Herein, we can obtain two natural and significant findings: when N increases, 1) the gap between the source and target sampling environments can be gradually mitigated; 2) the target can be better approximated within the Wasserstein ball. These findings prompt us to collect adequate domains against domain shift. For seeking convenience, we design a novel yet simple Extrapolation Domain strategy induced by the Mixup scheme, namely EDM. Through a reverse Mixup scheme to generate the extrapolated domains, combined with the interpolated domains, we expand the interpolation space spanned by the sources, providing more abundant domains to increase sampling intersections to shorten r. Moreover, EDM is easy to implement and be plugged-and-played. In experiments, EDM has been plugged into several methods in both closed and open set settings, achieving up to 5.73% improvement.

In open-set recognition (OSR), a promising strategy is exploiting pseudo-unknown data outside given K known classes as an additional K+1-th class to explicitly model potential open space. However, treating unknown classes without distinction is unequal for them relative to known classes due to the category-agnostic and scale-agnostic of the unknowns. This inevitably not only disrupts the inherent distributions of unknown classes but also incurs both class-wise and instance-wise imbalances between known and unknown classes. Ideally, the OSR problem should model the whole class space as K+∞, but enumerating all unknowns is impractical. Since the core of OSR is to effectively model the boundaries of known classes, this means just focusing on the unknowns nearing the boundaries of targeted known classes seems sufficient. Thus, as a compromise, we convert the open classes from infinite to K, with a novel concept Target-Aware Universum (TAU) and propose a simple yet effective framework Dual Contrastive Learning with Target-Aware Universum (DCTAU). In details, guided by the targeted known classes, TAU automatically expands the unknown classes from the previous 1 to K, effectively alleviating the distribution disruption and the imbalance issues mentioned above. Then, a novel Dual Contrastive (DC) loss is designed, where all instances irrespective of known or TAU are considered as positives to contrast with their respective negatives. Experimental results indicate DCTAU sets a new state-of-the-art.

Learning universal time series representations applicable to various types of downstream tasks is challenging but valuable in real applications. Recently, researchers have attempted to leverage the success of self-supervised contrastive learning (SSCL) in Computer Vision(CV) and Natural Language Processing(NLP) to tackle time series representation. Nevertheless, due to the special temporal characteristics, relying solely on empirical guidance from other domains may be ineffective for time series and difficult to adapt to multiple downstream tasks. To this end, we review three parts involved in SSCL including 1) designing augmentation methods for positive pairs, 2) constructing (hard) negative pairs, and 3) designing SSCL loss. For 1) and 2), we find that unsuitable positive and negative pair construction may introduce inappropriate inductive biases, which neither preserve temporal properties nor provide sufficient discriminative features. For 3), just exploring segment- or instance-level semantics information is not enough for learning universal representation. To remedy the above issues, we propose a novel self-supervised framework named TimesURL. Specifically, we first introduce a frequency-temporal-based augmentation to keep the temporal property unchanged. And then, we construct double Universums as a special kind of hard negative to guide better contrastive learning. Additionally, we introduce time reconstruction as a joint optimization objective with contrastive learning to capture both segment-level and instance-level information. As a result, TimesURL can learn high-quality universal representations and achieve state-of-the-art performance in 6 different downstream tasks, including short- and long-term forecasting, imputation, classification, anomaly detection and transfer learning.

Learning from noisy data has attracted much attention, where most methods focus on closed-set label noise. However, a more common scenario in the real world is the presence of both open-set and closed-set noise. Existing methods typically identify and handle these two types of label noise separately by designing a specific strategy for each type. However, in many real-world scenarios, it would be challenging to identify open-set examples, especially when the dataset has been severely corrupted. Unlike the previous works, we explore how models behave when faced with open-set examples, and find that a part of open-set examples gradually get integrated into certain known classes, which is beneficial for the separation among known classes. Motivated by the phenomenon, we propose a novel two-step contrastive learning method CECL (Class Expansion Contrastive Learning) which aims to deal with both types of label noise by exploiting the useful information of open-set examples. Specifically, we incorporate some open-set examples into closed-set classes to enhance performance while treating others as delimiters to improve representative ability. Extensive experiments on synthetic and real-world datasets with diverse label noise demonstrate the effectiveness of CECL.

Federated learning is a popular distributed and privacy-preserving learning paradigm in machine learning. Recently, some federated learning algorithms have been proposed to solve the distributed minimax problems. However, these federated minimax algorithms still suffer from high gradient or communication complexity. Meanwhile, few algorithm focuses on using adaptive learning rate to accelerate these algorithms. To fill this gap, in the paper, we study a class of nonconvex minimax optimization, and propose an efficient adaptive federated minimax optimization algorithm (i.e., AdaFGDA) to solve these distributed minimax problems. Specifically, our AdaFGDA builds on the momentum-based variance reduced and local-SGD techniques, and it can flexibly incorporate various adaptive learning rates by using the unified adaptive matrices. Theoretically, we provide a solid convergence analysis framework for our AdaFGDA algorithm under non-i.i.d. setting. Moreover, we prove our AdaFGDA algorithm obtains a lower gradient (i.e., stochastic first-order oracle, SFO) complexity of $\tilde{O}(\epsilon^{-3})$ with lower communication complexity of $\tilde{O}(\epsilon^{-2})$ in finding $\epsilon$-stationary point of the nonconvex minimax problems. Experimentally, we conduct some experiments on the deep AUC maximization and robust neural network training tasks to verify efficiency of our algorithms.

Online continual learning (OCL) requires the models to learn from constant, endless streams of data. While significant efforts have been made in this field, most were focused on mitigating the \textit{catastrophic forgetting} issue to achieve better classification ability, at the cost of a much heavier training workload. They overlooked that in real-world scenarios, e.g., in high-speed data stream environments, data do not pause to accommodate slow models. In this paper, we emphasize that \textit{model throughput}-- defined as the maximum number of training samples that a model can process within a unit of time -- is equally important. It directly limits how much data a model can utilize and presents a challenging dilemma for current methods. With this understanding, we revisit key challenges in OCL from both empirical and theoretical perspectives, highlighting two critical issues beyond the well-documented catastrophic forgetting: (\romannumeral1) Model's ignorance: the single-pass nature of OCL challenges models to learn effective features within constrained training time and storage capacity, leading to a trade-off between effective learning and model throughput; (\romannumeral2) Model's myopia: the local learning nature of OCL on the current task leads the model to adopt overly simplified, task-specific features and \textit{excessively sparse classifier}, resulting in the gap between the optimal solution for the current task and the global objective. To tackle these issues, we propose the Non-sparse Classifier Evolution framework (NsCE) to facilitate effective global discriminative feature learning with minimal time cost. NsCE integrates non-sparse maximum separation regularization and targeted experience replay techniques with the help of pre-trained models, enabling rapid acquisition of new globally discriminative features. Extensive experiments demonstrate the substantial improvements of our framework in performance, throughput and real-world practicality.

Abstract Anomaly detection is a method employed to identify data points or patterns that significantly deviate from expected or normal behaviour within a dataset. This approach aims to detect observations regarded as unusual, erroneous, anomalous, rare, or potentially indicative of fraudulent or malicious activity. Open‐set recognition, also referred to as open‐set identification or open‐set classification, is a pattern recognition task that extends traditional classification by addressing the presence of unknown or novel classes during the testing phase. This approach highlights a strong connection between anomaly detection and open‐set recognition, as both seek to identify samples originating from unknown classes or distributions. Open‐set recognition methods frequently involve modelling both known and unknown classes during training, allowing for the capture of the distribution of known classes while explicitly addressing the space of unknown classes. Techniques in open‐set recognition may include outlier detection, density estimation, or configuring decision boundaries to better differentiate between known and unknown classes. This special issue calls for original contributions introducing novel datasets, innovative architectures, and advanced training methods for tasks related to visual anomaly detection and open‐set recognition.

Universal domain adaptation (UniDA) is a general unsupervised domain adaptation setting, which addresses both domain and label shifts in adaptation. Its main challenge lies in how to identify target samples in unshared or unknown classes. Previous methods commonly strive to depict sample “confidence” along with a threshold for rejecting unknowns, and align feature distributions of shared classes across domains. However, it is still hard to pre-specify a “confidence” criterion and threshold which are adaptive to different tasks, and a mis-prediction of unknowns further incurs mis-alignment of features in shared classes. In this paper, we propose a new UniDA method with adaptive U nknown A uthentication by C lassifier P aradox (UACP), considering that samples with paradoxical predictions are probably unknowns belonging to none of the source classes. In UACP, a composite classifier is jointly designed with two types of predictors. That is, a multi-class (MC) predictor classifies samples to one of the multiple source classes, while a binary one-vs-all predictor further verifies the prediction by MC predictor. Samples with verification failure or paradox are identified as unknowns. Further, instead of feature alignment for shared classes, implicit domain alignment is conducted in output space such that samples across domains share the same decision boundary, though with feature discrepancy. Empirical results validate UACP under both open-set and universal UDA settings.

Logit adjustment is an effective long-tailed visual recognition strategy to encourage a significant margin between rare and dominant labels. Existing methods typically employ the globally fixed label frequencies throughout the training to adjust margins. However, in practice, we observe that the local (in-batch) label frequencies change dynamically or even vanish for some classes (especially the tail classes) in batch-dependent training, which is inconsistent with global ones. Furthermore, our analyses reveal that the intra-classcollinear samplesactually do not contribute to the gradient update, but substantially increase the corresponding local label frequencies. Such contributions are spurious due to over-counting the label frequencies without contributing to the gradient. All of these will cause serious interference in precisely estimating local frequencies of the authentic contribution, leading to inauthentic margins. To simultaneously address the above issues, this paper innovatively proposes Dynamic Learnable Logit Adjustment (DLLA) loss to learn the local label frequencies within dynamic mini-batches precisely. Specifically, DLLA owns two complementary parts: 1)rank-metriceliminates spurious contributions from collinear samples by calculating the algebraic rank of the feature subspace in the mini-batch. 2)class-supplementensures all classes appear in every mini-batch by inserting the corresponding learnable class prototype, for which we resort to neural collapse theory to make them align to the ideal regular simplex structure. Extensive experiments on standard benchmark datasets verify the effectiveness of our method.

Multi-label image classification relies on a large-scale, well-maintained dataset, which may easily be mislabeled due to various subjective reasons. Existing methods for coping with noise usually focus on improving the model robustness in the case of single-label noise. However, compared with noisy single-label learning, noisy multi-label learning is more practical and challenging. To reduce the negative impact of noisy multi-annotations, we propose a universal approach for noisy multi-label learning (UNM). In UNM, we propose the label-wise embedding network which investigates the semantic alignment between label embeddings and their corresponding output features to learn robust feature representations. Meanwhile, mining the co-occurrence of multi-labels is also added to regularize the noisy network predictions. We cyclically change the fitting status of our label-wise embedding network to distinguish the noisy samples and generate pseudo labels for them. As a result, UNM provides an effective way to exploit the label-wise features and semantic label embeddings in noisy scenarios. To verify the generalizability of our method, we also test our method on Partial Multi-label Learning (PML) and Multi-label Learning with Missing Labels (MLML). Extensive experiments on benchmark datasets including Microsoft COCO, Pascal VOC, and Visual Genome explicitly validate the proposed method.

Unsupervised domain adaptation (UDA) is an emerging learning paradigm that models on unlabeled datasets by leveraging model knowledge built on other labeled datasets, in which the statistical distributions of these datasets are usually not identical. Formally, UDA is to leverage knowledge from a labeled source domain to promote an unlabeled target domain. Although there have been a variety of methods proposed to address the UDA problem, most of them are dedicated to single-source-to-single-target domain, while the works on single-source-to-multitarget domain are relatively rare. Compared to the single-source domain with single-target domain scenario, the UDA from single-source domain to multitarget domain is more challenging since it needs to consider not only the relationships between the source and the target domains but also those among the target domains. To this end, this article proposes a kind of dictionary learning-based unsupervised multitarget domain adaptation method (DL-UMTDA). In DL-UMTDA, a common dictionary is constructed to correlate the single-source and multitarget domains, while individual dictionaries are designed to exploit the private knowledge for the target domains. Through learning the corresponding dictionary representation coefficients in the UDA process, the correlations from the source to the target domains as well as these potential relationships between the target domains can be effectively exploited. In addition, we design an alternating algorithm to solve the DL-UMTDA model with theoretical convergence guarantee. Finally, extensive experiments on benchmark (Office + Caltech) and real datasets (AgeDB, Morph, and CACD) validate the superiority of the proposed method.

Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems, but they are still trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit stochastic gradient descent (ISGD) method to train PINNs for improving the stability of training process. We heuristically analyze how ISGD overcome stiffness in the gradient flow dynamics of PINNs, especially for problems with multi-scale solutions. We theoretically prove that for two-layer fully connected neural networks with large hidden nodes, randomly initialized ISGD converges to a globally optimal solution for the quadratic loss function. Empirical results demonstrate that ISGD works well in practice and compares favorably to other gradient-based optimization methods such as SGD and Adam, while can also effectively address the numerical stiffness in training dynamics via gradient descent.

When taking images against strong light sources, the resulting images often contain heterogeneous flare artifacts. These artifacts can importantly affect image visual quality and downstream computer vision tasks. While collecting real data pairs of flare-corrupted/flare-free images for training flare removal models is challenging, current methods utilize the direct-add approach to synthesize data. However, these methods do not consider automatic exposure and tone mapping in image signal processing pipeline (ISP), leading to the limited generalization capability of deep models training using such data. Besides, existing methods struggle to handle multiple light sources due to the different sizes, shapes and illuminance of various light sources. In this paper, we propose a solution to improve the performance of lens flare removal by revisiting the ISP and remodeling the principle of automatic exposure in the synthesis pipeline and design a more reliable light sources recovery strategy. The new pipeline approaches realistic imaging by discriminating the local and global illumination through convex combination, avoiding global illumination shifting and local over-saturation. Our strategy for recovering multiple light sources convexly averages the input and output of the neural network based on illuminance levels, thereby avoiding the need for a hard threshold in identifying light sources. We also contribute a new flare removal testing dataset containing the flare-corrupted images captured by ten types of consumer electronics. The dataset facilitates the verification of the generalization capability of flare removal methods. Extensive experiments show that our solution can effectively improve the performance of lens flare removal and push the frontier toward more general situations.

In open set recognition (OSR), almost all existing methods are designed specially for recognizing individual instances, even these instances are collectively coming in batch. Recognizers in decision either reject or categorize them to some known class using empirically-set threshold. Thus the decision threshold plays a key role. However, the selection for it usually depends on the knowledge of known classes, inevitably incurring risks due to lacking available information from unknown classes. On the other hand, a more realistic OSR system should NOT just rest on a reject decision but should go further, especially for discovering the hidden unknown classes among the reject instances, whereas existing OSR methods do not pay special attention. In this paper, we introduce a novel collective/batch decision strategy with an aim to extend existing OSR for new class discovery while considering correlations among the testing instances. Specifically, a collective decision-based OSR framework (CD-OSR) is proposed by slightly modifying the Hierarchical Dirichlet process (HDP). Thanks to HDP, our CD-OSR does not need to define the decision threshold and can implement the open set recognition and new class discovery simultaneously. Finally, extensive experiments on benchmark datasets indicate the validity of CD-OSR.

Federated Semi-Supervised Learning alleviates the necessity for fully labeled data in Federated Learning. However, it does not sufficiently prioritize model privacy or the personalized requirements of clients. To address these concerns, our key idea is to communicate no longer individual model parameters but their black-box models, implying to provide other clients with only an input-output interface of individual models. The communication mechanism is model-agnostic, thereby facilitating adaption to heterogeneous models for each client through customization. Consequently, we propose a framework called Personalized Federated Semi-Supervised Learning with Black-Box Models (B2PFSSL) to enhance the privacy of communication. To prevent negative knowledge transfer due to data heterogeneity, we design a two-stage strategy that filters at both the model and data levels, enabling clients to obtain large training datasets by including more high-quality pseudo-labeled data under conditions of scarce labeled data. The experimental results indicate that B2PFSSL achieves competitive performance while reducing the amount of information exposed during communication. Furthermore, it can foster productive collaboration among diverse model architectures in model heterogeneous Federated Learning.

Evaluating the performance of low-light image enhancement (LLE) is highly subjective, thus making integrating human preferences into image enhancement a necessity. Existing methods fail to consider this and present a series of potentially valid heuristic criteria for training enhancement models. In this paper, we propose a new paradigm, i.e., aesthetics-guided low-light image enhancement (ALL-E), which introduces aesthetic preferences to LLE and motivates training in a reinforcement learning framework with an aesthetic reward. Each pixel, functioning as an agent, refines itself by recursive actions, i.e., its corresponding adjustment curve is estimated sequentially. Extensive experiments show that integrating aesthetic assessment improves both subjective experience and objective evaluation. Our results on various benchmarks demonstrate the superiority of ALL-E over state-of-the-art methods. Source code: https://dongl-group.github.io/project pages/ALLE.html

Learning binary classifiers from positive and unlabeled data (PUL) is vital in many real-world applications, especially when verifying negative examples is difficult. Despite the impressive empirical performance of recent PUL methods, challenges like accumulated errors and increased estimation bias persist due to the absence of negative labels. In this paper, we unveil an intriguing yet long-overlooked observation in PUL: \textit{resampling the positive data in each training iteration to ensure a balanced distribution between positive and unlabeled examples results in strong early-stage performance. Furthermore, predictive trends for positive and negative classes display distinctly different patterns.} Specifically, the scores (output probability) of unlabeled negative examples consistently decrease, while those of unlabeled positive examples show largely chaotic trends. Instead of focusing on classification within individual time frames, we innovatively adopt a holistic approach, interpreting the scores of each example as a temporal point process (TPP). This reformulates the core problem of PUL as recognizing trends in these scores. We then propose a novel TPP-inspired measure for trend detection and prove its asymptotic unbiasedness in predicting changes. Notably, our method accomplishes PUL without requiring additional parameter tuning or prior assumptions, offering an alternative perspective for tackling this problem. Extensive experiments verify the superiority of our method, particularly in a highly imbalanced real-world setting, where it achieves improvements of up to $11.3\%$ in key metrics.

In real-world applications, we often encounter multi-view learning tasks where we need to learn from multiple sources of data or use multiple sources of data to make decisions. Multi-view representation learning, which can learn a unified representation from multiple data sources, is a key pre-task of multi-view learning and plays a significant role in real-world applications. Accordingly, how to improve the performance of multi-view representation learning is an important issue. In this work, inspired by human collective intelligence shown in group decision making, we introduce the concept of view communication into multi-view representation learning. Furthermore, by simulating human communication mechanism, we propose a novel multi-view representation learning approach that can fulfill multi-round view communication. Thus, each view of our approach can exploit the complementary information from other views to help with modeling its own representation, and mutual help between views is achieved. Extensive experiment results on six datasets from three significant fields indicate that our approach substantially improves the average classification accuracy by 4.536% in medicine and bioinformatics fields as well as 4.115% in machine learning field.

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a well-labeled source domain to a related and unlabeled target domain with identical label space. The main workhorse in UDA is domain alignment and has proven successful. However, it is practically difficult to find an appropriate source domain with identical label space. A more practical scenario is partial domain adaptation (PDA) where the source label space subsumes the target one. Unfortunately, due to the non-identity between label spaces, it is extremely hard to obtain an ideal alignment, conversely, easier resulting in mode collapse and negative transfer. These motivate us to find a relatively simpler alternative to solve PDA. To achieve this, we first explore a theoretical analysis, which says that the target risk is bounded by both model smoothness and between-domain discrepancy. Then, we instantiate the model smoothness as an intra-domain structure preserving (IDSP) while giving up possibly riskier domain alignment. To our best knowledge, this is the first naive attempt for PDA without alignment. Finally, our empirical results on benchmarks demonstrate that IDSP is not only superior to the PDA SOTAs (e.g., ∼ +10% on Cl → Rw and ∼ +8% on Ar → Rw), but also complementary to domain alignment in the standard UDA.

As an effective data augmentation method, Mixup synthesizes an extra amount of samples through linear interpolations. Despite its theoretical dependency on data properties, Mixup reportedly performs well as a regularizer and calibrator contributing reliable robustness and generalization to deep model training. In this paper, inspired by Universum Learning which uses out-of-class samples to assist the target tasks, we investigate Mixup from a largely under-explored perspective - the potential to generate in-domain samples that belong to none of the target classes, that is, universum. We find that in the framework of supervised contrastive learning, Mixup-induced universum can serve as surprisingly high-quality hard negatives, greatly relieving the need for large batch sizes in contrastive learning. With these findings, we propose Universum-inspired supervised Contrastive learning (UniCon), which incorporates Mixup strategy to generate Mixup-induced universum as universum negatives and pushes them apart from anchor samples of the target classes. We extend our method to the unsupervised setting, proposing Unsupervised Universum-inspired contrastive model (Un-Uni). Our approach not only improves Mixup with hard labels, but also innovates a novel measure to generate universum data. With a linear classifier on the learned representations, UniCon shows state-of-the-art performance on various datasets. Specially, UniCon achieves 81.7% top-1 accuracy on CIFAR-100, surpassing the state of art by a significant margin of 5.2% with a much smaller batch size, typically, 256 in UniCon vs. 1024 in SupCon (Khosla et al., 2020) using ResNet-50. Un-Uni also outperforms SOTA methods on CIFAR-100. The code of this paper is released on https://github.com/hannaiiyanggit/UniCon.

Subspace clustering has attracted much attention because of its ability to group unlabeled high-dimensional data into multiple subspaces. Existing graph-based subspace clustering methods focus on either the sparsity of data affinity or the low rank of data affinity. Thus, the quality of data affinity plays an essential role in the performance of subspace clustering. However, the real-world data are generally high-dimensional, complex, and heterogeneous multi-source data, so that the data affinity learned by these methods cannot be completely dependent. Moreover, since these approaches always ignore the intrinsic structure of data, their grouping effect is relatively low. In this paper, we propose a novel unsupervised algorithm, called Structure-Aware Subspace Clustering (SASC), to address the above issues. SASC considers local and global correlation structures simultaneously to capture the intrinsic structure. Further, it integrates the captured structure into representation learning to gain a relatively precise data affinity. It is powerful to promote an all-around grouping effect and enhances the robustness and applicability of subspace clustering. Experiments on various benchmark datasets, including bioinformatics, handwritten digit, object image, and speech signal, demonstrate the effectiveness of the proposed algorithm.

Similarity-based method gives rise to a new class of methods for multi-label learning and also achieves promising performance. In this paper, we generalize this method, resulting in a new framework for classification task. Specifically, we unite similarity-based learning and generalized linear models to achieve the best of both worlds. This allows us to capture interdependencies between classes and prevent from impairing performance of noisy classes. Each learned parameter of the model can reveal the contribution of one class to another, providing interpretability to some extent. Experiment results show the effectiveness of the proposed approach on multi-class and multi-label data sets.

As an effective method for XOR problems, generalized eigenvalue proximal support vector machine (GEPSVM) recently has gained widespread attention accompanied with many variants proposed. Although these variants strengthen the classification performance to different extents, the number of fitting hyperplanes, similar to GEPSVM, for each class is still limited to just one. Intuitively, using single hyperplane seems not enough, especially for the datasets with complex feature structures. Therefore, this article mainly focuses on extending the fitting hyperplanes for each class from single one to multiple ones. However, such an extension from the original GEPSVM is not trivial even though, if possible, the elegant solution via generalized eigenvalues will also not be guaranteed. To address this issue, we first make a simple yet crucial transformation for the optimization problem of GEPSVM and then propose a novel multiplane convex proximal support vector machine (MCPSVM), where a set of hyperplanes determined by the features of the data are learned for each class. We adopt a strictly (geodesically) convex objective to characterize this optimization problem; thus, a more elegant closed-form solution is obtained, which only needs a few lines of MATLAB codes. Besides, MCPSVM is more flexible in form and can be naturally and seamlessly extended to the feature weighting learning, whereas GEPSVM and its variants can hardly straightforwardly work like this. Extensive experiments on benchmark and large-scale image datasets indicate the advantages of our MCPSVM.

Subspace clustering is a class of extensively studied clustering methods where the spectral-type approaches are its important subclass. Its key first step is to desire learning a representation coefficient matrix with block diagonal structure. To realize this step, many methods were successively proposed by imposing different structure priors on the coefficient matrix. These impositions can be roughly divided into two categories, i.e., indirect and direct. The former introduces the priors such as sparsity and low rankness to indirectly or implicitly learn the block diagonal structure. However, the desired block diagonalty cannot necessarily be guaranteed for noisy data. While the latter directly or explicitly imposes the block diagonal structure prior such as block diagonal representation (BDR) to ensure so-desired block diagonalty even if the data is noisy but at the expense of losing the convexity that the former's objective possesses. For compensating their respective shortcomings, in this article, we follow the direct line to propose adaptive BDR (ABDR) which explicitly pursues block diagonalty without sacrificing the convexity of the indirect one. Specifically, inspired by Convex BiClustering, ABDR coercively fuses both columns and rows of the coefficient matrix via a specially designed convex regularizer, thus naturally enjoying their merits and adaptively obtaining the number of blocks. Finally, experimental results on synthetic and real benchmarks demonstrate the superiority of ABDR to the state-of-the-arts (SOTAs).

Adversarial examples crafted by an explicit adversary have attracted significant attention in machine learning. However, the security risk posed by a potential false friend has been largely overlooked. In this paper, we unveil the threat of hypocritical examples---inputs that are originally misclassified yet perturbed by a false friend to force correct predictions. While such perturbed examples seem harmless, we point out for the first time that they could be maliciously used to conceal the mistakes of a substandard (i.e., not as good as required) model during an evaluation. Once a deployer trusts the hypocritical performance and applies the "well-performed" model in real-world applications, unexpected failures may happen even in benign environments. More seriously, this security risk seems to be pervasive: we find that many types of substandard models are vulnerable to hypocritical examples across multiple datasets. Furthermore, we provide the first attempt to characterize the threat with a metric called hypocritical risk and try to circumvent it via several countermeasures. Results demonstrate the effectiveness of the countermeasures, while the risk remains non-negligible even after adaptive robust training.

Decision tree (DT) attracts persistent research attention due to its impressive empirical performance and interpretability in numerous applications. However, the growth of traditional yet widely-used univariate decision trees (UDTs) is quite time-consuming as they need to traverse all the features to find the splitting value with the maximal reduction of the impurity at each internal node. In this paper, we newly design a splitting criterion to speed up the growth. The criterion is induced from Geometric Mean Metric Learning (GMML) and then optimized under its diagonalized metric matrix constraint, consequently, a closed-form rank of feature discriminant abilities can at once be obtained and the top 1 feature at each node used to grow an intent DT (called as dGMML-DT, where d is an abbreviation for diagonalization). We evaluated the performance of the proposed methods and their corresponding ensembles on benchmark datasets. The experiment shows that dGMML-DT achieves comparable or better classification results more efficiently than the UDTs with 10x average speedup. Furthermore, dGMML-DT can straightforwardly be extended to its multivariable counterpart (dGMML-MDT) without needing laborious operations.

Detecting abnormal nodes from attributed networks is of great importance in many real applications, such as financial fraud detection and cyber security. This task is challenging due to both the complex interactions between the anomalous nodes with other counterparts and their inconsistency in terms of attributes. This paper proposes a self-supervised learning framework that jointly optimizes a multi-view contrastive learning-based module and an attribute reconstruction-based module to more accurately detect anomalies on attributed networks. Specifically, two contrastive learning views are firstly established, which allow the model to better encode rich local and global information related to the abnormality. Motivated by the attribute consistency principle between neighboring nodes, a masked autoencoder-based reconstruction module is also introduced to identify the nodes which have large reconstruction errors, then are regarded as anomalies. Finally, the two complementary modules are integrated for more accurately detecting the anomalous nodes. Extensive experiments conducted on five benchmark datasets show our model outperforms current state-of-the-art models.

Adversarial training, originally designed to resist test-time adversarial examples, has shown to be promising in mitigating training-time availability attacks. This defense ability, however, is challenged in this paper. We identify a novel threat model named stability attack, which aims to hinder robust availability by slightly manipulating the training data. Under this threat, we show that adversarial training using a conventional defense budget $\epsilon$ provably fails to provide test robustness in a simple statistical setting, where the non-robust features of the training data can be reinforced by $\epsilon$-bounded perturbation. Further, we analyze the necessity of enlarging the defense budget to counter stability attacks. Finally, comprehensive experiments demonstrate that stability attacks are harmful on benchmark datasets, and thus the adaptive defense is necessary to maintain robustness.

In reality, learning from multi-view multi-label data inevitably confronts three challenges: missing labels, incomplete views, and non-aligned views. Existing methods mainly concern the first two and commonly need multiple assumptions to attack them, making even state-of-the-arts involve at least two explicit hyper-parameters such that model selection is quite difficult. More toughly, they will fail in handling the third challenge, let alone addressing the three jointly. In this paper, we aim at meeting these under the least assumption by building a concise yet effective model with just one hyper-parameter. To ease insufficiency of available labels, we exploit not only the consensus of multiple views but also the global and local structures hidden among multiple labels. Specifically, we introduce an indicator matrix to tackle the first two challenges in a regression form while aligning the same individual labels and all labels of different views in a common label space to battle the third challenge. In aligning, we characterize the global and local structures of multiple labels to be high-rank and low-rank, respectively. Subsequently, an efficient algorithm with linear time complexity in the number of samples is established. Finally, even without view-alignment, our method substantially outperforms state-of-the-arts with view-alignment on five real datasets.

In unsupervised domain adaptation (UDA), a target-domain model is trained by the supervised knowledge from a source domain. Although UDA has recently received much attention, most existing UDA methods have ignored the alignment in label space while mostly concentrating on alignment of feature space. Even worse, they have payed less attention to the dynamic relationship between domain alignment and discrimination, leading to degenerated performance. In this work, we propose a new kind of UDA through aligning in both the feature and label spaces (DAFL), in which a dynamic weight is designed and deployed between domain alignment and discrimination enhancement according to their conditions. Specifically, the cross-domain distribution divergence is reduced by the weighted class-level feature space alignment as well as the compacted and discriminative label space alignment. Furthermore, the balancing weight between adaptation alignment and discrimination enhancement is dynamically adjusted to regularize the adversarial domain adaptation. Then, the generalization ability of the DAFL model is enhanced by adding discrepant classification with theoretical analysis. Finally, extensive experiments validate effectiveness and superiority of the proposed approach.

Canonical correlation analysis (CCA) is a typical statistical model used to analyze the correlation components between different view representations of the same objects. When the label information is available with the data representations, CCA can be extended to its discriminative counterparts by incorporating supervision in the analysis. Although most discriminative variants of CCA have achieved improved results, nearly all of their objective functions are nonconvex, implying that optimal solutions are difficult to obtain. More important, that cross-view representations from the same sample should be consistent, that is, the cross-view semantic consistency has however not been modeled. To overcome these drawbacks, in this article, we propose a discriminant semantic correlation analysis (DSCA) model by modeling the cross-view semantic consistency for each object in the sample space rather than in the commonly used feature space. To boost the nonlinear discriminating capability of DSCA, we extend it from the Euclidean to the geodesic space by transforming the metric and incorporating both the cross-view semantic and representation correlation information and consequently obtain our final model with convex objective, namely, convex DSCA (C-DSCA). Finally, with extensive experiments and comparisons, we validate the effectiveness and superiority of the proposed method.

Domain adaptation (DA) aims at facilitating the target model training by leveraging knowledge from related but distribution-inconsistent source domain. Most of the previous DA works concentrate on homogeneous scenarios, where the source and target domains are assumed to share the same feature space. Nevertheless, frequently, in reality, the domains are not consistent in not only data distribution but also the representation space and feature dimensions. That is, these domains are heterogeneous. Although many works have attempted to handle such heterogeneous DA (HDA) by transforming HDA to homogeneous counterparts or performing DA jointly with domain transformation, nearly all of them just concentrate on the feature and distribution alignment across domains, neglecting the structure and classification space preservation for domains themselves. In this work, we propose a novel HDA model, namely, heterogeneous classification space alignment (HCSA), which leverages knowledge from both the source samples and model parameters to the target. In HCSA, structure preservation, distribution, and classification space alignment are implemented, jointly with feature representation by transferring both the source-domain representation and model knowledge. Moreover, we design an alternating algorithm to optimize the HCSA model with guaranteed convergence and complexity analysis. In addition, the HCSA model is further extended with deep network architecture. Finally, we experimentally evaluate the effectiveness of the proposed method by showing its superiority to the compared approaches.

In open set recognition (OSR), almost all existing methods are designed specially for recognizing individual instances, even these instances are collectively coming in batch. Recognizers in decision either reject or categorize them to some known class using empirically-set threshold. Thus the decision threshold plays a key role. However, the selection for it usually depends on the knowledge of known classes, inevitably incurring risks due to lacking available information from unknown classes. On the other hand, a more realistic OSR system should NOT just rest on a reject decision but should go further, especially for discovering the hidden unknown classes among the reject instances, whereas existing OSR methods do not pay special attention. In this paper, we introduce a novel collective/batch decision strategy with an aim to extend existing OSR for new class discovery while considering correlations among the testing instances. Specifically, a collective decision-based OSR framework (CD-OSR) is proposed by slightly modifying the Hierarchical Dirichlet process (HDP). Thanks to HDP, our CD-OSR does not need to define the decision threshold and can implement the open set recognition and new class discovery simultaneously. Finally, extensive experiments on benchmark datasets indicate the validity of CD-OSR.

In addition to high accuracy, robustness is becoming increasingly important for machine learning models in various applications. Recently, much research has been devoted to improving the model robustness by training with noise perturbations. Most existing studies assume a fixed perturbation level for all training examples, which however hardly holds in real tasks. In fact, excessive perturbations may destroy the discriminative content of an example, while deficient perturbations may fail to provide helpful information for improving the robustness. Motivated by this observation, we propose to adaptively adjust the perturbation levels for each example in the training process. Specifically, a novel active learning framework is proposed to allow the model interactively querying the correct perturbation level from human experts. By designing a cost-effective sampling strategy along with a new query type, the robustness can be significantly improved with a few queries. Both theoretical analysis and experimental studies validate the effectiveness of the proposed approach.

Delusive attacks aim to substantially deteriorate the test accuracy of the learning model by slightly perturbing the features of correctly labeled training examples. By formalizing this malicious attack as finding the worst-case training data within a specific $\infty$-Wasserstein ball, we show that minimizing adversarial risk on the perturbed data is equivalent to optimizing an upper bound of natural risk on the original data. This implies that adversarial training can serve as a principled defense against delusive attacks. Thus, the test accuracy decreased by delusive attacks can be largely recovered by adversarial training. To further understand the internal mechanism of the defense, we disclose that adversarial training can resist the delusive perturbations by preventing the learner from overly relying on non-robust features in a natural setting. Finally, we complement our theoretical findings with a set of experiments on popular benchmark datasets, which show that the defense withstands six different practical attacks. Both theoretical and empirical results vote for adversarial training when confronted with delusive adversaries.

This paper presents the first study on Sign-aware Perturbations Regression (SaPR), where the observed response variables contain the aware sign (negative or positive) perturbations.In order to predict the non-perturbation response variables, we propose a novel parameter estimator SZOM (i.e.,Setting Zero Operator Method), which aims at taking full advantage of the aware perturbations information to correct the mistake values in the estimation process with computationally efficiency.In this paper, the two aspects of theoretical analysis are proposed to deeply understand our method.Firstly, we establish the perturbation parameter error upper bound and prove consistency guarantee in the linear regression scenario.Secondly, we introduce the generalization error bound for the proposed SZMO, which indicates that the error bound is related to the value and the number of negative and positive perturbations.The effectiveness of the proposed approach is well validated by the experimental results on both synthetic and real datasets.

In real-world recognition/classification tasks, limited by various objective factors, it is usually difficult to collect training samples to exhaust all classes when training a recognizer or classifier. A more realistic scenario is open set recognition (OSR), where incomplete knowledge of the world exists at training time, and unknown classes can be submitted to an algorithm during testing, requiring the classifiers to not only accurately classify the seen classes, but also effectively deal with unseen ones. This paper provides a comprehensive survey of existing open set recognition techniques covering various aspects ranging from related definitions, representations of models, datasets, evaluation criteria, and algorithm comparisons. Furthermore, we briefly analyze the relationships between OSR and its related tasks including zero-shot, one-shot (few-shot) recognition/learning techniques, classification with reject option, and so forth. Additionally, we also review the open world recognition which can be seen as a natural extension of OSR. Importantly, we highlight the limitations of existing approaches and point out some promising subsequent research directions in this field.

Learning an expressive representation from multi-view data is a key step in various real-world applications. In this paper, we propose a semi-supervised multi-view deep discriminant representation learning (SMDDRL) approach. Unlike existing joint or alignment multi-view representation learning methods that cannot simultaneously utilize the consensus and complementary properties of multi-view data to learn inter-view shared and intra-view specific representations, SMDDRL comprehensively exploits the consensus and complementary properties as well as learns both shared and specific representations by employing the shared and specific representation learning network. Unlike existing shared and specific multi-view representation learning methods that ignore the redundancy problem in representation learning, SMDDRL incorporates the orthogonality and adversarial similarity constraints to reduce the redundancy of learned representations. Moreover, to exploit the information contained in unlabeled data, we design a semi-supervised learning framework by combining deep metric learning and density clustering. Experimental results on three typical multi-view learning tasks, i.e., webpage classification, image classification, and document classification demonstrate the effectiveness of the proposed approach.

Like k-means and Gaussian mixture model (GMM), fuzzy c-means (FCM) with soft partition has also become a popular clustering algorithm and still is extensively studied. However, these algorithms and their variants still suffer from some difficulties such as determination of the optimal number of clusters which is a key factor for clustering quality. A common approach for overcoming this difficulty is to use the trial-and-validation strategy, i.e., traversing every integer from large number like √n to 2 until finding the optimal number corresponding to the peak value of some cluster validity index. But it is scarcely possible to naturally construct an adaptively agglomerative hierarchical cluster structure as using the trial-and-validation strategy. Even if possible, existing different validity indices also lead to different number of clusters. To effectively mitigate the problems while motivated by convex clustering, in this article we present a centroid auto-fused hierarchical fuzzy c-means method (CAF-HFCM) whose optimization procedure can automatically agglomerate to form a cluster hierarchy, more importantly, yielding an optimal number of clusters without resorting to any validity index. Although a recently proposed robust-learning fuzzy c-means (RL-FCM) can also automatically obtain the best number of clusters without the help of any validity index, so-involved three hyperparameters need to adjust expensively, conversely, our CAF-HFCM involves just one hyperparameter which makes the corresponding adjustment relatively easier and more operational. Further, as an additional benefit from our optimization objective, the CAF-HFCM effectively reduces the sensitivity to the initialization of clustering performance. Moreover, our proposed CAF-HFCM method is able to be straightforwardly extended to various variants of FCM. Finally, extensive experiments on both synthetic and real data sets demonstrate the effectiveness and efficiency of CAF-HFCM.

Although the least-squares regression (LSR) has achieved great success in regression tasks, its discriminating ability is limited since the margins between classes are not specially preserved. To mitigate this issue, dragging techniques have been introduced to remodel the regression targets of LSR. Such variants have gained certain performance improvement, but their generalization ability is still unsatisfactory when handling real data. This is because structure-related information, which is typically contained in the data, is not exploited. To overcome this shortcoming, in this article, we construct a multioutput regression model by exploiting the intraclass correlations and input-output relationships via a structure matrix. We also discriminatively enlarge the regression margins by embedding a metric that is guided automatically by the training data. To better handle such structured data with ordinal labels, we encode the model output as cumulative attributes and, hence, obtain our proposed model, termed structure-exploiting discriminative ordinal multioutput regression (SEDOMOR). In addition, to further enhance its distinguishing ability, we extend the SEDOMOR to its nonlinear counterparts with kernel functions and deep architectures. We also derive the corresponding optimization algorithms for solving these models and prove their convergence. Finally, extensive experiments have testified the effectiveness and superiority of the proposed methods.

Graph-based semi-supervised learning (GSSL) studies the problem where in addition to a set of data points with few available labels, there also exists a graph structure that describes the underlying relationship between data items. In practice, structure uncertainty often occurs in graphs when edges exist between data with different labels, which may further results in prediction uncertainty of labels. Considering that Gaussian process generalizes well with few labels and can naturally model uncertainty, in this paper, we propose an Uncertainty aware Graph Gaussian Process based approach (UaGGP) for GSSL. UaGGP exploits the prediction uncertainty and label smooth regularization to guide each other during learning. To further subdue the effect of irrelevant neighbors, UaGGP also aggregates the clean representation in the original space and the learned representation. Experiments on benchmarks demonstrate the effectiveness of the proposed approach.

In the paper, we propose a class of accelerated stochastic gradient-free and projection-free (a.k.a., zeroth-order Frank-Wolfe) methods to solve the constrained stochastic and finite-sum nonconvex optimization. Specifically, we propose an accelerated stochastic zeroth-order Frank-Wolfe (Acc-SZOFW) method based on the variance reduced technique of SPIDER/SpiderBoost and a novel momentum accelerated technique. Moreover, under some mild conditions, we prove that the Acc-SZOFW has the function query complexity of $O(d\sqrt{n}\epsilon^{-2})$ for finding an $\epsilon$-stationary point in the finite-sum problem, which improves the exiting best result by a factor of $O(\sqrt{n}\epsilon^{-2})$, and has the function query complexity of $O(d\epsilon^{-3})$ in the stochastic problem, which improves the exiting best result by a factor of $O(\epsilon^{-1})$. To relax the large batches required in the Acc-SZOFW, we further propose a novel accelerated stochastic zeroth-order Frank-Wolfe (Acc-SZOFW*) based on a new variance reduced technique of STORM, which still reaches the function query complexity of $O(d\epsilon^{-3})$ in the stochastic problem without relying on any large batches. In particular, we present an accelerated framework of the Frank-Wolfe methods based on the proposed momentum accelerated technique. The extensive experimental results on black-box adversarial attack and robust black-box classification demonstrate the efficiency of our algorithms.

Objective-based clustering is a class of important clustering analysis techniques; however, these methods are easily beset by local minima due to the non-convexity of their objective functions involved, as a result, impacting final clustering performance. Recently, a convex clustering method (CC) has been on the spot light and enjoys the global optimality and independence on the initialization. However, one of its downsides is non-robustness to data contaminated with outliers, leading to a deviation of the clustering results. In order to improve its robustness, in this paper, an outlier-aware robust convex clustering algorithm, called as RCC, is proposed. Specifically, RCC extends the CC by modeling the contaminated data as the sum of the clean data and the sparse outliers and then adding a Lasso-type regularization term to the objective of the CC to reflect the sparsity of outliers. In this way, RCC can both resist the outliers to great extent and still maintain the advantages of CC, including the convexity of the objective. Further we develop a block coordinate descent approach with the convergence guarantee and find that RCC can usually converge just in a few iterations. Finally, the effectiveness and robustness of RCC are empirically corroborated by numerical experiments on both synthetic and real datasets.

These days, many multiview learning methods have been proposed by integrating the complementary information of multiple views and can significantly improve the performance of machine learning tasks comparing with single-view learning methods. However, most of these methods fail to learn better models when the multiview data are unpaired (or partially paired) or incomplete (or partially complete). Although some previous attempts have been made to address these problems, these methods often lead to poor results when dealing with incomplete multiview data that contain a relatively large number of missing instances. In fact, this incomplete problem is more challenging than the unpaired problem since less shared information can be caught by the model in the former case. In this paper, we propose a shared Gaussian process (GP) latent variable model for incomplete multiview clustering to gain the merits of two worlds (i.e., GP and multiview learning). Specifically, it learns a set of intentionally aligned representative auxiliary points in individual views jointly to not only compensate for missing instances but also implement the group-level constraint. Thus, the shared information among these views can be explicitly built into the model. All of the hyper-parameters and auxiliary points are simultaneously learned by variational inference. Compared with the existing methods, our method naturally inherits the advantages of GP. Furthermore, it is also straightforwardly extended to cases with more than two views without adding any complexity in formulation. In the experiments, we compare it with the state-of-the-art methods for incomplete multiview data clustering to demonstrate its superiorities.

Canonical correlation analysis (CCA) is a typical and useful learning paradigm in big data analysis for capturing correlation across multiple views of the same objects. When dealing with data with additional ordinal information, traditional CCA suffers from poor performance due to ignoring the ordinal relationships within the data. Such data is becoming increasingly common, as either temporal or sequential information is often associated with the data collection process. To incorporate the ordinal information into the objective function of CCA, the so-called ordinal discriminative CCA has been presented in the literature. Although ordinal discriminative CCA can yield better ordinal regression results, its performance deteriorates when data is corrupted with noise and outliers, as it tends to smear the order information contained in class centers. To address this issue, in this article we construct a robust manifold-preserved ordinal discriminative correlation regression (rmODCR). The robustness is achieved by replacing the traditional ( l 2 -norm) class centers with l p -norm centers, where p is efficiently estimated according to the moments of the data distributions, as well as by incorporating the manifold distribution information of the data in the objective optimization. In addition, we further extend the robust manifold-preserved ordinal discriminative correlation regression to deep convolutional architectures. Extensive experimental evaluations have demonstrated the superiority of the proposed methods.

Real data are often with multiple modalities or from multiple heterogeneous sources, thus forming so-called multi-view data, which receives more and more attentions in machine learning. Multi-view clustering (MVC) becomes its important paradigm. In real-world applications, some views often suffer from instances missing. Clustering on such multi-view datasets is called incomplete multi-view clustering (IMC) and quite challenging. To date, though many approaches have been developed, most of them are offline and have high computational and memory costs especially for large scale datasets. To address this problem, in this paper, we propose an One-Pass Incomplete Multi-view Clustering framework (OPIMC). With the help of regularized matrix factorization and weighted matrix factorization, OPIMC can relatively easily deal with such problem. Different from the existing and sole online IMC method, OPIMC can directly get clustering results and effectively determine the termination of iteration process by introducing two global statistics. Finally, extensive experiments conducted on four real datasets demonstrate the efficiency and effectiveness of the proposed OPIMC method.

Proximal gradient method has been playing an important role to solve many machine learning tasks, especially for the nonsmooth problems. However, in some machine learning problems such as the bandit model and the black-box learning problem, proximal gradient method could fail because the explicit gradients of these problems are difficult or infeasible to obtain. The gradient-free (zeroth-order) method can address these problems because only the objective function values are required in the optimization. Recently, the first zeroth-order proximal stochastic algorithm was proposed to solve the nonconvex nonsmooth problems. However, its convergence rate is O(1/√T) for the nonconvex problems, which is significantly slower than the best convergence rate O(T1) of the zerothorder stochastic algorithm, where T is the iteration number. To fill this gap, in the paper, we propose a class of faster zeroth-order proximal stochastic methods with the variance reduction techniques of SVRG and SAGA, which are denoted as ZO-ProxSVRG and ZO-ProxSAGA, respectively. In theoretical analysis, we address the main challenge that an unbiased estimate of the true gradient does not hold in the zerothorder case, which was required in previous theoretical analysis of both SVRG and SAGA. Moreover, we prove that both ZO-ProxSVRG and ZO-ProxSAGA algorithms have O(T1) convergence rates. Finally, the experimental results verify that our algorithms have a faster convergence rate than the existing zeroth-order proximal stochastic algorithm.

In heterogeneous domain adaptation (HDA), since the feature spaces of the source and target domains are different, knowledge transfer from the source to the target domain is really challenging. How to align the different feature spaces and then adaptively transfer the related knowledge is critical for HDA. In this paper, we develop an adaptive teacher-and-student model for heterogeneous domain adaptation (AtsHDA). In AtsHDA, the source domain as a teacher and the target domain as a student are aligned or co-adapted to each other first, so that their correlation can be maximized. Then the target domain adaptively learns from the source domain. Specifically, there is a balance between the learning by the target domain itself and the instruction from the source domain. That is, when the guidance from the source domain is helpful for learning, the learning of target classifier emphasizes the instruction of source knowledge, and considers its own knowledge more, otherwise. Further, an ensemble method is designed to decide such a balance. Finally, empirical results show that AtsHDA can achieve competitive results compared with the state-of-arts.

In this paper, we propose a faster stochastic alternating direction method of multipliers (ADMM) for nonconvex optimization by using a new stochastic path-integrated differential estimator (SPIDER), called as SPIDER-ADMM. Moreover, we prove that the SPIDER-ADMM achieves a record-breaking incremental first-order oracle (IFO) complexity for finding an $\epsilon$-approximate solution. As one of major contribution of this paper, we provide a new theoretical analysis framework for nonconvex stochastic ADMM methods with providing the optimal IFO complexity. Based on this new analysis framework, we study the unsolved optimal IFO complexity of the existing non-convex SVRG-ADMM and SAGA-ADMM methods, and prove their the optimal IFO complexity. Thus, the SPIDER-ADMM improves the existing stochastic ADMM methods. Moreover, we extend SPIDER-ADMM to the online setting, and propose a faster online SPIDER-ADMM. Our theoretical analysis also derives the IFO complexity of the online SPIDER-ADMM. Finally, the experimental results on benchmark datasets validate that the proposed algorithms have faster convergence rate than the existing ADMM algorithms for nonconvex optimization.

Alternating direction method of multipliers (ADMM) is a popular optimization tool for the composite and constrained problems in machine learning. However, in many machine learning problems such as black-box learning and bandit feedback, ADMM could fail because the explicit gradients of these problems are difficult or even infeasible to obtain. Zeroth-order (gradient-free) methods can effectively solve these problems due to that the objective function values are only required in the optimization. Recently, though there exist a few zeroth-order ADMM methods, they build on the convexity of objective function. Clearly, these existing zeroth-order methods are limited in many applications. In the paper, thus, we propose a class of fast zeroth-order stochastic ADMM methods (\emph{i.e.}, ZO-SVRG-ADMM and ZO-SAGA-ADMM) for solving nonconvex problems with multiple nonsmooth penalties, based on the coordinate smoothing gradient estimator. Moreover, we prove that both the ZO-SVRG-ADMM and ZO-SAGA-ADMM have convergence rate of $O(1/T)$, where $T$ denotes the number of iterations. In particular, our methods not only reach the best convergence rate of $O(1/T)$ for the nonconvex optimization, but also are able to effectively solve many complex machine learning problems with multiple regularized penalties and constraints. Finally, we conduct the experiments of black-box binary classification and structured adversarial attack on black-box deep neural network to validate the efficiency of our algorithms.

Gaussian Process Latent Variable Model (GPLVM) is a powerful nonlinear dimension reduction model and has been widely used in many machine learning scenarios. However, the original GPLVM and its variants do not explicitly model the correlations among the original features, leading to the underutilization of underlying information involved in the data. To compensate for this shortcoming, we propose a feature-correlation-aware GPLVM (fcaGPLVM) to simultaneously learn the latent variables and the feature correlations. The main contributions of this paper are 1) introducing a set of extra latent variables into the original GPLVM and proposing a feature-correlation-aware kernel function to explicitly model the feature-description information and infer the feature correlations; 2) defining a joint objective function and developing a stochastic optimization algorithm based on the stochastic variational inference (SVI) to learn all the latent variables. To the best of our knowledge, this is the first work that explicitly considers the feature correlations in the GPLVM and makes many existing GPLVMs become its special cases. Furthermore, it can be applied to both unsupervised and supervised learnings to improve the performance of dimension reduction. Experimental results show that in these two learning scenarios the proposed models outperform their state-of-the-art counterparts.

Nowadays, multi-view clustering has attracted more and more attention. To date, almost all the previous studies assume that views are complete. However, in reality, it is often the case that each view may contain some missing instances. Such incompleteness makes it impossible to directly use traditional multi-view clustering methods. In this paper, we propose a Doubly Aligned Incomplete Multi-view Clustering algorithm (DAIMC) based on weighted semi-nonnegative matrix factorization (semi-NMF). Specifically, on the one hand, DAIMC utilizes the given instance alignment information to learn a common latent feature matrix for all the views. On the other hand, DAIMC establishes a consensus basis matrix with the help of L2,1-Norm regularized regression for reducing the influence of missing instances. Consequently, compared with existing methods, besides inheriting the strength of semi-NMF with ability to handle negative entries, DAIMC has two unique advantages: 1) solving the incomplete view problem by introducing a respective weight matrix for each view, making it able to easily adapt to the case with more than two views; 2) reducing the influence of view incompleteness on clustering by enforcing the basis matrices of individual views being aligned with the help of regression. Experiments on four real-world datasets demonstrate its advantages.

Semi-supervised learning is especially important in data mining applications because it can make use of plentiful unlabeled data to train the high-quality learning models. Semi-Supervised Support Vector Machine (S3VM) is a powerful semi-supervised learning model. However, the high computational cost and non-convexity severely impede the S3VM method in large-scale applications. Although several learning algorithms were proposed for S3VM, scaling up S3VM is still an open problem. To address this challenging problem, in this paper, we propose a new incremental learning algorithm to scale up S3VM (IL-S3VM) based on the path following technique in the framework of Difference of Convex (DC) programming. The traditional DC programming based algorithms need multiple outer loops and are not suitable for incremental learning, and traditional path following algorithms are limited to convex problems. Our new IL-S3VM algorithm based on the path-following technique can directly update the solution of S3VM to converge to a local minimum within one outer loop so that the efficient incremental learning can be achieved. More importantly, we provide the finite convergence analysis for our new algorithm. To the best of our knowledge, our new IL-S3VM algorithm is the first efficient path following algorithm for a non-convex problem (i.e., S3VM) with local minimum convergence guarantee. Experimental results on a variety of benchmark datasets not only confirm the finite convergence of IL-S3VM, but also show a huge reduction of computational time compared with existing batch and incremental learning algorithms, while retaining the similar generalization performance.

Feature missing is a serious problem in many applications, which may lead to low quality of training data and further significantly degrade the learning performance. While feature acquisition usually involves special devices or complex process, it is expensive to acquire all feature values for the whole dataset. On the other hand, features may be correlated with each other, and some values may be recovered from the others. It is thus important to decide which features are most informative for recovering the other features as well as improving the learning performance. In this paper, we try to train an effective classification model with least acquisition cost by jointly performing active feature querying and supervised matrix completion. When completing the feature matrix, a novel objective function is proposed to simultaneously minimize the reconstruction error on observed entries and the supervised loss on training data. When querying the feature value, the most uncertain entry is actively selected based on the variance of previous iterations. In addition, a bi-objective optimization method is presented for cost-aware active selection when features bear different acquisition costs. The effectiveness of the proposed approach is well validated by both theoretical analysis and experimental study.

In the paper, we propose a class of dynamic conditional Gaussian graphical models (DCGGMs) based on a set of nonidentical distribution observations, which changes smoothly with time or condition. Specifically, the DCGGMs model the dynamic output network influenced by conditioning input variables, which are encoded by a set of varying parameters. Moreover, we propose a joint smooth graphical Lasso to estimate the DCGGMs, which combines kernel smoother with sparse group Lasso penalty. At the same time, we design an efficient accelerated proximal gradient algorithm to solve this estimator. Theoretically, we establish the asymptotic properties of our model on consistency and sparsistency under the high-dimensional settings. In particular, we highlight a class of consistency theory for dynamic graphical models, in which the sample size can be seen as n4/5for estimating a local graphical model when the bandwidth parameter h of kernel smoother is chosen as h = n-1/5for describing the dynamic. Finally, the extensive numerical experiments on both synthetic and real datasets are provided to support the effectiveness of the proposed method.

The O6-methylguanine-DNA methyltransferase (MGMT) promoter methylation and isocitrate dehydrogenase 1 (IDH1) mutation in high-grade gliomas (HGG) have proven to be the two important molecular indicators associated with better prognosis. Traditionally, the statuses of MGMT and IDH1 are obtained via surgical biopsy, which has limited their wider clinical implementation. Accurate presurgical prediction of their statuses based on preoperative multimodal neuroimaging is of great clinical value for a better treatment plan. Currently, the available data set associated with this study has several challenges, such as small sample size and complex, nonlinear (image) feature-to-(molecular) label relationship. To address these issues, we propose a novel multi-label nonlinear matrix completion (MNMC) model to jointly predict both MGMT and IDH1 statuses in a multi-task framework. Specifically, we first employ a nonlinear random Fourier feature mapping to improve the linear separability of the data, and then use transductive multi-task feature selection (performed in a nonlinearly transformed feature space) to refine the imputed soft labels, thus alleviating the overfitting problem caused by small sample size. We further design an optimization algorithm with a guaranteed convergence ability based on a block prox-linear method to solve the proposed MNMC model. Finally, by using a single-center, multimodal brain imaging and molecular pathology data set of HGG, we derive brain functional and structural connectomics features to jointly predict MGMT and IDH1 statuses. Results demonstrate that our proposed method outperforms the previously widely used single- and multi-task machine learning methods. This paper also shows the promise of utilizing brain connectomics for HGG prognosis in a non-invasive manner.

For a multicategory classification problem, discriminative least squares regression (DLSR) explicitly introduces an -dragging technique to enlarge the margin between the categories, yielding superior classification performance from a margin perspective. In this brief, we reconsider this classification problem from a metric learning perspective and propose a framework of metric learning-guided least squares classifier (MLG-LSC) learning. The core idea is to learn a unified metric matrix for the error of LSR, such that such a metric matrix can yield small distances for the same category, while large ones for the different categories. As opposed to the -dragging in DLSR, we call this the error-dragging (e-dragging). Different from DLSR and its related variants, our MLG-LSC implicitly carries out the e-dragging and can naturally reflect the roughly relative distance relationships among the categories from a metric learning perspective. Furthermore, our optimization objective functions are strictly (geodesically) convex and thus can obtain their corresponding closed-form solutions, resulting in higher computational performance. Experimental results on a set of benchmark data sets indicate the validity of our learning framework.

In this paper, we propose a joint conditional graphical Lasso to learn multiple conditional Gaussian graphical models, also known as Gaussian conditional random fields, with some similar structures. Our model builds on the maximum likelihood method with the convex sparse group Lasso penalty. Moreover, our model is able to model multiple multivariate linear regressions with unknown noise covariances via a convex formulation. In addition, we develop an efficient approximated Newton's method for optimizing our model. Theoretically, we establish the asymptotic properties of our model on consistency and sparsistency under the high-dimensional settings. Finally, extensive numerical results on simulations and real data sets demonstrate that our method outperforms the compared methods on structure recovery and structured output prediction. To the best of our knowledge, the joint learning of multiple multivariate regressions with unknown covariance is first studied.

Webpage classification has attracted a lot of research interest. Webpage data is often multi-view and high-dimensional, and the webpage classification application is usually semi-supervised. Due to these characteristics, using semi-supervised multi-view feature learning (SMFL) technique to deal with the webpage classification problem has recently received much attention. However, there still exists room for improvement for this kind of feature learning technique. How to effectively utilize the correlation information among multi-view of webpage data is an important research topic. Correlation analysis on multi-view data can facilitate extraction of the complementary information. In this paper, we propose a novel SMFL approach, named semi-supervised multi-view correlation feature learning (SMCFL), for webpage classification. SMCFL seeks for a discriminant common space by learning a multi-view shared transformation in a semi-supervised manner. In the discriminant space, the correlation between intra-class samples is maximized, and the correlation between inter-class samples and the global correlation among both labeled and unlabeled samples are minimized simultaneously. We transform the matrix-variable based nonconvex objective function of SMCFL into a convex quadratic programming problem with one real variable, and can achieve a global optimal solution. Experiments on widely used datasets demonstrate the effectiveness and efficiency of the proposed approach.

Multi-instance multi-label learning(MIML) has been successfully applied into many real-world applications. Along with the enhancing of the expressive power, the cost of labelling a MIML example increases significantly. And thus it becomes an important task to train an effective MIML model with as few labelled examples as possible. Active learning, which actively selects the most valuable data to query their labels, is a main approach to reducing labeling cost. Existing active methods achieved great success in traditional learning tasks, but cannot be directly applied to MIML problems. In this paper, we propose a MIML active learning algorithm, which exploits diversity and uncertainty in both the input and output space to query the most valuable information. This algorithm designs a novel query strategy for MIML objects specifically and acquires more precise information from the oracle without addition cost. Based on the queried information, the MIML model is then effectively trained by simultaneously optimizing the relative rank among instances and labels.

Human age estimation is an important research topic and can find its applications in such as commodity recommendation and security monitoring. The establishment of existing estimators basically follows a same pipeline, i.e., an estimator is built from a given training dataset like FG-NET and then evaluated on a holdout testing set to determine its effectiveness. In doing so, a usually-followed assumption is that both training and testing sets should share the same age distribution and the same feature representation, implying that 1) once the true age of a human image to be tested is out of the age range of the training set, a mis-estimation is naturally inevitable; 2) estimators built on datasets with different feature representations cannot be directly applied to make predictions on testing sets of each other unless re-trained, because their features are usually different (i.e., the databases are heterogeneous). To the best of our knowledge, the age distributions of different aging databases are usually not consistent and complementary to each other. Motivated by this fact, in order to incorporate such a complementarity in age distributions to improve the generalization ability of the age estimator, in this paper we propose a unified cross-heterogeneous-database age estimation method by first projecting the training samples, usually represented with different features, of different aging databases into a common feature space, and then constructing an age estimator in their mixed sample space. By this way, the age-distribution-incompleteness of the aging datasets can be alleviated by co-representation among them and thus the discriminating ability of the age estimator can be reinforced. Finally, experimental results demonstrate the superiority of the proposed method.

Error-correcting output coding (ECOC) is one of the most widely used strategies for dealing with multi-class problems by decomposing the original multi-class problem into a series of binary sub-problems. In traditional ECOC-based methods, binary classifiers corresponding to those sub-problems are usually trained separately without considering the relationships among these classifiers. However, as these classifiers are established on the same training data, there may be some inherent relationships among them. Exploiting such relationships can potentially improve the generalization performances of individual classifiers, and, thus, boost ECOC learning algorithms. In this paper, we explore to mine and utilize such relationship through a joint classifier learning method, by integrating the training of binary classifiers and the learning of the relationship among them into a unified objective function. We also develop an efficient alternating optimization algorithm to solve the objective function. To evaluate the proposed method, we perform a series of experiments on eleven datasets from the UCI machine learning repository as well as two datasets from real-world image recognition tasks. The experimental results demonstrate the efficacy of the proposed method, compared with state-of-the-art methods for ECOC-based multi-class classification.

One major challenge in computer vision is to go beyond the modeling of individual objects and to investigate the bi- (one-versus-one) or tri- (one-versus-two) relationship among multiple visual entities, answering such questions as whether a child in a photo belongs to the given parents. The child-parents relationship plays a core role in a family, and understanding such kin relationship would have a fundamental impact on the behavior of an artificial intelligent agent working in the human world. In this work, we tackle the problem of one-versus-two (tri-subject) kinship verification and our contributions are threefold: 1) a novel relative symmetric bilinear model (RSBM) is introduced to model the similarity between the child and the parents, by incorporating the prior knowledge that a child may resemble one particular parent more than the other; 2) a spatially voted method for feature selection, which jointly selects the most discriminative features for the child-parents pair, while taking local spatial information into account; and 3) a large-scale tri-subject kinship database characterized by over 1,000 child-parents families. Extensive experiments on KinFaceW, Family101, and our newly released kinship database show that the proposed method outperforms several previous state of the art methods, while could also be used to significantly boost the performance of one-versus-one kinship verification when the information about both parents are available.

We consider joint learning of multiple sparse matrix Gaussian graphical models and propose the joint matrix graphical Lasso to discover the conditional independence structures among rows (columns) in the matrix variable under distinct conditions. The proposed approach borrows strength across the different graphical models and is based on the maximum likelihood with penalized row and column precision matrices, respectively. In particular, our model is more parsimonious and flexible than the joint vector graphical models. Furthermore, we establish the asymptotic properties of our model on consistency and sparsistency. And the asymptotic analysis shows that our model enjoys a better convergence rate than the joint vector graphical models. Extensive simulation experiments demonstrate that our methods outperform state-of-the-art methods in identifying graphical structures and estimating precision matrices. Moreover, the effectiveness of our methods is also illustrated via a real data set analysis. Sparsistency is shorthand for consistency of the sparsity pattern of the parameters.

Two factors characterize a good feature selection algorithm: its accuracy and stability. This paper aims at introducing a new approach to stable feature selection algorithms. The innovation of this paper centers on a class of stable feature selection algorithms called feature weighting as regularized energy-based learning (FREL). Stability properties of FREL using L1 or L2 regularization are investigated. In addition, as a commonly adopted implementation strategy for enhanced stability, an ensemble FREL is proposed. A stability bound for the ensemble FREL is also presented. Our experiments using open source real microarray data, which are challenging high dimensionality small sample size problems demonstrate that our proposed ensemble FREL is not only stable but also achieves better or comparable accuracy than some other popular stable feature weighting methods.

Non-Local Euclidean Medians (NLEM) has recently been proposed and shows more effective than Non-Local Means (NLM) in removing heavy noise. In this letter, we find the inconsistency between the two dissimilarity measures in NLEM can affect its robustness, thus develop an improved version (INLEM) to compensate such an inconsistency. Further, we provide a concise convergence proof for the iterative algorithm used in both NLEM and INLEM. Finally, our experiments on synthetic and natural images show that INLEM achieves encouraging results.

Though semi-supervised classification learning has attracted great attention over past decades, semi-supervised classification methods may show worse performance than their supervised counterparts in some cases, consequently reducing their confidence in real applications. Naturally, it is desired to develop a safe semi-supervised classification method that never performs worse than the supervised counterparts. However, to the best of our knowledge, few researches have been devoted to safe semi-supervised classification. To address this problem, in this paper, we invent a safety-control mechanism for safe semi-supervised classification by adaptive tradeoff between semi-supervised and supervised classification in terms of unlabeled data. In implementation, based on our recent semi-supervised classification method based on class memberships (SSCCM), we develop a safety-aware SSCCM (SA-SSCCM). SA-SSCCM, on the one hand, exploits the unlabeled data to help learning (as SSCCM does) under the assumption that unlabeled data can help learning, and on the other hand, restricts its prediction to approach that of its supervised counterpart least-square support vector machine (LS-SVM) under the assumption that unlabeled data can hurt learning. Therefore, prediction by SA-SSCCM becomes a tradeoff between those by semi-supervised SSCCM and supervised LS-SVM, respectively, in terms of the unlabeled data. As in SSCCM, the optimization problem in SA-SSCCM can be efficiently solved by the alternating iterative strategy, and the iteration convergence can theoretically be guaranteed. Experiments over several real datasets show the promising performance of SA-SSCCM compared with LS-SVM, SSCCM, and off-the-shelf safe semi-supervised classification methods.

Recently, besides the performance, the stability (robustness, i.e., the variation in feature selection results due to small changes in the data set) of feature selection is received more attention. Ensemble feature selection where multiple feature selection outputs are combined to yield more robust results without sacrificing the performance is an effective method for stable feature selection. In order to make further improvements of the performance (classification accuracy), the diversity regularized ensemble feature weighting framework is presented, in which the base feature selector is based on local learning with logistic loss for its robustness to huge irrelevant features and small samples. At the same time, the sample complexity of the proposed ensemble feature weighting algorithm is analyzed based on the VC-theory. The experiments on different kinds of data sets show that the proposed ensemble method can achieve higher accuracy than other ensemble ones and other stable feature selection strategy (such as sample weighting) without sacrificing stability

Abstract Recent work has shown the advantages of using high level representation such as attribute-based descriptors over low-level feature sets in face verification. However, in most work each attribute is coded with extremely short information length (e.g., “is Male”, “has Beard”) and all the attributes belonging to the same object are assumed to be independent of each other when using them for prediction. To address the above two problems, we propose a discriminative distributed-representation for attribute description; on the basis of this description, we present a novel method to model the relationship between attributes and exploit such relationship to improve the performance of face verification, in the meantime taking uncertainty in attribute responses into account. Specifically, inspired by the vector representation of words in the literature of text categorization, we first represent the meaning of each attribute as a high-dimensional vector in the subject space, then construct an attribute-relationship graph based on the distribution of attributes in that space. With this graph, we are able to explicitly constrain the searching space of parameter values of a discriminative classifier to avoid over-fitting. The effectiveness of the proposed method is verified on two challenging face databases (i.e., LFW and PubFig) and the a-Pascal object dataset. Furthermore, we extend the proposed method to the case with continuous attributes with promising results.

The cluster assumption, which assumes that "similar instances should share the same label," is a basic assumption in semi-supervised classification learning, and has been found very useful in many successful semi-supervised classification methods. It is rarely noticed that when the cluster assumption is adopted, there is an implicit assumption that every instance should have a crisp class label assignment. In real applications, however, there are cases where it is difficult to tell that an instance definitely belongs to one class and does not belong to other neighboring classes. In such cases, it is more adequate to assume that "similar instances should share similar label memberships" rather than sharing a crisp label assignment. Here "label memberships" can be represented as a vector, where each element corresponds to a class, and the value at the element expresses the likelihood of the concerned instance belonging to the class. By adopting this modified cluster assumption, in this paper we propose a new semi-supervised classification method, that is, semi-supervised classification based on class membership (SSCCM). Specifically, we try to solve the decision function and adequate label memberships for instances simultaneously, and constrain that an instance and its "local weighted mean" (LWM) share the same label membership vector, where the LWM is a robust image of the instance, constructed by calculating the weighted mean of its neighboring instances. We formulate the problem in a unified objective function for the labeled, unlabeled data and their LWMs based on the square loss function, and take an alternating iterative strategy to solve it, in which each step generates a closed-form solution, and the convergence is guaranteed. The solution will provide both the decision function and the label membership function for classification, their classification results can verify each other, and the reliability of semi-supervised classification learning might be enhanced by checking the consistency between those two predictions. Experiments show that SSCCM obtains encouraging results compared to state-of-the-art semi-supervised classification methods.

ABSTRACT Signal‐strength‐based location estimation in wireless sensor networks is to locate the physical positions of unknown sensorsviathe received signal strengths. In this field, there are few localization researches sufficiently exploiting topology structures of the network in both signal space and physical space. The goal of this paper is to first establish two effective localization models based on specific manifold (or local) structures of both signal space and physical (location) space by using our previous locality preserving canonical correlation analysis (LPCCA) model and a newly‐proposed locality correlation analysis (LCA) model, and then develop their corresponding novel location algorithms, called location estimation—LPCCA (LE—LPCCA) and location estimation—LCA (LE—LCA). Since both LPCCA and LCA relatively sufficiently take into account locality characteristics of the manifold structures in both the spaces, our localization algorithms developed from them consequently achieve better localization accuracy than other publicly available advanced algorithms. Copyright © 2011 John Wiley & Sons, Ltd.

Support vector machine (SVM), as one of the most popular classifiers, aims to find a hyperplane that can separate two classes of data with maximal margin. SVM classifiers are focused on achieving more separation between classes than exploiting the structures in the training data within classes. However, the structural information, as an implicit prior knowledge, has recently been found to be vital for designing a good classifier in different real-world problems. Accordingly, using as much prior structural information in data as possible to help improve the generalization ability of a classifier has yielded a class of effective structural large margin classifiers, such as the structured large margin machine (SLMM) and the Laplacian support vector machine (LapSVM). In this paper, we unify these classifiers into a common framework from the concept of "structural granularity" and the formulation for optimization problems. We exploit the quadratic programming (QP) and second-order cone programming (SOCP) methods, and derive a novel large margin classifier, we call the new classifier the structural regularized support vector machine (SRSVM). Unlike both SLMM at the cross of the cluster granularity and SOCP and LapSVM at the cross of the point granularity and QP, SRSVM is located at the cross of the cluster granularity and QP and thus follows the same optimization formulation as LapSVM to overcome large computational complexity and non-sparse solution in SLMM. In addition, it integrates the compactness within classes with the separability between classes simultaneously. Furthermore, it is possible to derive generalization bounds for these algorithms by using eigenvalue analysis of the kernel matrices. Experimental results demonstrate that SRSVM is often superior in classification and generalization performances to the state-of-the-art algorithms in the framework, both with the same and different structural granularities.

Localizing sensors based on Received Signal Strength Indicator (RSSI) localization technique in wireless sensor network can be treated as building a mapping between signal and physical spaces, and the mapping is established from a set of given paired signal strengths and physical location data of known sensors. However, in some realistic scenarios, such a set of completely-paired sensor data is not always accessible, which brings a big challenge for localization of sensors. The localization research in such a scenario is currently almost ignored. In this paper, we develop a novel algorithm to tackle this problem in localization with paired as well as many unpaired data by adapting our previously-proposed Locality Correlation Analysis model; the new algorithm is named as Partially Paired Locality Correlation Analysis (PPLCA). Experimental results in both outdoor and indoor environments do show the feasibility and effectiveness of the proposed algorithm.

AUC-SVM directly maximizes the area under the ROC curve (AUC) through minimizing its hinge loss relaxation, and the decision function is determined by those support vector sample pairs playing the same roles as the support vector samples in SVM. Such a learning paradigm generally emphasizes more on the local discriminative information just associated with these support vectors whereas hardly takes the overall view of data into account, thereby it may incur loss of the global distribution information in data favorable for classification. Moreover, due to the high computational complexity of AUC-SVM induced by the large number of training sample pairs quadratic in the number of samples, sampling is usually adopted, incurring a further loss of the distribution information in data. In order to compensate the distribution information loss and simultaneously boost the AUC-SVM performance, in this paper, we develop a novel structure-embedded AUC-SVM (SAUC-SVM for short) through embedding the global structure information in the whole data into AUC-SVM. With such an embedding, the proposed SAUC-SVM incorporates the local discriminative information and global structure information in data into a uniform formulation and consequently guarantees better generalization performance. Comparative experiments on both synthetic and real datasets confirm its effectiveness.

Many locality-based unsupervised dimensionality reduction (DR) algorithms have recently been proposed and demonstrated to be effective to a certain degree in some classification tasks. In this paper, we aim to show that: (1) a discriminant disposal is intentionally or unintentionally induced from the construction of locality in these unsupervised algorithms, however, such a discrimination is often inconsistent with the actual class information, so here called disguised discrimination; (2) sensitivities of these algorithms to local neighbor parameters stem from the inconsistency between the disguised discrimination and the actual class information; (3) how such inconsistency impacts the classification performance of these algorithms. The experiments on the benchmark face datasets testify our statements that are expected to provide some insight into the unsupervised leaning based on locality.

Traditional pattern recognition involves two tasks: clustering learning and classification learning. Clustering result can enhance the generalization ability of classification learning, while the class information can improve the accuracy of clustering learning. Hence, both learning methods can complement each other. To fuse the advantages of both learning methods together, many existing algorithms have been developed in a sequential fusing way by first optimizing the clustering criterion and then the classification criterion associated with the obtained clustering results. However, such kind of algorithms naturally fails to achieve the simultaneous optimality for two criteria, and thus has to sacrifice either the clustering performance or the classification performance. To overcome that problem, in this paper, we present a multiobjective simultaneous learning framework (MSCC) for both clustering and classification learning. MSCC utilizes multiple objective functions to formulate the clustering and classification problems, respectively, and more importantly, it employs the Bayesian theory to make these functions all only dependent on a set of the same parameters, i.e., clustering centers which play a role of the bridge connecting the clustering and classification learning. By simultaneously optimizing the clustering centers embedded in these functions, not only the effective clustering performance but also the promising classification performance can be simultaneously attained. Furthermore, from the multiple Pareto-optimality solutions obtained in MSCC, we can get an interesting observation that there is complementarity to great extent between clustering and classification learning processes. Empirical results on both synthetic and real data sets demonstrate the effectiveness and potential of MSCC.

The goal of semisupervised kernel matrix learning (SS-KML) is to learn a kernel matrix on all the given samples on which just a little supervised information, such as class label or pairwise constraint, is provided. Despite extensive research, the performance of SS-KML still leaves some space for improvement in terms of effectiveness and efficiency. For example, a recent pairwise constraints propagation (PCP) algorithm has formulated SS-KML into a semidefinite programming (SDP) problem, but its computation is very expensive, which undoubtedly restricts PCPs scalability in practice. In this paper, a novel algorithm, called kernel propagation (KP), is proposed to improve the comprehensive performance in SS-KML. The main idea of KP is first to learn a small-sized sub-kernel matrix (named seed-kernel matrix) and then propagate it into a larger-sized full-kernel matrix. Specifically, the implementation of KP consists of three stages: 1) separate the supervised sample (sub)set X(l) from the full sample set X; 2) learn a seed-kernel matrix on X(l) through solving a small-scale SDP problem; and 3) propagate the learnt seed-kernel matrix into a full-kernel matrix on X . Furthermore, following the idea in KP, we naturally develop two conveniently realizable out-of-sample extensions for KML: one is batch-style extension, and the other is online-style extension. The experiments demonstrate that KP is encouraging in both effectiveness and efficiency compared with three state-of-the-art algorithms and its related out-of-sample extensions are promising too.

Compared to singular value decomposition (SVD), generalized low-rank approximations of matrices (GLRAM) can consume less computation time, obtain higher compression ratio, and yield competitive classification performance. GLRAM has been successfully applied to applications such as image compression and retrieval, and quite a few extensions have been successively proposed. However, in literature, some basic properties and crucial problems with regard to GLRAM have not been explored or solved yet. For this sake, we revisit GLRAM in this paper. First, we reveal such a close relationship between GLRAM and SVD that GLRAM's objective function is identical to SVD's objective function except the imposed constraints. Second, we derive a lower bound of GLRAM's objective function, and discuss when the lower bound can be touched. Moreover, from the viewpoint of minimizing the lower bound, we answer one open problem raised by Ye (Machine Learning, 2005), i.e., a theoretical justification of the experimental phenomenon that, under given number of reduced dimension, the lowest reconstruction error is obtained when the left and right transformations have equal number of columns. Third, we explore when and why GLRAM can perform well in terms of compression, which is a fundamental problem concerning the usability of GLRAM.

Traditional ranking mainly focuses on one type of data source, and effective modeling still relies on a sufficiently large number of labeled or supervised examples. However, in many real-world applications, in particular with the rapid growth of the Web 2.0, ranking over multiple interrelated (heterogeneous) domains becomes a common situation, where in some domains we may have a large amount of training data while in some other domains we can only collect very little. One important question is: "if there is not sufficient supervision in the domain of interest, how could one borrow labeled information from a related but heterogenous domain to build an accurate model?". This paper explores such an approach by bridging two heterogeneous domains via the latent space. We propose a regularized framework to simultaneously minimize two loss functions corresponding to two related but different information sources, by mapping each domain onto a "shared latent space", capturing similar and transferable oncepts. We solve this problem by optimizing the convex upper bound of the non-continuous loss function and derive its generalization bound. Experimental results on three different genres of data sets demonstrate the effectiveness of the proposed approach.

In this paper, we present an enhanced pictorial structure (PS) model for precise eye localization, a fundamental problem involved in many face processing tasks. PS is a computationally efficient framework for part-based object modelling. For face images taken under uncontrolled conditions, however, the traditional PS model is not flexible enough for handling the complicated appearance and structural variations. To extend PS, we 1) propose a discriminative PS model for a more accurate part localization when appearance changes seriously, 2) introduce a series of global constraints to improve the robustness against scale, rotation and translation, and 3) adopt a heuristic prediction method to address the difficulty of eye localization with partial occlusion. Experimental results on the challenging LFW (Labeled Face in the Wild) database show that our model can locate eyes accurately and efficiently under a broad range of uncontrolled variations involving poses, expressions, lightings, camera qualities, occlusions, etc.

Recognition in uncontrolled situations is one of the most important bottlenecks for practical face recognition systems. In particular, few researchers have addressed the challenge to recognize noncooperative or even uncooperative subjects who try to cheat the recognition system by deliberately changing their facial appearance through such tricks as variant expressions or disguise (e.g., by partial occlusions). This paper addresses these problems within the framework of similarity matching. A novel perception-inspired nonmetric partial similarity measure is introduced, which is potentially useful in dealing with the concerned problems because it can help capture the prominent partial similarities that are dominant in human perception. Two methods, based on the general golden section rule and the maximum margin criterion, respectively, are proposed to automatically set the similarity threshold. The effectiveness of the proposed method in handling large expressions, partial occlusions, and other distortions is demonstrated on several well-known face databases.

Multimodal recognition is an emerging technique to overcome the non-robustness of the unimodal recognition in real applications. Canonical correlation analysis (CCA) has been employed as a powerful tool for feature fusion in the realization of such multimodal system. However, CCA is the unsupervised feature extraction and it does not utilize the class information of the samples, resulting in the constraint of the recognition performance. In this paper, the class information is incorporated into the framework of CCA for combined feature extraction, and a novel method of combined feature extraction for multimodal recognition, called discriminative canonical correlation analysis (DCCA), is proposed. The experiments show that DCCA outperforms some related methods of both unimodal recognition and multimodal recognition.

In this paper, we develop a new effective multiple kernel learning algorithm. First, map the input data into m different feature spaces by m empirical kernels, where each generatedfeature space is takenas one viewof the input space. Then through the borrowing the motivating argument from Canonical Correlation Analysis (CCA)that can maximally correlate the m views in the transformed coordinates, we introduce a special term called Inter-Function Similarity Loss R IFSL into the existing regularization framework so as to guarantee the agreement of multi-view outputs. In implementation, we select the Modification of Ho-Kashyap algorithm with Squared approximation of the misclassification errors (MHKS) as the incorporated paradigm, and the experimental results on benchmark data sets demonstrate the feasibility and effectiveness of the proposed algorithm named MultiK-MHKS.

This paper presents a connected coherence tree algorithm (CCTA) for image segmentation with no prior knowledge. It aims to find regions of semantic coherence based on the proposed epsilon-neighbor coherence segmentation criterion. More specifically, with an adaptive spatial scale and an appropriate intensity-difference scale, CCTA often achieves several sets of coherent neighboring pixels which maximize the probability of being a single image content (including kinds of complex backgrounds). In practice, each set of coherent neighboring pixels corresponds to a coherence class (CC). The fact that each CC just contains a single equivalence class (EC) ensures the separability of an arbitrary image theoretically. In addition, the resultant CCs are represented by tree-based data structures, named connected coherence tree (CCT)s. In this sense, CCTA is a graph-based image analysis algorithm, which expresses three advantages: 1) its fundamental idea, epsilon-neighbor coherence segmentation criterion, is easy to interpret and comprehend; 2) it is efficient due to a linear computational complexity in the number of image pixels; 3) both subjective comparisons and objective evaluation have shown that it is effective for the tasks of semantic object segmentation and figure-ground separation in a wide variety of images. Those images either contain tiny, long and thin objects or are severely degraded by noise, uneven lighting, occlusion, poor illumination, and shadow.

The matrix, as an extended pattern representation to the vector, has proven to be effective in feature extraction. However, the subsequent classifier following the matrix-pattern- oriented feature extraction is generally still based on the vector pattern representation (namely, MatFE + VecCD), where it has been demonstrated that the effectiveness in classification just attributes to the matrix representation in feature extraction. This paper looks at the possibility of applying the matrix pattern representation to both feature extraction and classifier design. To this end, we propose a so-called fully matrixized approach, i.e., the matrix-pattern-oriented feature extraction followed by the matrix-pattern-oriented classifier design (MatFE + MatCD). To more comprehensively validate MatFE + MatCD, we further consider all the possible combinations of feature extraction (FE) and classifier design (CD) on the basis of patterns represented by matrix and vector respectively, i.e., MatFE + MatCD, MatFE + VecCD, just the matrix-pattern-oriented classifier design (MatCD), the vector-pattern-oriented feature extraction followed by the matrix-pattern-oriented classifier design (VecFE + MatCD), the vector-pattern-oriented feature extraction followed by the vector-pattern-oriented classifier design (VecFE + VecCD) and just the vector-pattern-oriented classifier design (VecCD). The experiments on the combinations have shown the following: 1) the designed fully matrixized approach (MatFE + MatCD) has an effective and efficient performance on those patterns with the prior structural knowledge such as images; and 2) the matrix gives us an alternative feasible pattern representation in feature extraction and classifier designs, and meanwhile provides a necessary validation for "ugly duckling" and "no free lunch" theorems.

Dimensionality reduction is among the keys in mining high-dimensional data. This paper studies semi-supervised dimensionality reduction. In this setting, besides abundant unlabeled examples, domain knowledge in the form of pairwise constraints are available, which specifies whether a pair of instances belong to the same class (must-link constraints) or different classes (cannot-link constraints). We propose the SSDR algorithm, which can preserve the intrinsic structure of the unlabeled data as well as both the must-link and cannot-link constraints defined on the labeled examples in the projected low-dimensional space. The SSDR algorithm is efficient and has a closed form solution. Experiments on a broad range of data sets show that SSDR is superior to many established dimensionality reduction methods.

Pseudoinverse Linear Discriminant Analysis (PLDA) is a classical and pioneer method that deals with the Small Sample Size (SSS) problem in LDA when applied to such applications as face recognition. However, it is expensive in computation and storage due to direct manipulation on extremely large d × d matrices, where d is the dimension of the sample image. As a result, although frequently cited in literature, PLDA is hardly compared in terms of classification performance with the newly proposed methods. In this paper, we propose a new feature extraction method named RSw + LDA, which is (1) much more efficient than PLDA in both computation and storage; and (2) theoretically equivalent to PLDA, meaning that it produces the same projection matrix as PLDA. Further, to make PLDA deal better with data of nonlinear distribution, we propose a Kernel PLDA (KPLDA) method with the well-known kernel trick. Finally, our experimental results on AR face dataset, a challenging dataset with variations in expression, lighting and occlusion, show that PLDA (or RSw + LDA) can achieve significantly higher classification accuracy than the recently proposed Linear Discriminant Analysis via QR decomposition and Discriminant Common Vectors, and KPLDA can yield better classification performance compared to PLDA and Kernel PCA.

The goal of this comment is to first point out two loopholes in the paper by Li (2006): (1) so-designed efficient maximal margin criterion (MMC) algorithm for small sample size (SSS) problem is problematic and (2) the discussion on the equivalence with the null-space-based methods in SSS problem does not hold. Then, we will present a really efficient MMC algorithm for SSS problem.

The performance of many computer vision and machine learning algorithms critically depends on the quality of the similarity measure defined over the feature space. Previous works usually utilize metric distances which are ofen epistemologically different from the perceptual distance of human beings. In this paper a novel non-metric partial similarity measure is introduced, which is born to automatically capture the prominent partial similarity between two images while ignoring the confusing unimportant dissimilarity. This measure is potentially useful in face recognition since it can help identify the inherent intra-personal similarity and thus reducing the influence caused by large variations such as expression and occlusions. Moreover; to make this method practical, this paper proposes an automatic and class-dependent similarity threshold setting mechanism based on the maximal margin criterion, and uses a Self- Organization Map-based embedding technique to alleviate the computational problem. Experimental results show the feasibility and effectiveness of the proposed method.

Choosing an appropriate kernel is one of the key problems in kernel-based methods. Most existing kernel selection methods require that the class labels of the training examples are known. In this paper, we propose an adaptive kernel selection method for kernel principal component analysis, which can effectively learn the kernels when the class labels of the training examples are not available. By iteratively optimizing a novel criterion, the proposed method can achieve nonlinear feature extraction and unsupervised kernel learning simultaneously. Moreover, a non-iterative approximate algorithm is developed. The effectiveness of the proposed algorithms are validated on UCI datasets and the COIL-20 object recognition database.

Most classical template-based frontal face recognition techniques assume that multiple images per person are available for training, while in many real-world applications only one training image per person is available and the test images may be partially occluded or may vary in expressions. This paper addresses those problems by extending a previous local probabilistic approach presented by Martinez, using the self-organizing map (SOM) instead of a mixture of Gaussians to learn the subspace that represented each individual. Based on the localization of the training images, two strategies of learning the SOM topological space are proposed, namely to train a single SOM map for all the samples and to train a separate SOM map for each class, respectively. A soft kappa nearest neighbor (soft kappa-NN) ensemble method, which can effectively exploit the outputs of the SOM topological space, is also proposed to identify the unlabeled subjects. Experiments show that the proposed method exhibits high robust performance against the partial occlusions and variant expressions.

The existing morphological auto-associative memory models based on the morphological operations, typically including morphological auto-associative memories (auto-MAM) proposed by Ritter et al. and our fuzzy morphological auto-associative memories (auto-FMAM), have many attractive advantages such as unlimited storage capacity, one-shot recall speed and good noise-tolerance to single erosive or dilative noise. However, they suffer from the extreme vulnerability to noise of mixing erosion and dilation, resulting in great degradation on recall performance. To overcome this shortcoming, we focus on FMAM and propose an enhanced FMAM (EFMAM) based on the empirical kernel map. Although it is simple, EFMAM can significantly improve the auto-FMAM with respect to the recognition accuracy under hybrid-noise and computational effort. Experiments conducted on the thumbnail-sized faces (28 x 23 and 14 x 11) scaled from the ORL database show the average accuracies of 92%, 90%, and 88% with 40 classes under 10%, 20%, and 30% randomly generated hybrid-noises, respectively, which are far higher than the auto-FMAM (67%, 46%, 31%) under the same noise levels.

Fuzzy c-means clustering (FCM) with spatial constraints (FCM_S) is an effective algorithm suitable for image segmentation. Its effectiveness contributes not only to the introduction of fuzziness for belongingness of each pixel but also to exploitation of spatial contextual information. Although the contextual information can raise its insensitivity to noise to some extent, FCM_S still lacks enough robustness to noise and outliers and is not suitable for revealing non-Euclidean structure of the input data due to the use of Euclidean distance (L2 norm). In this paper, to overcome the above problems, we first propose two variants, FCM_S1 and FCM_S2, of FCM_S to aim at simplifying its computation and then extend them, including FCM_S, to corresponding robust kernelized versions KFCM_S, KFCM_S1 and KFCM_S2 by the kernel methods. Our main motives of using the kernel methods consist in: inducing a class of robust non-Euclidean distance measures for the original data space to derive new objective functions and thus clustering the non-Euclidean structures in data; enhancing robustness of the original clustering algorithms to noise and outliers, and still retaining computational simplicity. The experiments on the artificial and real-world datasets show that our proposed algorithms, especially with spatial constraints, are more effective.