Online continual learning (OCL) aims at learning a non-stationary data stream in a way of reading each data sample only once, and hence suffers from the trade-off of catastrophic forgetting and insufficient learning. In this work, we firstly analytically establish relationship between loss functions and model parameters from the Bayesian perspective. Based on our analysis, we subsequently propose a parameter merging method with gradient-guided supermasks. Our method leverages 1-order and 2-order gradient information to construct supermasks that determine the merging weights between the old and new models. Our method performs direct arithmetic operations on parameters to update models, beyond traditional gradient descent. We further discover that a widely-used premise that 1-order gradients can be negligible is invalid in OCL, due to slow convergence incurred by insufficient learning. Additionally, we utilize a dual-model dual-view distillation strategy that can align output distributions of the new and merged models for each sample, further enhancing model performance. Extensive experiments are conducted on four benchmarks in OCL settings, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-100. Experimental results demonstrate that our method is effective, and achieves a substantial boost over previous methods.

Despite significant progress in the field, it is still challenging to create personalized visual representations that align closely with the desires and preferences of individ-ual users. This process requires users to articulate their ideas in words that are both comprehensible to the models and accurately capture their vision, posing difficul-ties for many users. In this paper, we tackle this challenge by leveraging historical user interactions with the system to enhance user prompts. We propose a novel approach that involves rewriting user prompts based on a newly collected large-scale text-to-image dataset with over 300k prompts from 3115 users. Our rewriting model enhances the expressiveness and alignment of user prompts with their intended visual outputs. Experimental results demonstrate the superiority of our methods over baseline approaches, as evidenced in our new offline evaluation method and online tests. Our code and dataset are available at https://github.com/zzjchen/Tailored-Visions

Distillation strategies are currently the primary approaches for mitigating forgetting in class incremental learning (CIL). Existing methods generally inherit previous knowledge from a single teacher. However, teachers with different mechanisms are talented at different tasks, and inheriting diverse knowledge from them can enhance compatibility with new knowledge. In this paper, we propose the MTD method to find multiple diverse teachers for CIL. Specifically, we adopt weight permutation, feature perturbation, and diversity regularization techniques to ensure diverse mechanisms in teachers. To reduce time and memory consumption, each teacher is represented as a small branch in the model. We adapt existing CIL distillation strategies with MTD and extensive experiments on CIFAR-100, ImageNet-100, and ImageNet-1000 show significant performance improvement. Our code is available at https://github.com/HaitaoWen/CLearning.

Projected GAN, a pre-trained GAN, has been found to perform well in generating images with only a few training samples. However, it struggles with extended training, which may lead to decreased performance over time. This is because the pre-trained discriminator consistently surpasses the generator, creating an unstable training environment. In this work, we propose a solution to this issue by introducing closed-loop control (CLC) into the dynamics of Projected GAN, stabilizing training, and improving generation performance. Our proposed method consistently reduces the Fréchet Inception Distance (FID) of the previous methods; for example, it reduces the FID of Projected GAN by 4.31 on the Obama dataset. Our finding is fundamental and can be used in other pre-trained GANs.

With the rapid development of wearable cameras, it is now feasible to considerably increase the collection of egocentric video for first-person visual perception. However, the development is hindered by a shortage of multi-modal egocentric activity datasets. Furthermore, the catastrophic forgetting problem of multimodal continual activity learning, as a branch of continual learning, has not been thoroughly explored, which makes accumulating a larger collection of multi-modal activity data more urgent. To address this shortage, we propose a multi-modal egocentric activity dataset for continual activity learning named UESTC-MMEA-CL in this paper. The dataset is collected using our self-developed glasses with a first-person camera and wearable sensors, and it contains synchronized data of video, accelerometers, and gyroscopes for 32 types of daily activities performed by 10 participants who wore our glasses. Statistical analysis of the sensor data is given to show the auxiliary effects of activity recognition. We report the results of egocentric activity recognition of three modalities (RGB, acceleration, and gyroscope) separately and jointly on a base network architecture. We thoroughly evaluated four baseline methods with different multimodal combinations to explore the catastrophic forgetting in continual learning on UESTC-MMEA-CL. We hope that the UESTC-MMEA-CL dataset can act as a facilitator for future studies on continual learning for first-person activity recognition in wearable applications. You can download preliminary data fromhttps://ivipclab.github.io/publication_uestc-mmea-cl/mmea-cl. The data is currently used to solve the problems of multimodal continual learning of activities.

Unsupervised continual learning (UCL) has made remarkable progress over the past two years, significantly expanding the application of continual learning (CL). However, existing UCL approaches have only focused on transferring continual strategies from supervised to unsupervised. They have overlooked the relationship issue between visual features and representational continuity. This work draws attention to the texture bias problem in existing UCL methods. To address this problem, we propose a new UCL framework called InfoUCL, in which we develop InfoDrop contrastive loss to guide continual learners to extract more informative shape features of objects and discard useless texture features simultaneously. The proposed InfoDrop contrastive loss is general and can be combined with various UCL methods. Extensive experiments on various benchmarks have demonstrated that our InfoUCL framework can lead to higher classification accuracy and superior robustness to catastrophic forgetting.

Continual learning requires a model to incrementally learn a sequence of tasks and aims to predict well on all the learned tasks so far, which notoriously suffers from the catastrophic forgetting problem. In this paper, we find a new type of bias appearing in continual learning, coined as task-induced bias. We place continual learning into a causal framework, based on which we find the task-induced bias is reduced naturally by two underlying mechanisms in task and domain incremental learning. However, these mechanisms do not exist in class incremental learning (CIL), in which each task contains a unique subset of classes. To eliminate the task-induced bias in CIL, we devise a causal intervention operation so as to cut off the causal path that causes the task-induced bias, and then implement it as a causal debias module that transforms biased features into unbiased ones. In addition, we propose a training pipeline to incorporate the novel module into existing methods and jointly optimize the entire architecture. Our overall approach does not rely on data replay, and is simple and convenient to plug into existing methods. Extensive empirical study on CIFAR-100 and ImageNet shows that our approach can improve accuracy and reduce forgetting of well-established methods by a large margin.

Class incremental learning (CIL) is one of the most challenging scenarios in continual learning. Existing work mainly focuses on strategies like memory replay, regularization, or dynamic architecture but ignores a crucial aspect: mode connectivity. Recent studies have shown that different minima can be connected by a low-loss valley, and ensembling over the valley shows improved performance and robustness. Motivated by this, we try to investigate the connectivity in CIL and find that the high-loss ridge exists along the linear connection between two adjacent continual minima. To dodge the ridge, we propose parameter-saving OPtimizing Connectivity (OPC) based on Fourier series and gradient projection for finding the low-loss path between minima. The optimized path provides infinite low-loss solutions. We further propose EOPC to ensemble points within a local bent cylinder to improve performance on learned tasks. Our scheme can serve as a plug-in unit, extensive experiments on CIFAR-100, ImageNet-100, and ImageNet-1K show consistent improvements when adapting EOPC to existing representative CIL methods. Our code is available at https://github.com/HaitaoWen/EOPC.

Multi-view clustering is an important research topic due to its capability to utilize complementary information from multiple views. However, there are few methods to consider the negative impact caused by certain views with unclear clustering structures, resulting in poor multi-view clustering performance. To address this drawback, we proposeself-supervised discriminative feature learning fordeepmulti-viewclustering (SDMVC). Concretely, deep autoencoders are applied to learn embedded features for each view independently. To leverage the multi-view complementary information, we concatenate all views’ embedded features to form the global features, which can overcome the negative impact of some views’ unclear clustering structures. In a self-supervised manner, pseudo-labels are obtained to build a unified target distribution to perform multi-view discriminative feature learning. During this process, global discriminative information can be mined to supervise all views to learn more discriminative features, which in turn are used to update the target distribution. Besides, this unified target distribution can make SDMVC learn consistent cluster assignments, which accomplishes the clustering consistency of multiple views while preserving their features’ diversity. Experiments on various types of multi-view datasets show that SDMVC outperforms 14 competitors including classic and state-of-the-art methods. The code is available athttps://github.com/SubmissionsIn/SDMVC.

The goal of few-shot classification is to classify new categories with few labeled examples within each class. Nowadays, the excellent performance in handling few-shot classification problems is shown by metric-based meta-learning methods. However, it is very hard for previous methods to discriminate the fine-grained sub-categories in the embedding space without fine-grained labels. This may lead to unsatisfactory generalization to fine-grained sub-categories, and thus affects model interpretation. To tackle this problem, we introduce the contrastive loss into few-shot classification for learning latent fine-grained structure in the embedding space. Furthermore, to overcome the drawbacks of random image transformation used in current contrastive learning in producing noisy and inaccurate image pairs (i.e., views), we develop a learning-to-learn algorithm to automatically generate different views of the same image. Extensive experiments on standard few-shot learning benchmarks demonstrate the superiority of our method.

Accurate and robust head pose estimation is a challenging computer vision task. In most existing methods, single-modal RGB or depth images are directly used for head pose estimation. The obvious drawbacks of these methods are two fold: (1) Traditional shallow models are not good at learning representative features. (2) They are single-modal approaches, resulting in sensitivity to noise. As such, in this work we propose a novel multi-modal regression model for head pose estimation, named mixture of deep regression networks (MoDRN). It only uses good examples for one modality to learn sub-network parameters. Thus, the sub-networks tend to be better trained and more robust to noise, making significant improved performance in their combination. Experiments on public datasets such as BIWI and BU-3DFE show the effectiveness of our approach.

Due to its importance, figure/ground segmentation in video has gained interest recently. The key factor of the segmentation is the construction of the spatio-temporal coherence. Previous works usually use the motion approximation as a measurement of the coherence, resulting in a low accuracy. In this paper, we present a novel method to measure the coherence, and an algorithm for target segmentation and tracking is proposed. Each image is abstracted by some compact and perceptually homogeneous elements, and by representing the elements as sparse linear combinations of dictionary templates, this algorithm capitalizes on the inherent low-rank structure of representations that are learned jointly. The coefficients of the constrained representation will act as the measurement of the spatio-temporal coherence. At last, a simple energy minimization solution with an online parameter-updating scheme is adopted in segmented stage, leading to a binary object's segmentation. Meanwhile, an adaptive dictionary is proposed to enhance the system's robust against occlusion. Our approach outperforms the state-of-the-art methods in object segmentation accuracy.

Facial age estimation is a challenging problem in computer vision. Existing methods can be classified into two categories: global and person-specific. In practice, the person-specific methods have shown better performance, however it still has some inherit problems such as over learning and mis-assignment of age estimators for unseen facial images. To fix these problems, this paper proposes correlated warped Gaussian processes (CWGP) regression for gender-specific age estimation. It uses two correlated regressors to accurately approximate the gender-specific mapping from facial features to age. Extensive experiments demonstrate the superiority of our method over state-of-the-art methods.

Mixture of regressions is one of the most well-known statistical techniques for the problem of head pose estimation. However, conventional approaches are often sensitive to noise and suffer from underdetermined problem when the training data is insufficient (i.e., the number of training samples for some regressors is less than the dimensionality of the image features). In this paper, we propose a novel approach, named mixture of related regressions (MReR) to address above limitations. By imposing an additional similarity constraint on related regressors, MReR can significantly enhance robustness and avoid uncertainty for head pose estimation. As a nontrivial byproduct, we also develop an EM-type algorithm to efficiently solve the MReR model. Experimental results on both synthetic and real-world datasets demonstrate the benefits of MReR.

Accurate and efficient detection of circular objects in images is a challenging computer vision problem. Existing circular object detection methods can be broadly classified into two categories: voting based and maximum likelihood estimation (MLE) based. The former is robust to noise, however its computational complexity and memory requirement are high. On the other hand, MLE based methods (e.g., robust least squares fitting) are more computationally efficient but sensitive to noise, and can not detect multiple circles. This letter proposes Probabilistic Pairwise Voting (PPV), a fast and robust algorithm for circular object detection based on an extension of Hough Transform. The main contributions are threefold. 1) We formulate the problem of circular object detection as finding the intersection of lines in the three dimensional parameter space (i.e., center and radius of the circle). 2) We propose a probabilistic pairwise voting scheme to robustly discover circular objects under occlusion, image noise and moderate shape deformations. 3) We use a mode-finding algorithm to efficiently find multiple circular objects. We demonstrate the benefits of our approach on two real-world problems: 1) detecting circular objects in natural images, and 2) localizing iris in face images.

Iris localization is an especially important step in the whole iris recognition system, for it determines the accuracy of matching partially. To improve its accuracy and efficiency, we propose a new iris localization algorithm in this paper, which detects the edge points of iris at some appropriate resolutions and fits circles to these edge points. The experiment results show that the proposed algorithm maintains a good compromise between accuracy and efficiency. The most important merit of this algorithm is that it¿s seldom disturbed by lash occlusion problem.