Haiyang Wang, Zhiliang Tian, Yuchen Pan, Xin Song, Xin Niu, Minlie Huang, Bin Zhou. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Test-time domain adaptation adjusts a source domain model to accommodate previously unseen domain shifts in a target domain during inference. In real-world scenarios, domain shifts continually evolve, and test data are often non-independent and identically distributed (non-i.i.d.). Existing methods update batch normalization (BN) statistics (mean and variance) based on test batch statistics to mitigate domain shifts and use a memory bank to provide approximate i.i.d. sampling by selectively storing samples. However, excessive updates to BN statistics lead to overfitting to specific domain shifts. To address this issue, we propose a resilient practical test-time adaptation method (ResiTTA), employing soft constraints on the BN statistics and a low-entropy sampling strategy, which reduces overfitting on domain shifts and enables rapid adaptation. Specifically, we develop a resilient batch normalization (BN) with estimated statistics and soft constraints between the source and the estimated statistics. The soft constraints regularize the estimated statistics to mitigate overfitting caused by the excessive updates. To avoid overfitting, we design a low-entropy memory bank that accounts for sample uncertainty and class balance. We adapt the source domain model via a teacher-student self-training adaptation on the samples from the memory, incorporating the soft constraints’ updates to BN. Our ResiTTA obtains state-of-the-art results on various benchmarks. We release our code1.

Remote sensing image generation is of great value for virtual environment creation and adversarial learning for fake news detection. It could also address the learning sample shortage in the region of interest. However, most current image generation methods are limited to producing images of fixed sizes, few studies on extensible natural image generation largely focus on the stitching of random contents, lacking effective exploration of contextual information, which weakens the coherence of the extended images. To address this problem, we propose an extensible generation method for remote sensing images with the model DiffRS. This approach allows for sequential extension of arbitrary sizes by exploring the generated neighboring regions. The method is particularly suitable for scenes like remote sensing images where a generation block could cover multiple independent targets, rather than natural image tasks which may stitch across regions to form a completely target. Compared to the state-of-the-art extensible generation methods, DiffRS could improve the large scale image generation with better structure consistency, richer details and higher realism. Experiments showed that DiffRS could improve the FID score by 4.6% and 3.2% respectively in comparison with the MultiDiffusion and Mixture of Diffuser models.

Multi-view clustering (MVC) methods aim to exploit consistent and complementary information among each view and achieve encouraging performance improvement than single-view counterparts. In practical applications, it is common to obtain instances with partially available information, raising researches of incomplete multi-view clustering (IMC) issues. Recently, several fast IMC methods have been proposed to process the large-scale partial data. Though with considerable acceleration, these methods seek view-shared anchors and ignore specific information among single views. To tackle the above issue, we propose a fast IMC with view-independent anchors (FIMVC-VIA) method in this article. Specifically, we learn individual anchors based on the diversity of distribution among each incomplete view and construct a unified anchor graph following the principle of consistent clustering structure. By constructing an anchor graph instead of pairwise full graph, the time and space complexities of our proposed FIMVC-VIA are proven to be linearly related to the number of samples, which can efficiently solve the large-scale task. The experiment performed on benchmarks with different missing rate illustrates the improvement in complexity and effectiveness of our method compared with other IMC methods. Our code is publicly available at https://github.com/Tracesource/ FIMVC-VIA.

Text classification tasks often encounter fewshot scenarios with limited labeled data, and addressing data scarcity is crucial.Data augmentation with mixup merges sample pairs to generate new pseudos, which can relieve the data deficiency issue in text classification.However, the quality of pseudo-samples generated by mixup exhibits significant variations.Most of the mixup methods fail to consider the varying degree of learning difficulty in different stages of training.And mixup generates new samples with one-hot labels, which encourages the model to produce a high prediction score for the correct class that is much larger than other classes, resulting in the model's over-confidence.In this paper, we propose a self-evolution learning (SE) based mixup approach for data augmentation in text classification, which can generate more adaptive and model-friendly pseudo samples for the model training.SE caters to the growth of the model learning ability and adapts to the ability when generating training samples.To alleviate the model over-confidence, we introduce an instance-specific label smoothing regularization approach, which linearly interpolates the model's output and one-hot labels of the original samples to generate new soft labels for label mixing up.Through experimental analysis, experiments show that our SE brings consistent and significant improvements upon different mixup methods.In-depth analyses demonstrate that SE enhances the model's generalization ability.

Neural architecture search (NAS) has made tremendous progress in the automatic design of effective neural network structures but suffers from a heavy computational burden. One-shot NAS significantly alleviates the burden through weight sharing and improves computational efficiency. Zero-shot NAS further reduces the cost by predicting the performance of the network from its initial state, which conducts no training. Both methods aim to distinguish between "good" and "bad" architectures, i.e., ranking consistency of predicted and true performance. In this paper, we propose Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies as the cheap teacher and adopts the margin ranking loss to distill the ranking knowledge. Specifically, we propose a margin subnet sampler to distill the ranking knowledge from zero-shot NAS to one-shot NAS by introducing Group distance as margin. Our evaluation of the NAS-Bench-201 and ResNet-based search space demonstrates that RD-NAS achieve 10.7% and 9.65% improvements in ranking ability, respectively. Our codes are available at https://github.com/pprp/CVPR2022-NAS-competition-Track1-3th-solution

Practical networks for edge devices adopt shallow depth and small convolutional kernels to save memory and computational cost, which leads to a restricted receptive field. Conventional efficient learning methods focus on lightweight convolution designs, ignoring the role of the receptive field in neural network design. In this paper, we propose the Meta-Pooling framework to make the receptive field learnable for a lightweight network, which consists of parameterized pooling-based operations. Specifically, we introduce a parameterized spatial enhancer, which is composed of pooling operations to provide versatile receptive fields for each layer of a lightweight model. Then, we present a Progressive Meta-Pooling Learning (PMPL) strategy for the parameterized spatial enhancer to acquire a suitable receptive field size. The results on the ImageNet dataset demonstrate that MobileNetV2 using Meta-Pooling achieves top1 accuracy of 74.6%, which outperforms MobileNetV2 by 2.3%.

Vision transformers have shown excellent performance in computer vision tasks. As the computation cost of their self-attention mechanism is expensive, recent works tried to replace the self-attention mechanism in vision transformers with convolutional operations, which is more efficient with built-in inductive bias. However, these efforts either ignore multi-level features or lack dynamic prosperity, leading to sub-optimal performance. In this paper, we propose a Dynamic Multi-level Attention mechanism (DMA), which captures different patterns of input images by multiple kernel sizes and enables input-adaptive weights with a gating mechanism. Based on DMA, we present an efficient backbone network named DMFormer. DMFormer adopts the overall architecture of vision transformers, while replacing the self-attention mechanism with our proposed DMA. Extensive experimental results on ImageNet-1K and ADE20K datasets demonstrated that DMFormer achieves state-of-the-art performance, which outperforms similar-sized vision transformers(ViTs) and convolutional neural networks (CNNs).

Mixed-Precision Quantization (MQ) can achieve a competitive accuracy-complexity trade-off for models. Conventional training-based search methods require time-consuming candidate training to search optimized per-layer bit-width configurations in MQ. Recently, some training-free approaches have presented various MQ proxies and significantly improve search efficiency. However, the correlation between these proxies and quantization accuracy is poorly understood. To address the gap, we first build the MQ-Bench-101, which involves different bit configurations and quantization results. Then, we observe that the existing training-free proxies perform weak correlations on the MQ-Bench-101. To efficiently seek superior proxies, we develop an automatic search of proxies framework for MQ via evolving algorithms. In particular, we devise an elaborate search space involving the existing proxies and perform an evolution search to discover the best correlated MQ proxy. We proposed a diversity-prompting selection strategy and compatibility screening protocol to avoid premature convergence and improve search efficiency. In this way, our Evolving proxies for Mixed-precision Quantization (EMQ) framework allows the auto-generation of proxies without heavy tuning and expert knowledge. Extensive experiments on ImageNet with various ResNet and MobileNet families demonstrate that our EMQ obtains superior performance than state-of-the-art mixed-precision methods at a significantly reduced cost. The code will be released.

Neural architecture search (NAS) has brought significant progress in recent image recognition tasks. Most existing NAS methods apply restricted search spaces, which limits the upper-bound performance of searched models. To address this issue, we propose a new search space named MobileNet3-MT. By reducing human-prior knowledge in omni dimensions of networks, MobileNet3-MT accommodates more potential candidates. For searching in this challenging search space, we present an efficient Multi-trial Evolution-based NAS method termed MENAS. Specifically, we accelerate the evolutionary search process by gradually pruning models in the population. Each model is trained with an early stop and replaced by its Lottery Tickets (the explored optimal pruned network). In this way, the full training pipeline of cumbersome networks is prevented and more efficient networks are automatically generated. Extensive experimental results on ImageNet-1K, CIFAR-10, and CIFAR-100 demonstrate that MENAS achieves state-of-the-art performance.

Multi-view clustering, which appropriately integrates information from multiple sources to reveal data’s inherent structure, is gaining traction in clustering. Though existing procedures have yielded satisfactory results, we observe that they have neglected the inherent local structure in the base kernels. This may cause adverse effects on clustering. To solve the problem, we introduce LF-MKC-LKA, a simple yet effective late fusion multiple kernel clustering with local kernel alignment maximisation approach. In particular, we first determine the nearest$k$neighbours in the average kernel space for each sample and record the information in the nearest neighbor indicator matrix. Then, the nearest neighbor indicator matrix can be used to generate local structure matrix of each sample. The local kernels of each view may then be generated using the local structure matrix, retaining just the highly confident local similarities for learning the intrinsic global manifold of data. They can also be utilised to keep the block diagonal structure and improve the robustness of the underlying kernels against noise.We input the local kernels of each view into the kernel$k$-means (KKM) algorithm and get the local base partitions. Finally, we use a three-step iterative optimization approach to maximize the alignment of the consensus partition using base partitions and a regularisation term. As demonstrated, a significant number of trials on 11 multi-kernel benchmark datasets have shown that the proposed LF-MKC-LKA is effective and efficient. A number of experiments are also designed to demonstrate the fast convergence, excellent performance, robustness and low parameter sensitivity of the algorithm. Our code can be find athttps://github.com/TiejianZhang/TMM21-LF-MKC-LKA.

Since the rapid dissemination of fake news brings a lot of negative effects on real society, automatic fake news detection has attracted increasing attention in recent years. In most circumstances, the fake news detection task is a multimodal problem that consists of textual and visual contents. Many existing methods simply integrate the textual and visual features as a shared representation but overlook their correlations, which may lead to sub-optimal results. To address this problem, we propose CMC, a two-stage fake news detection method with a novel knowledge distillation that captures Cross-Modal feature Correlations while training. In the first stage of CMC, the textual and visual networks are trained mutually in an ensemble learning paradigm. The proposed cross-modal knowledge distillation function is presented as a soft target to guide the training of a single-modal network with the correlations from the other peer. In the second stage of CMC, the two well-trained networks are fixed, and their extracted features are fed to a fusion mechanism. The fusion model is then trained to further improve the performance of multi-modal fake news detection. Extensive experiments on Weibo, PolitiFact, and GossipCop databases show that CMC outperforms the existing state-of-the-art methods by a large margin.

Deep learning has seen dramatic improvements in remote-sensing image scene classification. However, hard categories and hard examples widely exist in the data sets, due to the intraclass diversity and interclass similarity. In this letter, we propose a novel framework to address these issues. Specifically, our method first trains a general model to obtain the confusion matrix and select the hard categories. Then a sampling strategy is proposed to restructure the training set and an expert model is trained to focus on the hard categories. Finally, the knowledge of the expert model is distilled into the student model through a novel loss function, which encourages the student model to predict the hard label provided by manual annotation. Thus, the model can match the soft label provided by the expert model and pay more attention to the hard examples simultaneously. With this method, the student model cannot only deal with hard categories but also hard examples existing in other easy categories. To empirically demonstrate the effectiveness of the proposed method, we comprehensively evaluate the method on three publicly available benchmark data sets, the obtained results show that the proposed method outperforms the existing baseline methods and achieves superior results on all three data sets.

Effective spatiotemporal feature representation is crucial to the video-based action recognition task. Focusing on discriminate spatiotemporal feature learning, we propose Attentional Fused Temporal Transformation Network (AttnTTN) for action recognition on top of popular Temporal Segment Network (TSN) framework. In the network, Attentional Fusion Module (AttnFM) is designed to fuse the appearance and motion features at multiple ConvNet levels for each video snippet, forming a short-term video descriptor. With fused features as inputs, Temporal Transformation Networks (TTN) are employed to model middle-term temporal transformation between the neighboring temporal snippets following a sequential order. AttnTTN achieves the state-of-the-art results on two most popular action recognition datasets: UCF101 and HMDB51.

Deep learning recently has been widely used in many interactive application fields including but not limited to object recognition, speech recognition, natural language processing and so on. At the same time more and more attractive interactive applications (face recognition and augmented reality) are available on wearable and mobile devices. However, traditional deep learning methods such as CNN cost a lot of memory resources. This challenge makes it difficult to apply the powerful deep learning method on mobile memory limited platforms. In this paper we present a novel memory management strategy called mmCNN to solve this problem. This method helps us deploy a trained large size CNN on an any memory size platform including GPU, FPGA and memory-limited mobile devices. In our experiments, we run a feed-forward CNN process in an extremely small memory size (as low as 5MB) on a GPU platform. The result shows that our method saves more than 98% memory compared to a traditional CNN algorithm and further saves more than 90% compared to the sate-of-the-art related work "vDNN". Our work improve the computing scalability of interaction applications and break the memory bottleneck of using deep learning method on a memory-limited devices.

For deep CNN-based image classification models, we observe that confusions between classes with high visual similarity are much stronger than those where classes are visually dissimilar. With these unbalanced confusions, classes can be organized in communities, which is similar to cliques of people in the social network. Based on this, we propose a graph-based tool named "confusion graph" to quantify these confusions and further reveal the community structure inside the database. With this community structure, we can diagnose the model's weaknesses and improve the classification accuracy using specialized expert sub-nets, which is comparable to other state-of-the-art techniques. Utilizing this community information, we can also employ pre-trained models to automatically identify mislabeled images in the large scale database. With our method, researchers just need to manually check approximate 3% of the ILSVRC2012 classification database to locate almost all mislabeled samples.

This letter proposes a method using convolutional neural networks (CNNs) for airport detection on optical satellite images. To efficiently build a deep CNN with limited satellite image samples, a transfer learning approach had been employed by sharing the common image features of the natural images. To decrease the computing cost, an efficient region proposal method had been proposed based on the prior knowledge of the line segments distribution in an airport. The transfer learning ability on deep CNN for airport detection on satellite images had been first evaluated in this letter. The proposed method was tested on an image data set, including 170 different airports and 30 nonairports. The detection rate could reach 88.8% in experiments with seconds' computation time, which showed a great improvement over other the state-of-the-art methods.

This paper proposes a classification approach for hyperspectral image (HSI) using the local receptive fields based kernel extreme learning machine. Extreme learning machine (ELM) has drawn increasing attention in the pattern recognition filed due to its simpleness, speediness and good generalization ability. A kernel method is often used to promote ELM's performance, which is known as kernel ELM. The local receptive field concept originates from research in neuroscience. Considering the local correlations of spectral features, it is promising to improve the performance of HSI classification by combining local receptive fields with kernel ELM. Experimental results on the Pavia University dataset confirm the effectiveness of the proposed HSI classification method.

Real-life behaviors shown by the mobile users typically exhibit plenty noises, making it hard to construct an effective recommendation engine. In this paper, we present a fused model based on the LR algorithm and the GBDT algorithm to recommend vertical industry commodities in a mobile setting. A set of specifically designed methods are proposed to deal with the data preprocessing and feature extraction problem for the mobile recommendation scenario. The proposed method is evaluated on a large scale real-world dataset provided by the Alibaba mobile shopping department. Result on the F1 score has seen an improvement of 2%-36% compared with the baseline.

This paper presents a method for airport detection from optical satellite images using deep convolutional neural networks (CNN). To achieve fast detection with high accuracy, region proposal by searching adjacent parallel line segments has been applied to select candidate fields with potential runways. These proposals were further classified by a CNN model transfer learned from AlexNet to identify the final airport regions from other confusing classes. The proposed method has been tested on a remote sensing dataset consisting of 120 airports. Experiments showed that the proposed method could recognize airports from a large complex area in seconds with an accuracy of 84.1%.

This letter proposes a novel classification approach for a hyperspectral image (HSI) using a hierarchical local-receptive-field (LRF)-based extreme learning machine (ELM). As a fast and accurate pattern classification algorithm, ELM has been applied in numerous fields, including the HSI classification. The LRF concept originates from research in neuroscience. Considering the local correlations of spectral features, it is promising to improve the performance of HSI classification by introducing the LRFs. Recent research on deep learning has shown that hierarchical architectures with more layers can potentially extract abstract representation and invariant features for better classification performance. Therefore, we further extend the LRF-based ELM method to a hierarchical model for HSI classification. Experimental results on two widely used real hyperspectral data sets confirm the effectiveness of the proposed HSI classification approach.

Urban land use and land cover (LULC) classification is one of the core applications in Geographic Information Sys-tem(GIS). In this paper, a novel classification approach based on Deep Belief Network(DBN) for detailed urban mapping is proposed. Deep Belief Network (DBN) is a widely investigated and deployed deep learning model. By applying the DBN model, effective spatio-temporal mapping features can be automatically extracted to improve the classification performance. Six-date RADARSAT-2 Polarimetric SAR (PolSAR) data over the Great Toronto Area were used for evaluation. Experimental results showed that the proposed method can outperform SVM and contextual approaches using adaptive MRF.