Guodong Du, Zitao Fang, Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifei Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Honghai Liu, Min Zhang. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Spiking neural networks (SNNs) have gained prominence for their potential in neuromorphic computing and energy-efficient artificial intelligence, yet optimizing them remains a formidable challenge for gradient-based methods due to their discrete, spike-based computation. This paper attempts to tackle the challenges by introducing Cosine Annealing Differential Evolution (CADE), designed to modulate the mutation factor (F) and crossover rate (CR) of differential evolution (DE) for the SNN model, i.e., Spiking Element Wise (SEW) ResNet. Extensive empirical evaluations were conducted to analyze CADE. CADE showed a balance in exploring and exploiting the search space, resulting in accelerated convergence and improved accuracy compared to existing gradient-based and DE-based methods. Moreover, an initialization method based on a transfer learning setting was developed, pretraining on a source dataset (i.e., CIFAR-10) and fine-tuning the target dataset (i.e., CIFAR-100), to improve population diversity. It was found to further enhance CADE for SNN. Remarkably, CADE elevates the performance of the highest accuracy SEW model by an additional 0.52 percentage points, underscoring its effectiveness in fine-tuning and enhancing SNNs. These findings emphasize the pivotal role of a scheduler for F and CR adjustment, especially for DE-based SNN. Source Code on Github: https://github.com/Tank-Jiang/CADE4SNN.

While fine-tuning pretrained models has become common practice, these models often underperform outside their specific domains. Recently developed model merging techniques enable the direct integration of multiple models, each fine-tuned for distinct tasks, into a single model. This strategy promotes multitasking capabilities without requiring retraining on the original datasets. However, existing methods fall short in addressing potential conflicts and complex correlations between tasks, especially in parameter-level adjustments, posing a challenge in effectively balancing parameter competition across various tasks. This paper introduces an innovative technique named **PCB-Merging** (Parameter Competition Balancing), a *lightweight* and *training-free* technique that adjusts the coefficients of each parameter for effective model merging. PCB-Merging employs intra-balancing to gauge parameter significance within individual tasks and inter-balancing to assess parameter similarities across different tasks. Parameters with low importance scores are dropped, and the remaining ones are rescaled to form the final merged model. We assessed our approach in diverse merging scenarios, including cross-task, cross-domain, and cross-training configurations, as well as out-of-domain generalization. The experimental results reveal that our approach achieves substantial performance enhancements across multiple modalities, domains, model sizes, number of tasks, fine-tuning forms, and large language models, outperforming existing model merging methods.

Darwinian evolution of the biological brain is documented through multiple lines of evidence, although the modes of evolutionary changes remain unclear. Drawing inspiration from the evolved neural systems (e.g., visual cortex), deep learning models have demonstrated superior performance in visual tasks, among others. While the success of training deep neural networks has been relying on back-propagation (BP) and its variants to learn representations from data, BP does not incorporate the evolutionary processes that govern biological neural systems. This work proposes a neural network optimization framework based on evolutionary theory. Specifically, BP-trained deep neural networks for visual recognition tasks obtained from the ending epochs are considered the primordial ancestors (initial population). Subsequently, the population evolved with differential evolution. Extensive experiments are carried out to examine the relationships between Darwinian evolution and neural network optimization, including the correspondence between datasets, environment, models, and living species. The empirical results show that the proposed framework has positive impacts on the network, with reduced over-fitting and an order of magnitude lower time complexity compared to BP. Moreover, the experiments show that the proposed framework performs well on deep neural networks and big datasets.

Existing image restoration models are typically designed for specific tasks and struggle to generalize to out-of-sample degradations not encountered during training. While zero-shot methods can address this limitation by fine-tuning model parameters on testing samples, their effectiveness relies on predefined natural priors and physical models of specific degradations. Nevertheless, determining out-of-sample degradations faced in real-world scenarios is always impractical. As a result, it is more desirable to train restoration models with inherent generalization ability. To this end, this work introduces the Out-of-Sample Restoration (OSR) task, which aims to develop restoration models capable of handling out-of-sample degradations. An intuitive solution involves pre-translating out-of-sample degradations to known degradations of restoration models. However, directly translating them in the image space could lead to complex image translation issues. To address this issue, we propose a model reprogramming framework, which translates out-of-sample degradations by quantum mechanic and wave functions. Specifically, input images are decoupled as wave functions of amplitude and phase terms. The translation of out-of-sample degradation is performed by adapting the phase term. Meanwhile, the image content is maintained and enhanced in the amplitude term. By taking these two terms as inputs, restoration models are able to handle out-of-sample degradations without fine-tuning. Through extensive experiments across multiple evaluation cases, we demonstrate the effectiveness and flexibility of our proposed framework. Our codes are available at https://github.com/ddghjikle/Out-of-sample-restoration.

Blind image deblurring remains challenging due to the ill-posed nature of the traditional blurring function. Although previous supervised methods have achieved great breakthrough with synthetic blurry-sharp image pairs, their generalization ability to real-world blurs is limited by the discrepancy between synthetic and real blurs. To overcome this limitation, unsupervised deblurring methods have been proposed by using natural priors or generative adversarial networks. However, natural priors are vulnerable to random blur artifacts, while generators of generative adversarial networks always produce inaccurate details and unrealistic colors. Consequently, previous methods easily suffer from slow convergence and poor performance. In this work, we propose to formulate the traditional blurring function as the composition of multiple variates, thus allowing us explicitly define characteristics of residual images between blurry and sharp images. We also propose a multi-step self-supervised deblurring framework to address the slow convergence issue. Our framework continuously decomposes and composes input images, thus utilizing the uncertainty of blur artifacts to obtain diverse pseudo blurry-sharp image pairs for self-supervised learning. This framework is more efficient than previous methods, as it does not rely on natural priors or GANs. Extensive comparisons demonstrate that the proposed framework outperforms state-of-the-art unsupervised methods on both dynamic scene, human-aware centric motion, real-world and out-of-focus deblurring datasets. The codes are available at https://github.com/ddghjikle/MM-2023-USDF.

Oracle Bone Inscriptions (OBI) are ancient hieroglyphs originated in China and are considered one of the most famous writing systems in the world. Up to now, thousands of OBIs have been discovered, which require deciphering by experts to understand their contents. Experts typically need to restore, classify, and compare each character with previous inscriptions. Although existing research can assist with one of these operations, their performance falls short of practical requirements. In this work, we propose the OraclePoints framework, which represents OBI images as hybrid neural representations comprising features of images and point sets. The image representation provides inscription appearance and character structure, while the point representation makes it easy and effective to distinguish characters and noises. In addition, we demonstrate that OraclePoints can be easily integrated with existing models in a plug-and-play manner. Comprehensive experiments demonstrate that the proposed hybrid neural representation framework supports a range of OBI tasks, including character image retrieval, recognition, and denoising. It is also demonstrated that OraclePoints is helpful for deciphering OBIs by linking ancient characters to modern Chinese characters. Our codes are available at https://ddghjikle.github.io/.

Unsupervised domain adaptation enables neural networks to transfer from a labeled source domain to an unlabeled target domain by learning domain-invariant representations. Recent approaches achieve this by directly matching the marginal distributions of these two domains. Most of them, however, ignore exploration of the dynamic trade-off between domain alignment and semantic discrimination learning, thus rendering them susceptible to the problems of negative transfer and outlier samples. To address these issues, we introduce the dynamic parameterized learning framework. First, by exploring domain-level semantic knowledge, the dynamic alignment parameter is proposed, to adaptively adjust the optimization steps of domain alignment and semantic discrimination learning. Besides, for obtaining semantic-discriminative and domain-invariant representations, we propose to align training trajectories on both source and target domains. Comprehensive experiments are conducted to validate the effectiveness of the proposed methods, and extensive comparisons are conducted on seven datasets of three visual tasks to demonstrate their practicability.

Video deblurring, which aims at restoring the sharp video from blurry video, is drawing increasing attention in the field of computer vision. In this paper, a method called Multi-Attention Convolutional Neural Network (MACNN) consisting of the temporal-spatial attention module, the frame channel attention module, and the feature extraction-reconstruction module is proposed. First, we use the temporal-spatial attention module and the frame channel attention module to capture features with temporal and spatial information existing across neighboring frames. Then, these captured features are fused and reconstructed to restore the sharp frame. Last but not least, we train MACNN together with a content loss and a perceptual loss in an end-to-end manner to recover realistic video details. Both quantitative and qualitative evaluation results on standard benchmarks demonstrate the proposed MACNN is superior to the state-of-the-art methods in terms of accuracy, efficiency, and visual effect.

Single image dehazing, which aims at restoring a haze-free image from its correspondingly unconstrained hazy scene, is a fundamental yet challenging task and has gained immense popularity recently. However, the images recovered by some existing haze-removal methods often contain haze, artifacts, and color distortions, which severely degrade the visual quality and have negative impacts on subsequent computer vision tasks. To this end, we propose a network combining multi-scale hierarchical feature fusion and mixed convolution attention to progressively and adaptively enhance the dehazing performance. The haze levels and image structure information are accurately estimated by fusing multi-scale hierarchical features, thus the model restores images with less remaining haze. The proposed mixed convolution attention mechanism is capable of reducing feature redundancy, learning compact and effective internal representations and highlighting task-relevant features, thus, it can further help the model estimate images with sharper textural details and more vivid colors. Furthermore, a deep semantic loss is also proposed to highlight essential semantic information in deep features. The experimental results show that the proposed method outperforms state-of-the-art haze removal algorithms.

Video deblurring is still a challenging low-level vision task since spatio-temporal characteristics across both the spatial and temporal domains are difficult to model. In this article, to model the temporal information, we develop a non-local block which estimates inter-frame similarity and inter-frame difference. Specially, for modeling the spatial characteristics and restoring sharp frame details, we propose a recursive block that iteratively refines feature maps generated at the last iteration. In addition, a novel temporal loss function is introduced to ensure the temporal consistency of generated frames. Experimental results on public datasets demonstrate that our method achieves state-of-the-art performance both quantitatively and qualitatively.

An image can be decomposed into two parts: the basic content and details, which usually correspond to the low-frequency and high-frequency information of the image. For a hazy image, these two parts are often affected by haze in different levels, e.g., high-frequency parts are often affected more serious than low-frequency parts. In this paper, we approach the single image dehazing problem as two restoration problems of recovering basic content and image details, and propose a Dual-Path Recurrent Network (DPRN) to simultaneously tackle these two problems. Specifically, the core structure of DPRN is a dual-path block, which uses two parallel branches to learn the characteristics of the basic content and details of hazy images. Each branch consists of several Convolutional LSTM blocks and convolution layers. Moreover, a parallel interaction function is incorporated into the dual-path block, thus enables each branch to dynamically fuse the intermediate features of both the basic content and image details. In this way, both branches can benefit from each other, and recover the basic content and image details alternately, therefore alleviating the color distortion problem in the dehazing process. Experimental results show that the proposed DPRN outperforms state-of-the-art image dehazing methods in terms of both quantitative accuracy and qualitative visual effect.

Recently, deep neural networks have been widely used in the task of single image super-resolution. However, existing deep neural networks always take huge parameters to map low-resolution images to high-resolution ones. In addition, most of them only consider high-level features to reconstruct high-resolution images. These two methodologies not only cause the difficulty of practical applications but also the inefficiency of restoring image details. Therefore, in this work, the authors propose a novel sparse block to learn high-level features. Based on this block, a fusion method is proposed to fuse features from multiple levels. By incorporating these two approaches, a lightweight neural network, i.e. Sparse Block Fusion Network (SBFN), is proposed for end-to-end training. Through extensive experiments, it is demonstrated the proposed methods can achieve comparable performance with few parameters. By making comprehensive comparisons, effectiveness of SBFN is also verified in multiple benchmark datasets.

Sparse representation and dictionary learning have been successfully applied to encode dense data and facilitate image classification. Though existing dictionary learning methods achieve better performance than their counterparts, the class discriminative ability of learned dictionary is still limited. This paper proposes a novel supervised dictionary learning method based on the prior of the block diagonal phenomenon, i.e., each sample should be well reconstructed by the samples in the same class while poorly reconstructed by the samples in other class. Specifically, a block diagonal regularizer is imposed on the affinity matrix to enforce the sparse representation matrix to have an approximately block diagonal structure, which makes the learned dictionary more discriminative and suitable for classification tasks. Furthermore, we present an effective optimization strategy by combining the alternating minimization with the alternating direction method of multipliers (ADMM) for the proposed framework. Experimental results on six real-world datasets show that the proposed method is more effective than state-of-the-art dictionary learning methods.

Recently, deep convolution neural networks have achieved remarkable performance in the task of single image super-resolution (SISR). However, effectiveness of existing networks highly relies on their receptive field, which always increases with the depth of the network. In this work, we propose a novel module, named as residual group, to effectively learn feature maps by using dynamic receptive field. This residual group firstly uses a selective kernel convolution layer to dynamically learn multi-scale information from its input features. Then, several residual blocks are employed to further refine the learned feature. In addition, we also propose a selective feature fusion module to fuse appearance information in multi-level features. Within this module, the low-level features and high-level features are selectively fused to complement the high-level ones. Finally, by combining these two methods, we introduce a multi-level feature fusion network (MLFFN) for single image super-resolution (SISR). Through comprehensive experiments, we demonstrate that the proposed MLFFN achieves state-of-the-art performance both quantitatively and qualitatively.

Single image haze removal is a challenging ill-posed problem in computer vision. Instead of leveraging the traditional model or handcrafted image priors, an end-to-end multi-scale convolutional neural network is proposed for single image haze removal task by directly mapping the hazy image to its corresponding haze-free image. To better retain the coarse and fine information, a multi-scale block is elaborated and embedded into the proposed architecture. This block can extract the feature at varying scales with a model size that is as small as possible. The global skip connection is adopted to promote the model performance. Extensive experiment results demonstrate that the proposed network outperforms the state-of-the-art single image haze removal algorithms on both synthetical and real-world images. In addition, the size of the model in this paper dominates among the high performance methods based on convolutional neural networks.