Adapting Multi-modal Large Language Models (MLLMs) to target tasks often suffers from catastrophic forgetting, where acquiring new task-specific knowledge compromises performance on pre-trained tasks. In this paper, we introduce AdaDARE-γ, an efficient approach that alleviates catastrophic forgetting by controllably injecting new task-specific knowledge through adaptive parameter selection from fine-tuned models without requiring retraining procedures. This approach consists two key innovations: (1) an adaptive parameter selection mechanism that identifies and retains the most task-relevant parameters from fine-tuned models, and (2) a controlled task-specific information injection strategy that precisely balances the preservation of pre-trained knowledge with the acquisition of new capabilities. Theoretical analysis proves the optimality of our parameter selection strategy and establishes bounds for the task-specific information injection factor. Extensive experiments on InstructBLIP and LLaVA-1.5 across image captioning and visual question answering tasks demonstrate that AdaDARE-γ establishes new state-of-the-art results in balancing model performance. Specifically, it maintains 98.2% of pre-training effectiveness on original tasks while achieving 98.7% of standard fine-tuning performance on target tasks.

Patch-wise Transformer based time series forecasting achieves superior accuracy. However, this superiority relies heavily on intricate model design with massive parameters, rendering both training and inference expensive, thus preventing their deployments on edge devices with limited resources and low latency requirements. In addition, existing methods often work in an autoregressive manner, which take into account only historical values, but ignore valuable, easy-to-obtain context information, such as weather forecasts, date and time of day. To contend with the two limitations, we propose LiPFormer, a novel Lightweight Patch-wise Transformer with weak data enriching. First, to simplify the Transformer backbone, LiPFormer employs a novel lightweight cross-patch attention and a linear transformationbased attention to eliminate Layer Normalization and Feed Forward Network, two heavy components in existing Transformers. Second, we propose a lightweight, weak data enriching module to provide additional, valuable weak supervision to the training. It enhances forecasting accuracy without significantly increasing model complexity as it does not involve expensive, human-labeling but using easily accessible context information. This facilitates the weak data enriching to plug-and-play on existing models. Extensive experiments on nine benchmark time series datasets demonstrate that LiPFormer outperforms state-of-the-art methods in accuracy, while significantly reducing parameter scale, training duration, and GPU memory usage. Deployment on an edge device reveals that LiPFormer takes only 1/3 inference time compared to classic Transformers. In addition, we demonstrate that the weak data enriching can integrate seamlessly into various Transformer based models to enhance their accuracy, suggesting its generality.

This study explores the efficacy of fine-tuning large language models (LLMs) using non-task-specific datasets, challenging the traditional reliance on task-specific datasets. Based on the SUP-NATINST and Stanford Alpaca dataset, this research explores some schemes to enhance the application capabilities of LLMs in specific domains through strategic data selection. The main contributions include an innovative fine-tuning data selection strategy that emphasizes the importance of dataset selection by comparing non-task-specific datasets with task-specific datasets, demonstrating the potential advantages of unconventional datasets in improving model task performance. The study also reveals the model’s performance sensitivity to data ratios, challenges the concept of optimal data ratios, and explores the impact of data volume changes on model accuracy. Furthermore, it deepens the understanding of how different task types affect model performance fine-tuning. By analyzing the performance across different tasks and dimensions and dissecting the impact of data ratio changes, the research indicates that non-specific task data allocation can achieve better performance than specific task datasets.

Event argument extraction (EAE) is an important and critical task in natural language processing, whose goal is to transform unstructured text into structured information. However, previous methods are costly for annotation, which limits their application. With the emergence of large language models (LLMs), many approaches use prompt learning or instruction learning to solve the EAE task. However, due to the complexity of the EAE, the defined instructions are often abstract and difficult for the model to accurately understand the task objective, while the cost of gradient propagation based methods is too high. Therefore, we propose an Instruction Feature Adaptation event argument extraction (IFA) that uses very few shots to tune the task instruction features, while not requiring a highcost gradient optimisation method to enhance the model’s performance on the event argument extraction task. Experiments demonstrate that the method is able to optimise the instruction features, enabling the model to perform the event argument task more consistently based on the instructions, ultimately improving performance for low-resource. Code is available at https://github.com/yangzonghao1024/IFA.

Recently, adversarial attacks have shown to lead the state-of-the-art deep neural networks (DNNs) to misclassification. However, most adversarial attacks are generated according to whether they are perceptual to human visual system, measured by geometric metrics such as the$\ell _2$-norm, which ignores the common watermarks in cyber-physical systems. In this article, we propose a fast adversarial watermark attack (FAWA) method based on fast differential evolution technique, which optimally superimposes a watermark on an image to fool DNNs. We also attempt to explain the reason why the attack is successful and propose two hypotheses on the vulnerability of DNN classifiers and the influence of the watermark attack on higher-layer features extraction respectively. In addition, we propose two countermeasure methods against FAWA based on random rotation and median filtering respectively. Experimental results show that our method achieves 41.3 percent success rate in fooling VGG-16 and have good transferability. Our approach is also shown to be effective in deceiving deep learning as a service (DLaaS) systems as well as the physical world. The proposed FAWA, hypotheses, and the countermeasure methods, provide a timely help for DNN designers to gain some knowledge of model vulnerability while designing DNN classifiers and related DLaaS applications.

Adversarial examples (AEs) make Internet-of-Things (IoT) systems to face a great challenge. The tiny fluctuations in AEs mislead the learning model, let the model accept, and make wrong decisions. So, accurately identifying AEs is significant for improving the robustness in IoT systems. In this article, a method to identify AEs for time-series data is proposed. We analyze the characteristics of adversarial time-series examples and normal time-series samples and propose a time-series representation method. The tiny fluctuations in AEs are easily discovered by this representation. A metric is also designed to support this representation method to identify AEs with supervised learning and unsupervised learning methods. Our representation method and the metric is applied with supervised learning and unsupervised learning methods to identify time series in real data set and data sets in the official UCR repository. The experimental results show that our method has a great ability to identify AEs, compared to previous approaches.

The stability characteristics of scattering are of great significance for achieving radiometric calibration without calibration fields. This letter used dual-polarization synthetic aperture radar (SAR) data to explore the temporal and spatial stability of microwave scattering. Based on Sentinel-1 SAR data, we mainly studied the stability of the median value of SAR image slices in urban areas, under the HH- and HV-polarization modes. The urban slices with stable scattering were extracted by an improved deep neural network designed herein. Based on 33 images of the Houston region, the results show that this suitable network, which is trained by dual-polarization data, can effectively distinguish the stable slices from the unstable ones. In addition, some meaningful characteristics about SAR scattering stability of dual-polarization data are also obtained, which can provide a good reference for SAR nonfield radiometric calibration.