Multimodal visual language models are gaining prominence in open-world applications, driven by advancements in model architectures, training techniques, and high-quality data. However, their performance is often limited by insufficient task-specific data, leading to poor generalization and biased outputs. Existing efforts to increase task diversity in fine-tuning datasets are hindered by the labor-intensive process of manual task labeling, which typically produces only a few hundred task types. To address this, we propose TaskGalaxy, a large-scale multimodal instruction fine-tuning dataset comprising 19,227 hierarchical task types and 413,648 samples. TaskGalaxy utilizes GPT-4o to enrich task diversity by expanding from a small set of manually defined tasks, with CLIP and GPT-4o filtering those that best match open-source images, and generating relevant question-answer pairs. Multiple models are employed to ensure sample quality. This automated process enhances both task diversity and data quality, reducing manual intervention. Incorporating TaskGalaxy into LLaVA-v1.5 and InternVL-Chat-v1.0 models shows substantial performance improvements across 16 benchmarks, demonstrating the critical importance of task diversity. TaskGalaxy is publicly released at https://github.com/Kwai-YuanQi/TaskGalaxy.

In post-translational modification, covalent bonds on lysine and attached chemical groups significantly change proteins' physical and chemical properties. They shape protein structures, enhance function and stability, and are vital for physiological processes, affecting health and disease through mechanisms like gene expression, signal transduction, protein degradation, and cell metabolism. Although lysine (K) modification sites are considered among the most common types of post-translational modifications in proteins, research on K-PTMs has largely overlooked the synergistic effects between different modifications and lacked the techniques to address the problem of sample imbalance. Based on this, the Extreme Point Deviation Compensated Clustering (EPDCC) Undersampling algorithm was proposed in this study and combined with Cross-Scale Convolutional Neural Networks (CSCNNs) to develop a novel computational tool, MlyPredCSED, for simultaneously predicting multiple lysine modification sites. MlyPredCSED employs Multi-Label Position-Specific Triad Amino Acid Propensity and the physicochemical properties of amino acids to enhance the richness of sequence information. To address the challenge of sample imbalance, the innovative EPDCC Undersampling technique was introduced to adjust the majority class samples. The model's training and testing phase relies on the advanced CSCNN framework. MlyPredCSED, through cross-validation and testing, outperformed existing models, especially in complex categories with multiple modification sites. This research not only provides an efficient method for the identification of lysine modification sites but also demonstrates its value in biological research and drug development. To facilitate efficient use of MlyPredCSED by researchers, we have specifically developed an accessible free web tool: http://www.mlypredcsed.com.

Nowadays, machine learning is widely used in various applications. Training a model requires huge amounts of data, but it can pose a threat to user privacy. With the growing concern for privacy, the “Right to be Forgotten” has been proposed, which means that users have the right to request that their personal information be removed from machine learning models. The emergence of machine unlearning is a response to this need. Implementing machine unlearning is not easy because simply deleting samples from a database does not allow the model to “forget” the data. Therefore, this paper summarises the definition of the machine unlearning formulation, process, deletion requests, design requirements and validation, algorithms, applications, and future perspectives, in the hope that it will help future researchers in machine unlearning.

Parameter-Efficient Fine-Tuning methods based on vision-language models (such as CLIP) for few-shot learning have recently received considerable attention. However, previous works only fine-tune either the image or text branch, breaking the alignment of the original two branches, meanwhile fine-tuning both branches of the CLIP would inevitably introduce more trainable parameters and likely cause more severe over-fitting due to the limited training data. In this study, we propose a novel Dual-branch Adapter-Tuning framework (DAT), which collaboratively trains the visual adapter and textual adapter added to the two branches of the original CLIP with multiple consistency constraints. By effectively utilizing the semantically detailed class-specific prompts and outputs of the original CLIP to guide the fine-tuning of both branches, our method gains exceptional adaptation ability to the downstream few-shot learning tasks and alleviates the over-fitting issue, meanwhile maximally preserving the generalization ability of the original CLIP model. Our proposed framework has achieved superior performance on diverse datasets under various few-shot learning settings compared to the existing approaches. The source code is available athttps://github.com/SandyXi/DAT.

Out-of-distribution (OOD) detection is crucial in many real-world applications. However, intelligent models are often trained solely on in-distribution (ID) data, leading to overconfidence when misclassifying OOD data as ID classes. In this study, we propose a new learning framework which leverage simple Jigsaw-based fake OOD data and rich semantic embeddings (`anchors') from the ChatGPT description of ID knowledge to help guide the training of the image encoder. The learning framework can be flexibly combined with existing post-hoc approaches to OOD detection, and extensive empirical evaluations on multiple OOD detection benchmarks demonstrate that rich textual representation of ID knowledge and fake OOD knowledge can well help train a visual encoder for OOD detection. With the learning framework, new state-of-the-art performance was achieved on all the benchmarks. The code is available at https://github.com/Cverchen/TagFog.

Recent studies on out-of-distribution (OOD) detection focus on designing models or scoring functions that can effectively distinguish between unseen OOD data and in-distribution (ID) data. In this paper, we propose a simple yet novel ap- proach to OOD detection by leveraging the phenomenon that the average of feature vector elements from convolutional neural network (CNN) is typically larger for ID data than for OOD data. Specifically, the average of feature vector elements is used as part of the scoring function to further separate OOD data from ID data. We also provide mathematical analysis to explain this phenomenon. Experimental evaluations demonstrate that, when combined with a strong baseline, our method can achieve state-of-the-art performance on several OOD detection benchmarks. Furthermore, our method can be easily integrated into various CNN architectures and requires less computation. Source code address: https://github.com/SYSU-MIA-GROUP/statistical_discrepancy_ood.

Out-of-distribution (OOD) detection is vital for the safe application of intelligent systems in real-world scenarios. This paper proposes an enhancement to OOD detection by leveraging the consistency in cognition between two models, both pretrained on in-distribution (ID) data. Specifically, for a given test sample, we first apply Principal Component Analysis (PCA)-based projection on the feature vectors from each model. These obtained feature vectors (with correlation between dimensions decoupled by PCA projection) are then aligned using a multiple linear mapping, which is fitted using the least squares method on the training data. We hypothesize that the regression error for OOD data will be larger than that for ID data, making it a useful metric for OOD detection. Our experimental results demonstrate the effectiveness of this method. When combined with existing robust baselines, our approach achieves state-of-the-art performance in OOD detection.

Out-of-Distribution (OOD) detection is crucial when deploying machine learning models in open-world applications. The core challenge in OOD detection is mitigating the model's overconfidence on OOD data. While recent methods using auxiliary outlier datasets or synthesizing outlier features have shown promising OOD detection performance, they are limited due to costly data collection or simplified assumptions. In this paper, we propose a novel OOD detection framework FodFoM that innovatively combines multiple foundation models to generate two types of challenging fake outlier images for classifier training. The first type is based on BLIP-2's image captioning capability, CLIP's vision-language knowledge, and Stable Diffusion's image generation ability. Jointly utilizing these foundation models constructs fake outlier images which are semantically similar to but different from in-distribution (ID) images. For the second type, GroundingDINO's object detection ability is utilized to help construct pure background images by blurring foreground ID objects in ID images. The proposed framework can be flexibly combined with multiple existing OOD detection methods. Extensive empirical evaluations show that image classifiers with the help of constructed fake images can more accurately differentiate real OOD image from ID ones. New state-of-the-art OOD detection performance is achieved on multiple benchmarks. The code is available at https://github.com/Cverchen/ACMMM2024-FodFoM.

Federated Learning (FL) transforms collaborative machine learning by enabling data privacy-preserving model training across dispersed devices. Unfortunately, several challenges need to be concerned, such as the non-ID features of cross-client data and the possibility of backdoor attacks, which can result in inconsistent models, inefficiency, and security issues. To address these issues, we propose hypernetwork-based personalization and poisoning-free federated learning (HBIpFL), a unique architecture to improve security and personalization in FL. HBIpFL dramatically reduces communication overhead without sacrificing performance by using a hypernetwork-based parameter classifier to dynamically analyze and only upload the most important model parameters. Furthermore, it utilizes Local Gradient Ascent (LGA) methods to track training loss trends and identify possible backdoor intrusions, guaranteeing the resilience and dependability of the global model. We then compare HBIpFL to the state-of-the-art methods in the context of accuracy, communication efficiency, and defense against adversarial attacks. The results demonstrate that our HBIpFL offers a secure and effective FL environment for practical scenarios with a wide range of data distributions and strict privacy specifications.

The anatomy of service status through health monitoring models is essential to long-term structural safety. Since deformation is the intuitive representation of the dam's operating condition, it is crucial to investigate the deformation monitoring model with high accuracy and strong interpretability for the safety management of high arch dams. A novel deformation monitoring model is proposed by incorporating impulse response-based equivalent temperature (ET) and machine learning-aided separate modeling technique (SMT). This methodology has three main sources of novelty. First, the impulse response-based equivalent temperature and corresponding temperature component are derived from heat transfer and temperature convolution theories. Then, the model parameters are identified by an improved firefly algorithm. Second, the components in the monitored displacement are progressively stripped out by clustering analysis, separation under equal water level conditions, and a robust signal decomposition technique. Third, the separated deformation components and environmental factors are modeled by the multi-output deep extreme learning machine (DELM) with autoencoder, and the monitoring model ET-SMT-DELM is thus established. The world's highest arch dam is selected to illustrate the proposed model, and the prediction accuracy, early warning performance, component shares, and impulse response mechanism are comprehensively investigated by comparing with several typical baseline models. The results show that the overfitting of the proposed model is reduced, and the prediction and early warning performance is significantly improved. The resulting temperature impulse response function and component shares imply that the interpretability of the model is also enhanced.