Multimodal data significantly improves the performance of pretrained models, but its practical application is often limited by missing or incomplete data across modalities. There are two key challenges that existing methods of synthesizing missing data face: (1) semantic inaccuracies due to model hallucinations and (2) discrepancies in distribution preferences between generated and original data. To address these challenges, we propose a novel three-stage multimodal data augmentation framework (GFR), which Generate, Filter, and Rank missing modality data. Our framework leverages multimodal large models for diverse data generation, designs a scene graph matching-based filtering algorithm to ensure semantic consistency, and constructs a preference-aware ranking model to align the generated data with both the original distribution and task relevance. Our framework not only enhances semantic diversity and consistency in data generation but also effectively captures the implicit characteristics of the original dataset and the target model. We demonstrate the effectiveness of GFR across multiple datasets by testing different missing types and missing ratios.

The rapid development of generation models has brought potential risks, necessitating generated image detection. In the meantime, evaluating the generalization to unseen diffusion models has become the detectors’ major challenge. Existing methods attempt to train detectors using the reconstruction features obtained from diffusion reconstruction, but they use these features in a limited way, resulting in a lack of generalization ability. Therefore, we propose that there are some extra forgery clues implicit in the input image and reconstruction features, and design a Dual Branch re-Extraction (DBE) module to extract them. To obtain a more generalized feature representation, spatial and frequency features are extracted through the calculation of neighboring pixel relationships and the DWT-based high-frequency feature extractor, respectively. Extensive experiments demonstrate the superior performance of our method, with an average improvement of 5%/7% in AUROC/AP compared to recent state-of-the-art works.

Causal reasoning algorithms can evaluate the strength of causal relationships between events and play an essential role in revealing and understanding the underlying mechanisms of them. However, traditional algorithms find it difficult to handle the complex interaction effects between numerous variables, can only consider event information at a single moment, and are challenging to solve the uncertainty problem in temporal evolution. To address these challenges, this paper proposes a temporal knowledge graph causal reasoning model (TKGR) that combines fuzzy logic and graph attention networks. Graph attention networks are used to capture the complex interaction effects between numerous variables at a single moment. At the same time, a multi-head self-attention mechanism is used to aggregate temporal information from multiple time points. In response to the uncertainty problem in time series evolution, we also designed a fuzzy logic module to comprehensively consider the importance of features from different time points. Finally, this paper carried out multiple rounds of comparative experiments with traditional causal reasoning algorithms and ablation experiments on the simulation dataset of satellite reconnaissance missions. The results show that the TKGR model can effectively reason the strength of causal relationships, thus providing necessary reference for decisionmaking.

Code comments are essential for programming comprehension. Nevertheless, developers often neglect to update comments after modifying the source code. Wrong code comments may lead to bugs in the maintenance process, thus affecting the reliability of the software. So, timely comment mismatch detection is crucial for software development and maintenance. However, existing works have the following two limitations: 1) the lack of use of code structural and sequential information, and 2) the ignorance of existing associations between code and comments. In this paper, we propose a new model called ASKDetector (AST-Semantic and Key features fusion based mismatch Detector). For the first limitation, we encode code with an attention-based preorder traversal abstract syntax tree sequence to obtain both order and structural information. And CodeBERT is utilized to capture contextual semantic features further. For the second one, we encode extracted association information between the code snippets and comments to reduce the semantic gap. The correlations between the encoders are learned through a fusion layer and a multi-layer perceptron. The experimental results prove that our detector outperforms the state-of-the-art model in evaluation metrics, where our F1 and accuracy exceed an average of 3.4%.

In multimodal learning for visual recognition, missing modality is a common issue that can significantly impact the performance and robustness of vision-language models. Most existing approaches have only considered the situation where a single modality-either image or text-is missing and then use a data augmentation method to recover the missing modality data. However, in reality, it is common for either text or image to be missing, and in such cases, a data augmentation method that is effective for one modality might not be suitable for the other, thereby necessitating distinct methods for text and image data augmentation. There are also approaches aimed at enhancing the robustness of vision-language models to handle missing data inputs. However since most of these approaches often involve significant modifications to complex model structures and require extensive retraining, these solutions would be impractical with limited computational resources. To address the abovementioned limitations, we develop a Multi-source Augmentation and Composite Prompts method (MACP) to alleviate the performance degradation due to missing modalities from both data and model levels. On the data level, we designed a multi-source data augmentation framework that integrates different data augmentation methods and a data selector to restore the missing data for each image-text sample as well as possible. On the model level, we designed a method for generating prompt vectors that simultaneously indicate the missing modalities in the model input and the source of augmentation data. The prompts will enhance the ability of the vision-language model to handle different input types in low-resource situations by applying prompt tuning. Experimental results demonstrate the effectiveness of our approach in mitigating the impact of modality missing on three vision-language datasets. Code is available.

Developers often call APIs to improve development efficiency, but they misuse APIs due to lack of understanding of source code logic and other unavoidable reasons, resulting in serious consequences such as program crashes. Many studies that extract API usage constraints from API documentation or codebases expect to get out of this dilemma through API misuse detection. However, low recall remains a hurdle for researchers to overcome. In this work, we make full use of API documentation and codebases to construct constraint knowledge graph, and propose a new API misuse detector, MisuseHint. We precisely define API constraints into seven categories, utilize API caveat knowledge in documentation and API usage patterns in codebases, and fuse knowledge from both to build knowledge graph with rich constraints. To detect API misuses, we obtain API usage constraints in the knowledge graph and analyze static code to propose different strategies to determine whether API misuses exist. Through defect pattern analysis, object variable tracking, and Z3 SAT solver, our detector can identify various complex situations of code at a fine-grained level, especially solving various complex problems of Call Order and State Checking constraints. Experimental results on MUBench show that our recall reaches 39.78%, demonstrating the validity and theoretical feasibility of fusing documentation and codebases using knowledge graphs. MisuseHint achieves a recall of 76.34% when it is always given sufficient API constraints. This detector can practically help developers program effectively.