Multi-modal Retrieval-Augmented Generation (MMRAG) enables highly credible generation by integrating external multi-modal knowledge, thus demonstrating impressive performance in complex multi-modal scenarios. However, existing MMRAG methods fail to clarify the reasoning logic behind retrieval and response generation, which limits the explainability of the results. To address this gap, we propose to introduce reinforcement learning into multi-modal retrieval-augmented generation, enhancing the reasoning capabilities of multi-modal large language models through a two-stage reinforcement fine-tuning framework to achieve explainable multi-modal retrieval-augmented generation. Specifically, in the first stage, rule-based reinforcement fine-tuning is employed to perform coarse-grained point-wise ranking of multi-modal documents, effectively filtering out those that are significantly irrelevant. In the second stage, reasoning-based reinforcement fine-tuning is utilized to jointly optimize fine-grained list-wise ranking and answer generation, guiding multi-modal large language models to output explainable reasoning logic in the MMRAG process. Our method achieves state-of-the-art results on WebQA and MultimodalQA, two benchmark datasets for multi-modal retrieval-augmented generation, and its effectiveness is validated through comprehensive ablation experiments.

Large-scale pre-trained audio-language models excel in general multi-modal representation, facilitating their adaptation to downstream audio recognition tasks in a data-efficient manner. However, existing few-shot audio recognition methods based on audio-language models primarily focus on learning coarse-grained correlations, which are not sufficient to capture the intricate matching patterns between the multi-level information of audio and the diverse characteristics of category concepts. To address this gap, we propose multi-grained correspondence learning for bootstrapping audio-language models to improve audio recognition with few training samples. This approach leverages generative models to enrich multi-modal representation learning, mining the multi-level information of audio alongside the diverse characteristics of category concepts. Multi-grained matching patterns are then established through multi-grained key-value cache and multi-grained cross-modal contrast, enhancing the alignment between audio and category concepts. Additionally, we incorporate optimal transport to tackle temporal misalignment and semantic intersection issues in fine-grained correspondence learning, enabling flexible fine-grained matching. Our method achieves state-of-the-art results on multiple benchmark datasets for few-shot audio recognition, with comprehensive ablation experiments validating its effectiveness.

Distracted driving poses a serious threat to traffic safety and remains a widespread problem, highlighting the crucial need for effective recognition of distracted drivers. However, developing models that can generalize across diverse and changing real-world driving conditions is profoundly challenging. Variations in factors like lighting, weather, vehicle type, and drivers complicate generalization. Addressing this critical limitation is key to building recognition models with robust performance for practical deployment. This study presents a novel two-stage unsupervised domain adaptation framework to tackle the important challenge of recognizing distracted drivers across differing environments. The framework first constructs softly assigned class prototypes capturing underlying data structure by aggregating features locally and reweighting sample-prototype relationships globally, which increases the accuracy of class representations. The framework then aligns the probabilities between test samples and prototypes across source and target domains using soft distributional alignment, reducing domain gaps without explicit labeling of the target data. A growth control function balances prototype alignment with classification and adversarial losses. Experiments on distracted driver and object recognition datasets demonstrate this two-stage approach outperforms previous methods, especially under changing driving environments, which is an important problem distracted driving detection research must overcome to effectively enhance road safety.

The purpose of audio-text retrieval is to learn a cross-modal similarity function between audio and text, enabling a given audio/text to find similar text/audio from a candidate set. Recent audio-text retrieval models aggregate multi-modal features into a single-grained representation. However, single-grained representation is difficult to solve the situation that an audio is described by multiple texts of different granularity levels, because the association pattern between audio and text is complex. Therefore, we propose an adaptive aggregation strategy to automatically find the optimal pool function to aggregate the features into a comprehensive representation, so as to learn valuable multi-grained representation. And multi-grained comparative learning is carried out in order to focus on the complex correlation between audio and text in different granularity. Meanwhile, text-guided token interaction is used to reduce the impact of redundant audio clips. We evaluated our proposed method on two audio-text retrieval benchmark datasets of Audiocaps and Clotho, achieving the state-of-the-art results in text-to-audio and audio-to-text retrieval. Our findings emphasize the importance of learning multi-modal multi-grained representation.

Abstract This paper proposed a novel precise point set registration method based on feature fusion for three-dimensional data. Firstly, for the prominent foreground with dense and continuous cluster structure, we propose an automatic extraction method combining the principal component analysis projection and density-based clustering method. Secondly, for point sets containing noises, we introduce correntropy measurement into registration to weaken their influence. Thirdly, for the precise registration of uneven distribution of points in the same point set, we propose a feature fusion based algorithm which is distribution specific, using point-to-point measurement for densely distributed foreground and point-to-plane measurement for sparsely distributed background, in case that only one measurement method is used for the whole point set the registration gets trapped into local extremum. Finally, we give the optimization algorithm of the proposed method. We conduct experiments on real orthodontics scenes to verify the effectiveness of our proposed feature extraction method and registration algorithm, and experimental results demonstrate that both the proposed solutions are proper for their respective tasks than other existing methods.

In this study, the authors present a novel individual retrieval method based on oral cavity point cloud data and correntropy‐based registration algorithm. Since the three‐dimensional oral cavity data contains a large amount of noise and outliers, it may lead to a decrease in registration accuracy, which affects the accuracy of retrieval rate. Therefore, the authors introduce the correntropy into the rigid registration algorithm to solve this problem. Then, they filter the matched point cloud data and then use the mean squared error to judge the individual differences of the model data. Finally, the accurate retrieval of the oral cavity data is realised. Experimental results demonstrate the proposed retrieval three‐dimensional model algorithm can be successfully searched under different model data, which can help forensics use the characteristics of biological individuals to accurately search and identify, and improve recognition efficiency.