While fusing the capacities and advantages of various large language models offers a pathway to construct more powerful and versatile models, a fundamental challenge is to properly select advantageous model during training. Existing fusion methods primarily focus on the training mode that uses cross entropy on ground truth in a teacher-forcing setup to measure a model's advantage, which may provide limited insight towards model advantage. In this paper, we introduce a novel approach that enhances the fusion process by incorporating both the training and inference modes. Our method evaluates model advantage not only through cross entropy during training but also by considering inference outputs, providing a more comprehensive assessment. To combine the two modes effectively, we introduce ProFuser to progressively transition from inference mode to training mode. To validate ProFuser's effectiveness, we fused three models, including Vicuna-7B-v1.5, Llama-2-7B-Chat, and MPT-7B-8K-Chat, and demonstrated the improved performance in knowledge, reasoning, and safety compared to baseline methods.

Large language models (LLMs) have shown strong potential in enhancing text clustering when combined with traditional embedding models.However, existing methods predominantly treat LLMs as static pseudo-oracles, i.e., unidirectionally querying them for similarity assessment or data augmentation, while never seeking feedback from embedding models to improve them.In this work, we propose a training framework that enables bidirectional refinement between LLMs and embedding models.We first design task-aware prompts to guide the LLM in generating interpretations for the input texts.These interpretations are projected into the embedding space, in which interpretations that are preferred by the embedding model are selected based on their distribution densities.The selected interpretations are then used to fine-tune the LLM via preference optimization to prioritize the generation of helpful interpretations.Meanwhile, we enhance the embedding model via contrastive learning on the generated interpretations and perform clustering on the output embeddings, leading to iterative cotraining between the LLM and the embedding model.Experiments on 14 benchmark datasets across 5 tasks demonstrate the effectiveness of our method.

The detection of telecom fraud faces significant challenges due to the lack of high-quality multimodal training data that integrates audio signals with reasoning-oriented textual analysis. To address this gap, we present TeleAntiFraud-28k, the first open-source audio-text slow-thinking dataset specifically designed for automated telecom fraud analysis. Our dataset is constructed through three strategies: (1) Privacy-preserved text-truth sample generation using automatically speech recognition-transcribed call recordings (with anonymized original audio), ensuring real-world consistency through text-to-speech model regeneration; (2) Semantic enhancement via large language model based self-instruction sampling on authentic ASR outputs to expand scenario coverage; (3) Multi-agent adversarial synthesis, which simulates emerging fraud tactics through predefined communication scenarios and fraud typologies, enriches the conversation samples. The generated dataset contains 28,511 rigorously processed audio-text pairs with a total audio duration of more than 307 hours, complete with detailed annotations for fraud reasoning. The dataset is divided into three tasks: scenario classification, fraud detection, fraud type classification. Furthermore, we construct TeleAntiFraud-Bench, a standardized evaluation benchmark comprising proportionally sampled instances from TeleAntiFraud-28k, to facilitate systematic testing of model performance, reasoning capabilities, and thought processes on telecom fraud detection tasks. We also contribute a supervised fine-tuning model based on Qwen2-Audio, trained on the TeleAntiFraud-28k training set, while open-sourcing the data processing framework to enable community-driven dataset expansion. This work establishes a foundational framework for multimodal anti-fraud research while addressing critical challenges in data privacy and scenario diversity. The code of this paper is publicly available at https://github.com/JimmyMa99/TeleAntiFraud.