Reinforcement learning with verifiable rewards (RLVR) has significantly boosted the reasoning capability of language models (LMs).However, existing RLVR approaches train LMs based on their own on-policy responses and are constrained by the initial capability of LMs, thus prone to exploration stagnation, in which LMs fail to solve more training problems and cannot further learn from the training data.Some approaches try to address this by leveraging off-policy solutions to training problems, but rely on external expert guidance that is limited in availability and scalability.In this work, we propose LTE (Learning to reason from Trial and Error), an approach that hints LMs with their previously self-made mistakes, not requiring any external expert guidance.Experiments validate the effectiveness of LTE, which outperforms the normal group relative policy optimization (GRPO) by 5.02 in Pass@1 and 9.96 in Pass@k on average across six mathematical reasoning benchmarks for Qwen3-8B-Base and even performs better than methods that require external guidance.Further analysis confirms that LTE successfully mitigates exploration stagnation and enhances both exploitation and exploration during training.Our code is available at https://github.com/ JamyDon/LTE.17 × 13 = ?17 × 13 = 170 + 41 = 211.17 × 13 = 260 -39 = 231.17 × 13 = 100 + 21 = 121.17 × 13 = ?Hint: possible incorrect answers include 211, 231, 121 … 17 × 13 = 170 + 41 = 211.Wait, 211 is wrong.Must be 51! 17 × 13 = 170 + 51 = 221.17 × 13 = 260 -39 = 231.But the hint says 231 is incorrect … Aha, 260 -39 is 221! 17 × 13 = 100 + 21 = 121.But the hint refutes 121 … Oh, 17 × 13 = 100 + 70 + 30 + 21 = 221.Normal rollouts "die" here.LTE "revives" with hints!

In-Context derived Vector (ICV) methods extract task-relevant representations from large language models (LLMs) and reinject them during inference, achieving comparable performance to few-shot In-Context Learning (ICL) without repeated demonstration processing.However, existing ICV methods remain sensitive to ICL-specific factors, often use coarse or semantically fragmented representations as the source of the vector, and rely on heuristicbased injection positions, limiting their applicability.To address these issues, we propose Dynamic Vector (DyVec), which incorporates an Exhaustive Query Rotation (EQR) strategy to extract robust semantically aggregated latent representations by mitigating variance introduced by ICL.It then applies Dynamic Latent Segmentation and Injection to adaptively partition representations based on task complexity and leverages REINFORCE-based optimization to learn optimal injection positions for each segment.Experiments results show that DyVec outperforms few-shot ICL, LoRA, and prior ICV baselines.Further analysis highlights the effectiveness of dynamically segmenting and injecting semantically aggregated latent representations.DyVec provides a lightweight and data-efficient solution for inference-time task adaptation.

Deep multimodal semantic understanding that goes beyond the mere superficial content relation mining has received increasing attention in the realm of artificial intelligence. The challenges of collecting and annotating high-quality multi-modal data have underscored the significance of few-shot learning. In this paper, we focus on two critical tasks under this context: few-shot multi-modal sarcasm detection (MSD) and multi-modal sentiment analysis (MSA). To address them, we propose Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM). Specifically, we design three experts of soft prompts: a text prompt and an image prompt that extract modality-specific features to enrich the single-modal representation, and a unified prompt to assist multi-modal interaction. Additionally, we reorganize Transformer layers into several blocks and introduce cross-modal prompt attention between adjacent blocks, which smoothens the transition from single-modal representation to multi-modal fusion. On both MSD and MSA datasets in few-shot setting, our proposed model not only surpasses the 8.2B model InstructBLIP with merely 2% parameters (150M), but also significantly outperforms other widely-used prompt methods on VLMs or task-specific methods.

Ziyang Wang, Sanwoo Lee, Hsiu-Yuan Huang, Yunfang Wu. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.