Collaborative problem solving requires students to jointly reason, negotiate, and regulate their learning. Understanding collaborative problem solving through student dialogue can inform timely identification of productive and unproductive collaborative behaviors. In this study, we investigate the use of large language models to automatically classify collaborative problem-solving dialogue segments into two categories: Productive and Unproductive. To support deeper analysis, we additionally explore classification of eight detailed collaborative problem-solving sub-categories. We present an error-augmented few-shot prompting method that incorporates misclassified examples to refine model understanding of classification boundaries. Using dialogue data from a middle school collaborative game-based learning environment, our approach substantially improves classification accuracy over zero-shot baselines. Qualitative analysis of the resulting models further highlights which dialogue types are most frequently misclassified, suggesting design implications for adaptive scaffolding. These findings demonstrate that large language models, when guided with targeted prompting strategies, can effectively recognize productive and unproductive dialogue in collaborative learning.

The responsible deployment of Large Language Models (LLMs) necessitates rigorous safety evaluations.However, a critical challenge arises from inconsistencies between an LLM's internal refusal decisions and external safety assessments, hindering effective validation.This paper introduces the concept of the 'refusal gap' to formally define these discrepancies.We then present a novel, refusal-aware red teaming framework designed to automatically generate test cases that expose such gaps.Our framework employs 'refusal probes', which leverage the target model's hidden states, to detect internal model refusals.These are subsequently contrasted with judgments from an external safety evaluator.The identified discrepancy serves as a signal to guide a red-teaming model in crafting test cases that maximize this refusal gap.To further enhance test case diversity and address challenges related to sparse rewards, we introduce a hierarchical, curiositydriven mechanism that incentivizes both refusal gap maximization and broad topic exploration.Empirical results demonstrate that our method significantly outperforms existing reinforcement learning-based approaches in generating diverse test cases and achieves a substantially higher discovery rate of refusal gaps.

Large Visual Language Models (VLMs), such as GPT-4V, have achieved impressive results in generating detailed and nuanced responses. Although researchers have proposed various benchmarks to evaluate VLM performance, they have often neglected the examination of inherent security capabilities, particularly by evaluating the logical comprehension of image information. To address this gap, this paper introduces a novel dataset, FlowJD, specifically designed to evaluate logical flowchart jailbreak capabilities in VLMs. We conduct a comprehensive evaluation on GPT-4o, GPT-4V, and seven other state-of-the-art VLMs, revealing jailbreak rates of up to 92.8%. Our findings reveal significant vulnerabilities in current VLMs concerning logical flowchart jailbreak, emphasizing the urgent need for robust and effective defenses in future VLM development.Warning: Some of the examples may be harmful!