Large language models (LLMs) have shown promise in providing scalable mental health support, while evaluating their counseling capability remains crucial to ensure both efficacy and safety. Existing evaluations are limited by the static assessment that focuses on knowledge tests, the single perspective that centers on user experience, and the open-loop framework that lacks actionable feedback. To address these issues, we propose Ψ-Arena, an interactive framework for comprehensive assessment and optimization of LLM-based counselors, featuring three key characteristics: (1) Realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients; (2) Tripartite evaluation that integrates assessments from the client, supervisor, and counselor perspectives; (3) Closed-loop optimization that iteratively improves LLM counselors using diagnostic feedback. Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives. Moreover, reflection-based optimization results in up to a 141% improvement in counseling performance. We hope Ψ-Arena provides a foundational resource for advancing reliable and human-aligned LLM applications in mental healthcare.

Psychiatric interviewing is a strategic, goaloriented interaction that requires proactively steering the conversation to elicit latent information.However, existing methods often degenerate into rigid interrogation or aimless chitchat due to a lack of strategic planning.In this work, we introduce S 4 , a comprehensive framework grounded in Speech Act Theory, modeling the interview as a unified process of internal strategy (Illocution and Perlocution) and external realization (Locution).We synthesize a large-scale dataset with fine-grained psychiatric speech act annotations.Trained on this data, S 4 employs reinforcement learning driven by long-term therapeutic effects to optimize the strategic chaining of atomic acts, aiming to maximally elicit information and maintain patient engagement.Experiments demonstrate that S 4 significantly outperforms baselines, validating the effectiveness of our effectdriven strategic modeling.Action (A) Sample Locution (L) Definition & Intended Perlocution (P) I. Information Seeking (Directives: Eliciting Disclosure) Explore "How have you been sleeping?"Solicit Narrative: Ask open-ended questions to elicit detailed disclosure and expand symptom scope.Probe "Could you tell me more about that?" Deepen Inquiry: Follow up on ambiguity to clarify details and deepen focus.Confirm "Do you feel this way every day?" Pinpoint Fact: Ask closed-ended questions to verify diagnostic criteria.Clarify "By 'fatigue', I mean tiredness."Resolve Confusion: Provide explanations to align cognition and correct misunderstandings.II.Affective Regulation (Expressives: Modifying State) Validate "That sounds incredibly hard."Affirm Emotion: Acknowledge patient distress to lower defensiveness and build trust.Support "I understand.Please go on."Maintain Flow: Use back-channeling to demonstrate active listening and boost efficacy.Ease "Do you have any hobbies?"Reduce Tension: Engage in non-clinical conversation to de-escalate anxiety and humanize the agent. III. Interview Management (Representatives: Setting Frame)Initiate "Hi, I'm your AI counselor."Set Frame: Establish professional boundaries and the purpose of the session.Conclude "Thanks for sharing.Take care."Ensure Closure: Formally end the session to provide a safe exit and consolidation.

Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li, Weiran Nie, Cheng Li, Wei Peng, Minlie Huang. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.