Reflective listening is a fundamental communication skill in behavioral health counseling. It enables counselors to demonstrate an understanding of and empathy for clients’ experiences and concerns. Training to acquire and refine reflective listening skills is essential for counseling proficiency. Yet, it faces significant barriers, notably the need for specialized and timely feedback to improve counseling skills. In this work, we evaluate and compare several computational models, including transformer-based architectures, for their ability to assess the quality of counselors’ reflective listening skills. We explore a spectrum of neural-based models, ranging from compact, specialized RoBERTa models to advanced large-scale language models such as Flan, Mistral, and GPT-3.5, to score psychotherapy reflections. We introduce a psychotherapy dataset that encompasses three basic levels of reflective listening skills. Through comparative experiments, we show that a fine-tuned small RoBERTa model with a custom learning objective (Prompt-Aware margIn Ranking (PAIR)) effectively provides constructive feedback to counselors in training. This study also highlights the potential of machine learning in enhancing the training process for motivational interviewing (MI) by offering scalable and effective feedback alternatives for counseling training.

Computational models of psychotherapy often ignore vocal cues by relying solely on text. To address this, we propose MISQ, a framework that integrates speech features directly into language models using a speech encoder and lightweight adapter. MISQ improves behavioral analysis in counseling conversations, achieving ~5% relative gains over text-only or indirect speech methods—underscoring the value of vocal signals like tone and prosody.

In this paper, we study the problem of multi-reward reinforcement learning to jointly optimize for multiple text qualities for natural language generation. We focus on the task of counselor reflection generation, where we optimize the generators to simultaneously improve the fluency, coherence, and reflection quality of generated counselor responses. We introduce two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously. Specifically, we employ non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training. Through automatic and manual evaluations, we show that our proposed techniques, DynaOpt and C-DynaOpt, outperform existing naive and bandit baselines, showcasing their potential for enhancing language models.

Reflections are a core verbal skill used by mental health counselors to express understanding and acknowledgement of the client's experience and concerns.In this paper, we propose a system for the automatic evaluation of counselor reflections.Specifically, our system takes as input one dialog turn containing a client prompt likely leading to a reflection and a counselor response to it, and outputs a numeric score indicating the quality of the reflection made by the counselor.We compile a dataset consisting of reflections portraying different levels of reflective listening skills, and propose Prompt-Aware margIn Ranking (PAIR), a novel framework for reflection scoring that contrasts positive and negative prompt and response pairs using adhoc multi-gap and prompt-aware margin ranking losses.Through empirical evaluations and deployment of our system in a real-life educational environment, we show that our scoring model outperforms several baselines on different metrics, and can be used to provide useful feedback to counseling trainees.

The quality of a counseling intervention relies highly on the active collaboration between clients and counselors.In this paper, we explore several linguistic aspects of the collaboration process occurring during counseling conversations.Specifically, we address the differences between high-quality and low-quality counseling.Our approach examines participants' turn-by-turn interaction, their linguistic alignment, the sentiment expressed by speakers during the conversation, as well as the different topics being discussed.Our results suggest important language differences in lowand high-quality counseling, which we further use to derive linguistic features able to capture the differences between the two groups.These features are then used to build automatic classifiers that can predict counseling quality with accuracies of up to 88%.

We describe the use of an online patient portal to recruit and enroll primary care patients in a randomized trial testing the effectiveness of a colorectal cancer (CRC) screening decision support program. We use multiple logistic regression to identify patient characteristics associated with trial recruitment, enrollment, and engagement. We found that compared to Whites, Blacks had lower odds of viewing the portal message (OR = 0.46, 95% CI = 0.37-0.57), opening the attached link containing the study material (OR = 0.75, 95% CI = 0.62-0.92), and consenting to participate in the trial (OR = 0.85, 95% CI = 0.67-0.93). We also found that compared to Whites, Asians had lower odds of viewing the portal message (OR = 0.53, 95% CI = 0.33-0.64), opening the attached link containing the study material (OR = 0.76, 95% CI = 0.54-0.97), consenting to participate in the trial (OR = 0.68, 95% CI = 0.53-0.95), and completing the trial's baseline questionnaire (OR = 0.59, 95% CI = 0.36-0.90). While portals offer an opportunity to mitigate human bias in trial invitations, because of racial disparities-not only in who has a portal account, but in how they interact with trial recruitment and enrollment material within the portal-using portals alone for trial recruitment may generate study samples that are not racially diverse.

Verónica Pérez-Rosas, Rada Mihalcea, Kenneth Resnicow, Satinder Singh, Lawrence An. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.

Verónica Pérez-Rosas, Rada Mihalcea, Kenneth Resnicow, Satinder Singh, Lawrence An, Kathy J. Goggin, Delwyn Catley. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. 2017.