We propose a methodology for estimating human behaviors in psychotherapy sessions using multi-label and multi-task learning paradigms. We discuss the problem of behavioral coding in which data of human interactions are annotated with labels to describe relevant human behaviors of interest. We describe two related, yet distinct, corpora consisting of therapist-client interactions in psychotherapy sessions. We experimentally compare the proposed learning approaches for estimating behaviors of interest in these datasets. Specifically, we compare single and multiple label learning approaches, single and multiple task learning approaches, and evaluate the performance of these approaches when incorporating turn context. We demonstrate that the best multi-label, multi-task learning model with turn context achieves 18.9 and 19.5 percent absolute improvements with respect to a logistic regression classifier (for each behavioral coding task respectively) and 6.4 and 6.1 percent absolute improvements with respect to the best single-label, single-task deep neural network models. Lastly, we discuss the insights these modeling paradigms provide into these complex interactions including key commonalities and differences of behaviors within and between the two prevalent psychotherapy approaches–Motivational Interviewing and Cognitive Behavioral Therapy–considered.

In this work we address the problem of joint prosodic and lexical behavioral annotation for addiction counseling. We expand on past work that employed Recurrent Neural Networks (RNNs) on multimodal features by grouping and classifying subsets of classes. We propose two implementations: One is hierarchical classification, which uses the behavior confusion matrix to cluster similar classes and makes the prediction based on a tree structure. The second is a graph-based method which uses the result of the original classification just to find a certain subset of the most probable candidate classes, where the candidate sets of different predicted classes are determined by the class confusions. We make a second prediction with simpler classifier to discriminate the candidates. The evaluation shows that the strict hierarchical approach degrades performance, likely due to error propagation, while the graph-based hierarchy provides significant gains.

In this paper, we present an approach for predicting utterance level behaviors in psychotherapy sessions using both speech and lexical features. We train long short term memory (LSTM) networks with an attention mechanism using words, both manually and automatically transcribed, and prosodic features, at the word level, to predict the annotated behaviors. We demonstrate that prosodic features provide discriminative information relevant to the behavior task and show that they improve prediction when fused with automatically derived lexical features. Additionally, we investigate the weights of the attention mechanism to determine words and prosodic patterns which are of importance to the behavior prediction task.

We propose a computational methodology for automatically estimating human behavioral patterns using the multiple instance learning (MIL) paradigm. We describe the incremental diverse density algorithm, a particular formulation of multiple instance learning, and discuss its suitability for behavioral coding. We use a rich multi-modal corpus comprised of chronically distressed married couples having problem-solving discussions as a case study to experimentally evaluate our approach. In the multiple instance learning framework, we treat each discussion as a collection of short-term behavioral expressions which are manifested in the acoustic, lexical, and visual channels. We experimentally demonstrate that this approach successfully learns representations that carry relevant information about the behavioral coding task. Furthermore, we employ this methodology to gain novel insights into human behavioral data, such as the local versus global nature of behavioral constructs as well as the level of ambiguity in the expression of behaviors through each respective modality. Finally, we assess the success of each modality for behavioral classification and compare schemes for multimodal fusion within the proposed framework.

Therapist language plays a critical role in influencing the overall quality of psychotherapy. Notably, it is a major contributor to the perceived level of empathy expressed by therapists, a primary measure for judging their efficacy. We explore psycholinguistics inspired features for predicting therapist empathy. These features model language which conveys information about affective and cognitive processes, which is central to the therapist expressing understanding of the patient’s perspective. We describe the dimensional features obtained based on psycholinguisitic norms, and their application to predicting empathy expressed in motivational interviewing sessions for addiction counseling. We compare these to standard lexical features (n-grams) and demonstrate that these features contain complementary information for predicting therapist empathy. The highest empathy prediction results achieved are 75.28% UAR and 0.6112 Spearman’s correlation.

We present Barista, an open-source framework for concurrent speech processing based on the Kaldi speech recognition toolkit and the libcppa actor library. With Barista, we aim to provide an easy-to-use, extensible framework for constructing highly customizable concurrent (and/or distributed) networks for a variety of speech processing tasks. Each Barista network specifies a flow of data between simple actors, concurrent entities communicating by message passing, modeled after Kaldi tools. Leveraging the fast and reliable concurrency and distribution mechanisms provided by libcppa, Barista lets demanding speech processing tasks, such as real-time speech recognizers and complex training workflows, to be scheduled and executed on parallel (and/or distributed) hardware. Barista is released under the Apache License v2.0.

We present a novel method of learning multiple disjunct concepts with diverse density using an incremental approach. We demonstrate that by maximizing the diverse density over individual target concept points and minimizing the probability of their intersection, concepts can be learned incrementally. This method reduces the complexity of the algorithm from factorial, with respect to the number of targets, to exponential order. We demonstrate that this greedy approach successfully learns disjunctive target concepts with competitive classification accuracy on a benchmark multiple instance learning dataset in comparison to other common diverse density approaches. We also introduce a novel application of the multiple instance learning framework to an emotion recognition task using prosodic and spectral speech features.

This paper introduces a supervised statistical framework for estimating the signal-to-noise (SNR) ratio of speech signals. Information on how noise corrupts a signal can help us compensate for its effects, especially in real life applications where the usual assumption of white Gaussian noise does not hold and speech boundaries in the signal are not known. We use features from which we can detect speech regions in a signal, without using Voice Activity Detection, and estimate the energies of those regions. Then we use these features to train ordinary least squares regression models for various noise types. We compare this supervised method with state-of-the-art SNR estimation algorithms and show its superior performance with respect to the tested noise types.

We compare the performance of Directional Derivatives features for automatic speech recognition when extracted from different time-frequency representations. Specifically, we use the short-time Fourier transform, Mel-frequency, and Gammatone spectrograms as a base from which we extract spectrotemporal modulations. We then assess the noise robustness of each representation with varied number of frequency bins and dynamic range compression schemes for both word and phone recognition. We find that the choice of dynamic range compression approach has the most significant impact on recognition performance. Whereas, the performance differences between perceptually motivated filter-banks are minimal in the proposed framework. Furthermore, this work presents significant gains in speech recognition accuracy for low SNRs over MFCCs, GFCCs, and Directional Derivatives extracted from the log-Mel spectrogram.

Signal-derived measures can provide effective ways towards quantifying human behavior. Verbal Response Latencies (VRLs) of children with Autism Spectrum Disorders (ASD) during conversational interactions are able to convey valuable information about their cognitive and social skills. Motivated by the inherent gap between the external behavior and inner affective state of children with ASD, we study their VRLs in relation to their explicit but also implicit behavioral cues. Explicit cues include the children's language use, while implicit cues are based on physiological signals. Using these cues, we perform classification and regression tasks to predict the duration type (short/long) and value of VRLs of children with ASD while they interacted with an Embodied Conversational Agent (ECA) and their parents. Since parents are active participants in these triadic interactions, we also take into account their linguistic and physiological behaviors. Our results suggest an association between VRLs and these externalized and internalized signal information streams, providing complementary views of the same problem.

Speech and spoken language cues offer a valuable means to measure and model human behavior. Computational models of speech behavior have the potential to support health care through assistive technologies, informed intervention, and effi-cient long-term monitoring. The Interspeech 2013 Autism Sub-Challenge addresses two developmental disorders that manifest in speech: autism spectrum disorders and specific language im-pairment. We present classification results with an analysis on the development set including a discussion of potential con-founds in the data such as recording condition differences. We hence propose study of features within these domains that may inform realistic separability between groups as well as have the potential to be used for behavioral intervention and monitoring. We investigate template-based prosodic and formant modeling as well as goodness of pronunciation modeling, reporting above chance classification accuracies. Index Terms: autism spectrum disorders, intonation, specific language impairment, goodness of pronunciation

We introduce a novel spectro-temporal representation of speech by applying directional derivative filters to the Melspectrogram, with the aim of improving the robustness of automatic speech recognition. Previous studies have shown that two-dimensional wavelet functions, when tuned to appropriate spectral scales and temporal rates, are able to accurately capture the acoustic modulations of speech, even in high noise conditions. Therefore, spectro-temporal features extracted from the wavelet transformation of the spectrogram, offer additional noise robustness to important signal processing tasks, such as voice activity detection and speech recognition. In this paper, we explore the use of the steerable pyramid, a directional wavelet transform that is common in image processing, to derive a spectro-temporal feature representation of speech that can serve as an alternative to cepstral derivatives and Gabor filterbank features. We discuss their application for the task of robust automatic speech recognition. Experiments conducted on the Aurora-2 database demonstrate their competitive robustness to other state-of-the-art speech features, especially in low signalto-noise ratio conditions. Index Terms: spectro-temporal features, automatic speech recognition, directional wavelet transforms

We present a sequential Bayesian belief update algorithm for an emotional dialog agent’s inference and behavior. This agent’s purpose is to collect usage patterns of natural language description of emotions among a community of speakers, a task which can be seen as a type of computational ethnography. We describe our target application, an emotionally-intelligent agent that can ask questions and learn about emotions through playing the emotion twenty questions (EMO20Q) game. We formalize the agent’s algorithms mathematically and algorithmically and test our model experimentally in an experiment of 45 humancomputer dialogs with a range of emotional words as the independent variable. We found that 44% of these human-computer dialog games are completed successfully, in comparison with earlier work in which human-human dialogs resulted in 85% successful completion on average. Despite being lower than this upper-bound of human performance, especially on difficult emotion words, the subjects rated that the agent’s humanity was 6.1 on a 0 to 10 scale. This indicates that the algorithm we present produces realistic behavior, but that issues of data sparsity may remain.

Many organizations are introducing Service Oriented Architecture (SOA) as part of their business transformation projects to take advantage of the proposed benefits associated with using SOA. However, in many cases organizations don't necessarily know on which projects introducing SOA would be of value and show real benefits to the organization. In this paper we outline a method and pilot metrics framework (PMF) to help organization's select from a set of candidate projects those which would be most suitable for piloting SOA. The PMF is used as part of a method based on identifying a set of benefit and risk criteria, investigating each of the candidate projects, mapping them to the criteria and then selecting the most suitable project(s). The paper outlines a case study where the PMF was applied in a large government organization to help them select pilot projects and develop an overall strategy for introducing SOA into their organization.

Behavioral coding focuses on deriving higher-level behavioral annotations using observational data of human interactions. Automatically identifying salient events in the observed signal data could lead to a deeper understanding of how specific events in an interaction correspond to the perceived high-level behaviors of the subjects. In this paper, we analyze a corpus of married couples’ interactions, in which a number of relevant behaviors, e.g., level of acceptance, were manually coded at the sessionlevel. We propose a multiple instance learning approach called Diverse Density Support Vector Machines, trained with acoustic features, to classify extreme cases of these behaviors, e.g., low acceptance vs. high acceptance. This method has the benefit of identifying salient behavioral events within the interactions, which is demonstrated by comparable classification performance to traditional SVMs while using only a subset of the events from the interactions for classification. Index Terms: behavioral signal processing, multiple instance learning, diverse density, support vector machines

Embodied conversational agents (ECA) offer platforms for the collection of structured interaction and communication data. This paper discusses the data collected from the Rachel system, an ECA developed at the University of Southern California, for interactions with children with autism. Two dyads each com-posed of a child with autism and his parent participated in an experiment with two modes: interactions with and without the ECA present. The goal of this work is to assess the naturalness of the data recorded in the ECA interaction. This analysis was carried out using a classification framework with a prediction variable of the presence or absence of the ECA in the inter-action. The results demonstrate that it is possible to estimate whether or not a parent is interacting with the ECA using their speech data. However, it is not generally possible to do so for the child suggesting that the Rachel system is eliciting commu-nication data that is similar to that elicited through interactions between the child and his parent. Index Terms: Embodied conversational agent, multimodal in-terface, audio-video recording, autism, children’s speech