In this article, we present a database of multimodal communication features extracted from debate speeches in the 2019 North American Universities Debate Championships (NAUDC). Feature sets were extracted from the visual (facial expression, gaze, and head pose), audio (PRAAT), and textual (word sentiment and linguistic category) modalities of raw video recordings of competitive collegiate debaters (N=716 6-minute recordings from 140 unique debaters). Each speech has an associated competition debate score (range: 67-96) from experienced judges as well as competitor demographic and per-round reflection surveys. We observe the fully multimodal model performs best in comparison to models trained on various compositions of individual modalities. We also find that the weights of some features (such as the expression ofjoyand the use of the word ”we”) change in direction between the aforementioned models. We use these results to highlight the value of a multimodal dataset for studying competitive, collegiate debate.

Since online discussion platforms can limit the perception of social cues, effective collaboration over videochat requires additional attention to conversational skills. However, self-affirmation and defensive bias theories indicate that feedback may appear confrontational, especially when users are not motivated to incorporate them. We develop a feedback chatbot that employs Motivational Interviewing (MI), a directive counseling method that encourages commitment to behavior change, with the end goal of improving the user's conversational skills. We conduct a within-subject study with 21 participants in 8 teams to evaluate our MI-agent 'MIA' and a non-MI-agent 'Roboto'. After interacting with an agent, participants are tasked with conversing over videochat to evaluate candidate résumés for a job circular. Our quantitative evaluation shows that the MI-agent effectively motivates users, improves their conversational skills, and is likable. Through a qualitative lens, we present the strategies and the cautions needed to fulfill individual and team goals during group discussions. Our findings reveal the potential of the MI technique to improve collaboration and provide examples of conversational tactics important for optimal discussion outcomes.

Video-conferencing is essential for many companies, but its limitations in conveying social cues can lead to ineffective meetings. We present MeetingCoach, an intelligent post-meeting feedback dashboard that summarizes contextual and behavioral meeting information. Through an exploratory survey (N=120), we identified important signals (e.g., turn taking, sentiment) and used these insights to create a wireframe dashboard. The design was evaluated with in situ participants (N=16) who helped identify the components they would prefer in a post-meeting dashboard. After recording video-conferencing meetings of eight teams over four weeks, we developed an AI system to quantify the meeting features and created personalized dashboards for each participant. Through interviews and surveys (N=23), we found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity. Based on our findings, we provide suggestions for future feedback system designs of video-conferencing meetings.

In spite of their ubiquity, our interactions with contemporary conversational agents (CA), such as Alexa, are still transactional in nature and lack the expressiveness of human-human communication. Conversational agents equipped with the ability to detect and address users' emotional and cognitive states could make our interactions with them more human. In this work, we investigate whether an empathetic CA can help mitigate boredom. We design a protocol in order to first elicit boredom in users, and explore strategies that attempt to mitigate their boredom with the help of two conversational agents, an empathetic agent and a non-empathetic agent, administered in a Wizard-of-Oz setting. We quantify their efficacy by measuring the effects on user mood and task performance. Our user study with 34 participants shows that the empathetic CA outperforms the non-empathetic CA with respect to modulating users' mood and performance.

Toxic, abusive, or disrespectful behavior analysis is a non-trivial problem previously addressed mostly from the language perspective. In this paper, we present a novel video dataset containing disrespect and non-disrespect labels, and introduce such behavior analysis by using visual cues. The dataset is collected from YouTube news show videos of two-party conversations, in which a host and a guest interact through teleconferencing. Each video is labeled by three trained raters to identify disrespect expressed through face and gesture, voice, and language. By resolving confounding factors, we generate the corresponding pairwise samples of non-disrespect. To particularly show the influence of visual cues in disrespectful interactions, we present 222 labeled clips (duration=974.41(s), mean duration=4.39(s)). We extract and analyze the facial action units (AVs) prevalent in disrespectful behavior. Our result shows statistically significant differences after Bonferroni correction for Inner Brow raise (AV01), Lip Corner Depressor (AV15), and Chin Raiser (AV17). For prediction, we build two classifiers using logistic regression and linear Support Vector Machine with 62.61 % and 61.48 % accuracy, respectively. For an in-depth analysis of overall face and gesture features, we conduct a qualitative analysis using theme extraction. Our qualitative analysis provides further insights on leveraging synchronous and asynchronous features, along with combining text and audio data with visual cues to better detect disrespect.

Telling a great story often involves a deliberate alteration of emotions. In this paper, we objectively measure and analyze the narrative trajectories of stories in public speaking and their impact on subjective ratings. We conduct the analysis using the transcripts of over 2000 TED talks and estimate potential audience response using over 5 million spontaneous annotations from the viewers. We use IBM Watson Tone Analyzer to extract sentence-wise emotion, language, and social scores. Our study indicates that it is possible to predict (with AUC as high as 0.88) the subjective ratings of the audience by analyzing the narrative trajectories. Additionally, we find that some trajectories (for example, a flat trajectory of joy) correlate well with some specific ratings (e.g. "Longwinded') assigned by the viewers. Such an association could be useful in forecasting audience responses using objective analysis.

The proliferation of location-based social networks allows people to access location-based services as a group. We address Group Optimal Sequenced Route (GOSR) queries that enable a group to plan a trip with a minimum aggregate trip distance. The trip starts from the source locations of the group members, goes via a predefined sequence of different point of interests (POIs) such as a restaurant, shopping center and movie theater, and ends at the destination locations of the group members. The aggregate trip distance can be the total or the maximum trip distance of the group members. We introduce a novel approach to efficiently compute group optimal sequenced routes in road networks. We exploit elliptical properties to refine the POI search space and develop efficient algorithms for GOSR queries. Experiments show that our approach outperforms a naive approach significantly in terms of processing time and I/Os.