Interactive Machine Learning (IML) systems promise to democratize AI by enabling human influence over model behavior, yet the cognitive and behavioral implications of user control remain understudied. We present an investigation of how system controllability affects human factors in IML through the lens of an extensible research platform designed for both pedagogical and research applications. Our system features a toggleable hyperparameter control panel that transforms a streamlined annotation interface into an adjustable learning environment, allowing users to directly manipulate model training dynamics including learning rates, optimizers, and regularization parameters. Through a controlled laboratory study with 46 participants performing Named Entity Recognition (NER) tasks, we used multimodal measurements combining subjective assessments (NASA-TLX), physiological measures (pupil dilation, galvanic skin response), and behavioral metrics to understand the human cost of algorithmic control.

Multimodal human conversational interactions are characterized by individual differences, ambiguity, and complex socio-cultural and situational dynamics. Modeling session structured affective interactions, including emotion expression during communicative exchanges presents challenges for centralized machine learning approaches. Traditional machine learning pipelines for identifying these behaviors typically rely on learning a single generalizable model across all individuals in the dataset. However, data heterogeneity suggests that a single centralized model cannot capture individuals' respective nuances. To confirm this, we analyze multimodal behavioral datasets of expressive emotive interactions and demonstrate the statistical heterogeneity in individuals' data distributions. This serves as motivation for our focus on personalization. To achieve that, we turn to federated learning (FL). FL offers a compelling framework that enables integrating a centralized model (for shared behavioral trends) and client models (for individualized behavioral expressions), while additionally offering privacy preservation. However, so far, FL research has often focused on image benchmarks with non-IID partitioned data among the clients, ignoring the heterogeneity of naturalistic interactions. To address this gap, we propose FedSession, a novel personalized federated learning (PFL) strategy that accommodates heterogeneous data distributions by explicitly modeling sessions as structured interaction units, where each client corresponds to a real individual participating in a conversation-based affective interaction. FedSession incorporates an additional tier to capture how interlocutors align their behaviors to one another in interactions. Experiments on affect recognition tasks involving dyadic sessions demonstrate that FedSession outperforms existing PFL strategies. Unlike prior PFL strategies, which often struggle to outperform even localized training performance, FedSession effectively addresses the challenges of personalization in heterogeneous, interactive emotion recognition settings.

We present a hierarchical neuro-symbolic control framework that couples a classical symbolic planner with a transformer-based policy to address long-horizon decision-making under uncertainty. At the high level, the planner assembles an interpretable sequence of operators that guarantees logical coherence with task constraints, while at the low level each operator is rendered as a sub-goal token that conditions a decision transformer to generate fine-grained actions directly from raw observations. This bidirectional interface preserves the combinatorial efficiency and explainability of symbolic reasoning without sacrificing the adaptability of deep sequence models, and it permits a principled analysis that tracks how approximation errors from both planning and execution accumulate across the hierarchy. Empirical studies in stochastic grid-world domains demonstrate that the proposed method consistently surpasses purely symbolic, purely neural, and existing hierarchical baselines in both success and efficiency, highlighting its robustness for sequential tasks.

We report on a trans-Atlantic PhD student mobility program that connects two graduate research training initiatives in the US and Ireland, centered on developing future researchers in artificial intelligence (AI) and machine learning (ML). We discuss both the structure of the student exchange experiences and share key learnings from this international collaboration. The most important lesson learned is that providing a structured mobility program and matched visiting pairs is a highly effective way to improve learning outcomes compared to more typical ad-hoc individual visits.

In American Sign Language (ASL), a prominent resource gap exists for affective computing datasets. This manuscript explores preliminary findings from an ongoing multimodal ASL corpus collection and analysis study, focusing on human-generated modalities (e.g., eye tracking, facial expression, head movement) and the expression of frustration and confusion among deaf and hard of hearing study participants. These affective states can be important for understanding user experiences in human-computer interaction or for offering system feedback towards enhancing AI-human collaboration. Expanding a data collection methodology from prior work involving English-speaking participants, this exploratory study seeks to discern characteristics associated with confused or frustrated affect states in collected signed language interactions. Such insights have the potential to facilitate the development of models capable of recognizing emotional expressions. Initial results reveal distinctions in the characteristics of self-annotated instances of participant frustration and confusion, with certain features showing some divergence between the two emotions.

This paper addresses an existing resource gap for studying complex emotional states when a speaker collaborates with a partner to solve a task. We present a novel dialogue resource — the MULTICOLLAB corpus — where two interlocutors, an instructor and builder, communicated through a Zoom call while sensors recorded eye gaze, facial action units, and galvanic skin response, with transcribed speech signals, resulting in a unique, heavily multimodal corpus. The builder received instructions from the instructor. Half of the builders were privately told to disobey the instructor’s directions. After the task, participants watched the Zoom recording and annotated their instances of frustration. In this study, we introduce this new corpus and perform computational experiments with time series transformers, using early fusion through time for sensor data and late fusion for speech transcripts. We then average predictions from both methods to recognize instructor frustration. Using sensor and speech data in a 4.5 second time window, we find that the fusion of both models yields 21% improvement in classification accuracy (with a precision of 79% and F1 of 63%) over a comparison baseline, demonstrating that complex emotions can be recognized when rich multimodal data from transcribed spoken dialogue and biophysical sensor data are fused.

This study introduces a novel multimodal corpus for expressive task-based spoken language and dialogue, focused on language use under frustration and surprise, elicited from three tasks motivated by prior research and collected in an IRB-approved experiment. The resource is unique both because these are understudied affect states for emotion modeling in language, and also because it provides both individual and dyadic multimodally grounded language. The study includes a detailed analysis of annotations and performance results for multimodal emotion inference in language use.

Our AI-focused traineeships for graduate students integrate research and education components to contribute to diversifying the AI research workforce. We describe the program and introduce multiple strategies to achieve interdisciplinarity, diversity, equity, inclusion, and accessibility. Early evaluation results are included.

Were assessments of undergraduate researchers in a 10-week summer computing research experience impacted by the pandemic? We compare three cohort years: (1) pre-pandemic (in-person REU; prior to pandemic onset), (2) in-pandemic (remote REU, post-onset during ongoing pandemic), and (3) post-pandemic (in-person REU, post-onset with pandemic in the background). We discuss two forms of 5-point assessment ratings. First, we examine assessments of research skills on 34 questions, with a repeated measure of 3 assessments per cohort year at the beginning, middle, and end of their experience. Then, we examine assessment of presentation skills collected at the beginning vs. the end of the experience for pairs of students in all cohorts, considering 13 rating questions. Students' performance was assessed higher pre-pandemic. Also, being remote impacted completion performance. Lastly, effects linger after a return to in-person experiences, indicating adjustment challenges.

Event identification in technical logbooks poses challenges given the limited logbook data available in specific technical domains, the large set of possible classes, and logbook entries typically being in short form and non-standard technical language. Technical logbook data typically has both a domain, the field it comes from (e.g., automotive), and an application, what it is used for (e.g., maintenance). In order to better handle the problem of data scarcity, using a variety of technical logbook datasets, this paper investigates the benefits of using transfer learning from sources within the same domain (but different applications), from within the same application (but different domains) and from all available data. Results show that performing transfer learning within a domain provides statistically significant improvements, and in all cases but one the best performance. Interestingly, transfer learning from within the application or across the global dataset degrades results in all cases but one, which benefited from adding as much data as possible. A further analysis of the dataset similarities shows that the datasets with higher similarity scores performed better in transfer learning tasks, suggesting that this can be utilized to determine the effectiveness of adding a dataset in a transfer learning task for technical logbooks.

We provide an experience report about a remote framework for early undergraduate research experiences, which was thematically focused on sensing humans computationally. The framework included three complementary components. First, students experienced a team-based research cycle online, spanning formulating research questions, conducting literature review, performing fully remote human subject data collection experiments and data processing, analyzing and making inference over acquired data with computational experimentation, and disseminating findings. Second, the virtual program offered a set of professional development activities targeted to developing skills and knowledge for graduate school and research career trajectories. Third, it offered interactional and cohort-networking programming for community-building. We discuss not only the unique challenges of the virtual format and the steps put in place to address them but also the opportunities that being online afforded to innovate undergraduate research training remotely. We evaluate the remote training intervention through the organizing team's post-program reflection and the students' perceptions conveyed in exit interviews and a mid-program focus group. In addition to outlining lessons learned about more or less successful framework elements, we offer recommendations for applying the framework at other institutions as well as how to transfer activities to in-person formats.

We report on the use of open-source natural language processing capabilities in a web-based interface to allow undergraduate students to apply what they have learned about formal natural language structures. The learning activities encourage students to interpret data in new ways, think originally about natural language, and critique the back-end NLP models and algorithms visualized on the user front end. This work is of relevance to AI resources developed for education by focusing on inclusivity of students from many disciplinary backgrounds. Specifically, we comprehensively extended a web-based system with new resources. To test the students' reactions to NLP analyses that offer insights into both the strengths and limitations of AI systems, we incorporated a range of automated analyses focused on language-independent processing or meaning representations which still represent challenges for NLP. We conducted a survey-based evaluation with students in open-ended case-based assignments in undergraduate coursework. Responses indicated that the students reinforced their knowledge, applied critical thinking about language and NLP applications, and used the application not to solve the assignment for them, but as a tool in their own effort to address the task. We further discuss how using interpretable visualizations of system decisions is an opportunity to learn about ethical issues in NLP, and how making AI systems interpretable may broaden multidisciplinary interest in AI in early educational experiences.

Farhad Akhbardeh, Cecilia Ovesdotter Alm, Marcos Zampieri, Travis Desell. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

To examine perceptions of faculty mentors of undergraduate research and their supervisors, this work discusses the results of surveys administered after 3 years of a summer CS-focused REU Site program. One survey was completed by administrators of faculty research mentors--deans and chairs--and the other was completed by faculty mentors. The surveys indicated a disconnect between how the groups assessed undergraduate research mentoring as an indicator of faculty productivity, and overt vs. covert recognition of undergraduate mentoring. Additional topics explored the effectiveness of internal communication of program outcomes and ways to improve it, as well as post-program continued mentoring engagement linking to perceptions of long-term student benefits.

We present work-in-progress reflecting on the initial year of a distinctive summer Research Experiences for Undergraduates (REU) program. Our REU model combines fundamental research in computational sensing with a scholarly context that connects computer science with computational liberal arts. Students are intellectually stimulated to make sense of people's behaviors and cognitive processes with multimodal sensing hardware and software. In doing so, they explore the fundamental challenges found at the intersection of computing, the human experience, and scientific interrogation. The placement of the human experience at the core of the research theme enables an environment that stimulates and cultivates an innovative undergraduate research model. We highlight outcomes from the first year and discuss three emerging practices that are central to our REU framework: (1) team-based collaborative training; (2) transdisciplinary integration; and (3) systematic prioritization of inclusiveness. We also describe how these practices are incorporated into our overall undergraduate research framework and touch upon lessons learned from feedback collected.

We construct a humans-in-the-loop supervised learning framework that integrates crowdsourcing feedback and local knowledge to detect job-related tweets from individual and business accounts. Using data-driven ethnography, we examine discourse about work by fusing language-based analysis with temporal, geospational, and labor statistics information.

University students in the United States are routinely asked to provide feedback on the quality of the instruction they have received. Such feedback is widely used by university administrators to evaluate teaching ability, despite growing evidence that students assign lower numerical scores to women and people of color, regardless of the actual quality of instruction. In this paper, we analyze students’ written comments on faculty evaluation forms spanning eight years and five STEM disciplines in order to determine whether open-ended comments reflect these same biases. First, we apply sentiment analysis techniques to the corpus of comments to determine the overall affect of each comment. We then use this information, in combination with other features, to explore whether there is bias in how students describe their instructors. We show that while the gender of the evaluated instructor does not seem to affect students’ expressed level of overall satisfaction with their instruction, it does strongly influence the language that they use to describe their instructors and their experience in class.

Human image understanding is reflected by individuals' visual and linguistic behaviors, but the meaningful computational integration and interpretation of their multimodal representations remain a challenge. In this paper, we expand a framework for capturing image-region annotations in dermatology, a domain in which interpreting an image is influenced by experts' visual perception skills, conceptual domain knowledge, and task-oriented goals. Our work explores the hypothesis that eye movements can help us understand experts' perceptual processes and that spoken language descriptions can reveal conceptual elements of image inspection tasks. We cast the problem of meaningfully integrating visual and linguistic data as unsupervised bitext alignment. Using alignment, we create meaningful mappings between physicians' eye movements, which reveal key areas of images, and spoken descriptions of those images. The resulting alignments are then used to annotate image regions with medical concept labels. Our alignment accuracy exceeds baselines using both exact and delayed temporal correspondence. Additionally, comparison of alignment accuracy between a method that identifies clusters in the images based on eye movement vs. a method that identifies clusters using image features suggests that the two approaches perform well on different types of images and concept labels. This suggests that an image annotation framework should integrate information from more than one technique to handle heterogeneous images. We also investigate the performance of the proposed aligner for dermatological primary morphology concept labels, as well as for lesion size or type and distribution-based categories of images.

Domestic abuse affects people of every race, class, age, and nation. There is sig-nificant research on the prevalence and ef-fects of domestic abuse; however, such re-search typically involves population-based surveys that have high financial costs. This work provides a qualitative analysis of do-mestic abuse using data collected from the social and news-aggregation website red-dit.com. We develop classifiers to detect submissions discussing domestic abuse, achieving accuracies of up to 92%, a sub-stantial error reduction over its baseline. Analysis of the top features used in detect-ing abuse discourse provides insight into the dynamics of abusive relationships. 1

Nicolas Schrading, Cecilia Ovesdotter Alm, Raymond Ptucha, Christopher Homan. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

Traditional content-based image retrieval techniques, which primarily rely on image content at the pixel level, are not effective in accessing images at the semantic level. Defining approaches to incorporate experts' perceptual and conceptual capabilities of image understanding in their domain of expertise into the retrieval processes promises to help bridge this semantic gap. Towards accomplishing this, we design and implement a novel multimodal interactive system for image retrieval. To incorporate human expertise, the system stores expert-derived information extracted from two human sensor modalities that intuitively relate to image search, eye movements and verbal descriptions, both generated by medical experts. Experimental evaluation of the system shows that by transferring experts' perceptual expertise and domain knowledge into image-based computational procedures, our system can take advantage of the different human-centered modalities' respective strengths and improve the retrieval performance over just using image-based features.

Understanding and characterizing perceptual expertise is a major bottleneck in developing intelligent systems. In knowledge-rich domains such as dermatology, perceptual expertise influences the diagnostic inferences made based on the visual input. This study uses eye movement data from 12 dermatology experts and 12 undergraduate novices while they inspected 34 dermatological images. This work investigates the differences in global and local temporal fixation patterns between the two groups using recurrence quantification analysis (RQA). The RQA measures reveal significant differences in both global and local temporal patterns between the two groups. Results show that experts tended to refixate previously inspected areas less often than did novices, and their refixations were more widely separated in time. Experts were also less likely to follow extended scan paths repeatedly than were novices. These results suggest the potential value of RQA measures in characterizing perceptual expertise. We also discuss potential use of the RQA method in understanding the interactions between experts' visual and linguistic behavior.

Information from Twitter messages have become an important area for research in computational analysis of natural language. As yet, much latent user attribute analysis on Twitter is unexplored. One reason is that only few latent attributes are explicitly defined by users on Twitter. This work presents and analyzes a data set annotated by Twitter users themselves for age and other useful attributes for use in latent attribute inference applications. We report on statistical analysis of the collected latent attributes and tweet information using association mining.

Human perceptual expertise has significant influence on medical image inspection. However, little is known regarding whether experts differ in their cognitive processing or what effective visual strategies they employ for examining medical images. To remedy this, we conduct an eye tracking experiment and collect both eye movement and verbal description data from three groups of subjects with different medical training levels. Each subject examines and describes 42 photographic dermatological images. We then develop a hierarchical probabilistic framework to extract the common and unique eye movement patterns exhibited among multiple subjects' fixation and saccadic eye movements within each expertise-specific group. Furthermore, experts' annotations of thought units on the transcribed verbal descriptions are time-aligned with these eye movement patterns to identify their semantic meanings. In this work, we are able to uncover the manner in which these subjects alternated their viewing strategies over the course of inspection, and additionally extract their perceptual expertise so that it can be used for advanced medical image understanding.

Emotion is expressed by prosodic cues, and this study uses the active interactive Genetic Algorithm to search a wide space for sad and angry parameters of intensity, F0, and duration in perceptual resynthesis experiments with users. This method avoids large recorded databases and is flexible for exploring prosodic emotion parameters. Solutions from multiple runs are analyzed graphically and statistically. Average results indicate parameter evolution by emotion, and appear best for sad speech. Solutions are quite successfully classified by CART, with duration as main predictor. 1

Whereas experimental studies on emotional speech often control for neutral semantics, speech in naturalistic speech corpora is characterized by contextual cues and non-neutral semantic content. Moreover, the target emotion of an utterance is generally unknown and must be inferred by the listener. Within the context of having child-directed expressive text-to-speech synthesis as goal, we describe a perceptual study based on an expressive spoken corpus of children’s stories with unknown emotional targets, and report on interannotator agreement in a forced-choice discrimination task. Moreover, a threshold of high agreement was used to establish subsets of confident exemplar utterances for emotional classes, comprising 35% of the initial corpus. The exemplars were clustered based on the differences from the default mean neutral for 11 global acoustic features, yielding clusters cutting across emotion boundaries, some of which reflected arousal levels, with the neutral exemplars showing particularly complex distributions. Moreover, the mean features for four emotional exemplar categories were contrasted against the default, finding both expected and contradictory tendencies, compared to previous reports. The results indicate that semantic and prosodic cues collaborate to express and reinforce emotional contents, while emotional sequencing seems likely to be another factor which contributes to emotional perception in this domain.