Clinical free-text notes contain vital patient information. They are structured into labelled sections; recognizing these sections has been shown to support clinical decision-making and downstream NLP tasks. In this paper, we advance clinical section segmentation through three key contributions. First, we curate a new de-identified, section-labeled obstetrics notes dataset, to supplement the medical domains covered in public corpora such as MIMIC-III, on which most existing segmentation approaches are trained. Second, we systematically evaluate transformer-based supervised models for section segmentation on a curated subset of MIMIC-III (in-domain), and on the new obstetrics dataset (out-of-domain). Third, we conduct the first head-to-head comparison of supervised models for medical section segmentation with zero-shot large language models. Our results show that while supervised models perform strongly in-domain, their performance drops substantially out-of-domain. In contrast, zero-shot models demonstrate robust out-of-domain adaptability once hallucinated section headers are corrected. These findings underscore the importance of developing domain-specific clinical resources and highlight zero-shot segmentation as a promising direction for applying healthcare NLP beyond well-studied corpora, as long as hallucinations are appropriately managed.

Clarification Requests (CRs) play a crucial role in human communication. However, existing datasets are often limited to single-turn clarifications or simulated tasks. We present a novel task-oriented dialogue dataset, ClarVis, of CRs collected from real-time multi-user collaborative data-exploration sessions, where users analyze data together while interacting with a visualization-generating conversational assistant. This dataset fills important gaps in current CR datasets by identifying naturally occurring CRs grounded by real-world modalities like hearing, vision, and actions in the physical environment. Our dataset includes 6.3K utterances across collaborative tasks, where each CR is annotated with a grounding modality - Auditory (A), Visual (V), or Kinesthetic (K) - that captures the context the clarification pertains to. Further, we establish benchmark tasks for CR identification and grounding-modality classification, and evaluate them with traditional machine learning models as well as instruction-tuned large language models. The results highlight both the learnability and the difficulty of these tasks, and position ClarVis as a useful resource for studying clarifications in collaborative dialogue settings.

Abstract Clinical notes in Electronic Health Records (EHRs) capture rich temporal information on events, clinician reasoning, and lifestyle factors often missing from structured data. Leveraging them for predictive modeling can be impactful for timely identification of chronic diseases. However, they present core natural language processing (NLP) challenges: long text, irregular event distribution, complex temporal dependencies, privacy constraints, and resource limitations. We present two complementary methods for temporally and contextually grounded risk prediction from longitudinal notes. First, we introduce HITGNN, a hierarchical temporal graph neural network that integrates intranote temporal event structures, inter-visit dynamics, and medical knowledge to model patient trajectories with fine-grained temporal granularity. Second, we propose REVEAL, a lightweight test-time framework that distills LLMs’ reasoning into smaller verifier models. Applied to opportunistic screening for Type 2 Diabetes (T2D) using temporally realistic cohorts curated from private and public hospital corpora, HITGNN achieves the highest predictive accuracy—especially for near-term risk—while preserving privacy and limiting reliance on large proprietary models. REVEAL enhances sensitivity to true T2D cases and retains explanatory reasoning. Our ablations confirm the value of temporal structure and knowledge augmentation, and fairness analysis shows HITGNN performs more equitably across subgroups.

Temporal information extraction from unstructured text is essential for contextualizing events and deriving actionable insights, particularly in the medical domain.We address the task of extracting clinical events and their temporal relations using the well-studied I2B2 2012 Temporal Relations Challenge corpus.This task is inherently challenging due to complex clinical language, long documents, and sparse annotations.We introduce GRAPHTREX, a novel method integrating span-based entityrelation extraction, clinical large pre-trained language models (LPLMs), and Heterogeneous Graph Transformers (HGT) to capture local and global dependencies.Our HGT component facilitates information propagation across the document through innovative global landmarks that bridge distant entities and improves the state-of-the-art with 5.5% improvement in the tempeval F 1 score over the previous best and up to 8.9% improvement on long-range relations, which presents a formidable challenge.We further demonstrate generalizability by establishing a strong baseline on the E3C corpus.Not only does this work advance temporal information extraction, but also lays the groundwork for improved diagnostic and prognostic models through enhanced temporal reasoning.

Despite LLMs’ explicit alignment against demographic stereotypes, they have been shown to exhibit biases under various social contexts. In this work, we find that LLMs exhibit concerning biases in how they associate solution veracity with demographics. Through experiments across five human value-aligned LLMs on mathematics, coding, commonsense, and writing problems, we reveal two forms of such veracity biases: Attribution Bias, where models disproportionately attribute correct solutions to certain demographic groups, and Evaluation Bias, where models’ assessment of identical solutions varies based on perceived demographic authorship. Our results show pervasive biases: LLMs consistently attribute fewer correct solutions and more incorrect ones to African-American groups in math and coding, while Asian authorships are least preferred in writing evaluation. In additional studies, we show LLMs automatically assign racially stereotypical colors to demographic groups in visualization code, suggesting these biases are deeply embedded in models’ reasoning processes. Our findings indicate that demographic bias extends beyond surface-level stereotypes and social context provocations, raising concerns about LLMs’ deployment in educational and evaluation settings.

This paper studies the performance of large language models (LLMs), particularly regarding demographic fairness, in solving real-world healthcare tasks. We evaluate state-of-the-art LLMs with three prevalent learning frameworks across six diverse healthcare tasks and find significant challenges in applying LLMs to real-world healthcare tasks and persistent fairness issues across demographic groups. We also find that explicitly providing demographic information yields mixed results, while LLM’s ability to infer such details raises concerns about biased health predictions. Utilizing LLMs as autonomous agents with access to up-to-date guidelines does not guarantee performance improvement. We believe these findings reveal the critical limitations of LLMs in healthcare fairness and the urgent need for specialized research in this area.

We explore the potential of ChatGPT to generate conversations focused on self-care strategies for African-American patients with heart failure, a domain with limited specialized datasets. To simulate patient-health educator dialogues, we employed four prompting strategies: aspects, African American Vernacular English, Social Determinants of Health (SDOH), and SDOH-informed reasoning. Conversations were generated across key self-care aspects— food, exercise, and fluid intake—with varying turn lengths and incorporated patient-specific SDOH attributes such as age, gender, neighborhood, and socioeconomic status. Our findings show that effective prompt design is essential. While incorporating SDOH and reasoning improves dialogue quality, ChatGPT still lacks the empathy and engagement needed for meaningful healthcare communication.

We present Component ALignment for Abstract Meaning Representation (Calamr), a novel method for graph alignment that can support summarization and its evaluation. First, our method produces graphs that explain what is summarized through their alignments, which can be used to train graph based summarization learners. Second, although numerous scoring methods have been proposed for abstract meaning representation (AMR) that evaluate semantic similarity, no AMR based summarization metrics exist despite years of work using AMR for this task. Calamr provides alignments on which new scores can be based. The contributions of this work include a) a novel approach to aligning AMR graphs, b) a new summarization based scoring methods for similarity of AMR subgraphs composed of one or more sentences, and c) the entire reusable source code to reproduce our results.

The education domain has been a popular area of collaboration with NLP researchers for decades. However, many recent breakthroughs, such as large transformer based language models, have provided new opportunities for solving interesting, but difficult problems. One such problem is assigning sentiment to reviews of educators’ performance. We present EduSenti: a corpus of 1,163 Albanian and 624 English reviews of educational instructor’s performance reviews annotated for sentiment, emotion and educational topic. In this work, we experiment with fine-tuning several language models on the EduSenti corpus and then compare with an Albanian masked language trained model from the last XLM-RoBERTa checkpoint. We show promising results baseline results, which include an F1 of 71.9 in Albanian and 73.8 in English. Our contributions are: (i) a sentiment analysis corpus in Albanian and English, (ii) a large Albanian corpus of crawled data useful for unsupervised training of language models, and (iii) the source code for our experiments.

Health coaching helps patients achieve personalized and lifestyle-related goals, effectively managing chronic conditions and alleviating mental health issues. It is particularly beneficial, however cost-prohibitive, for low-socioeconomic status populations due to its highly personalized and labor-intensive nature. In this paper, we propose a neuro-symbolic goal summarizer to support health coaches in keeping track of the goals and a text-units-text dialogue generation model that converses with patients and helps them create and accomplish specific goals for physical activities. Our models outperform previous state-of-the-art while eliminating the need for predefined schema and corresponding annotation. We also propose a new health coaching dataset extending previous work and a metric to measure the unconventionality of the patient’s response based on data difficulty, facilitating potential coach alerts during deployment.

We find that language models have difficulties generating fallacious and deceptive reasoning.When asked to generate deceptive outputs, language models tend to leak honest counterparts but believe them to be false.Exploiting this deficiency, we propose a jailbreak attack method that elicits an aligned language model for malicious output.Specifically, we query the model to generate a fallacious yet deceptively real procedure for the harmful behavior.Since a fallacious procedure is generally considered fake and thus harmless by LLMs, it helps bypass the safeguard mechanism.Yet the output is factually harmful since the LLM cannot fabricate fallacious solutions but proposes truthful ones.We evaluate our approach over five safetyaligned large language models, comparing four previous jailbreak methods, and show that our approach achieves competitive performance with more harmful outputs.We believe the findings could be extended beyond model safety, such as self-verification and hallucination.

In the domain of music generation, prevailing methods focus on text-to-music tasks, predominantly relying on diffusion models. However, they fail to achieve good vocal quality in synthetic music compositions.To tackle this critical challenge, we present Ainur, a hierarchical diffusion model that concentrates on the lyrics-to-music generation task. Through its use of multimodal Lyrics-Audio Spectrogram Pre-training (CLASP) embeddings, Ainur distinguishes itself from past approaches by specifically enhancing the vocal quality of synthetically produced music. Notably, Ainur’s training and testing processes are highly efficient, requiring only a single GPU. According to experimental results, Ainur meets or exceeds the quality of other state-of-the-art models like MusicGen, MusicLM, and AudioLDM2 in both objective and subjective evaluations. Additionally, Ainur offers near real-time inference speed, which facilitate its use in practical, real-world applications.

Abstract We present a visual computing framework for analysing moral rhetoric on social media around controversial topics. Using Moral Foundation Theory, we propose a methodology for deconstructing and visualizing the when, where and who behind each of these moral dimensions as expressed in microblog data. We characterize the design of this framework, developed in collaboration with experts from language processing, communications and causal inference. Our approach integrates microblog data with multiple sources of geospatial and temporal data, and leverages unsupervised machine learning (generalized additive models) to support collaborative hypothesis discovery and testing. We implement this approach in a system named MOTIV. We illustrate this approach on two problems, one related to Stay‐at‐home policies during the COVID‐19 pandemic, and the other related to the Black Lives Matter movement. Through detailed case studies and discussions with collaborators, we identify several insights discovered regarding the different drivers of moral sentiment in social media. Our results indicate that this visual approach supports rapid, collaborative hypothesis testing, and can help give insights into the underlying moral values behind controversial political issues. Supplemental Material: https://osf.io/ygkzn/?view_only=6310c0886938415391d977b8aae8b749

Type 2 diabetes (T2D) is a major public health problem, and opportunistic screening to detect T2D at an early stage can help initiate interventions that delay or prevent the disease and its complications. In this study, we use electronic health records (EHR) and concepts extracted from clinical notes to predict future T2D risk. Our deep neural network-based model captures the temporal sequence of patient visits. We use explainable AI algorithms to assess the model decisions and observe alignment with the domain knowledge of clinical experts.

Natural Language Interfaces that facilitate data exploration tasks are rapidly gaining in interest in the research community because they enable users to focus their attention on the task of inquiry rather than the mechanics of chart construction. Yet, current systems rely solely on processing the user’s explicit commands to generate the user’s intended chart. These commands can be ambiguous due to natural language tendencies such as speech disfluency and underspecification. In this paper, we developed and studied how an always listening interface can help contextualize imprecise queries. Our study revealed that an always listening interface is able to use an on-going conversation to fill in missing properties for imprecise commands, disambiguate inaccurate commands without asking the user for clarification, as well as generate charts without being explicitly asked.

This paper proposes a neural network-based user simulator that can provide a multimodal interactive environment for training Reinforcement Learning (RL) agents in collaborative tasks involving multiple modes of communication. The simulator is trained on the existing ELDERLY-AT-HOME corpus and accommodates multiple modalities such as language, pointing gestures, and haptic-ostensive actions. The paper also presents a novel multimodal data augmentation approach, which addresses the challenge of using a limited dataset due to the expensive and time-consuming nature of collecting human demonstrations. Overall, the study highlights the potential for using RL and multimodal user simulators in developing and improving domestic assistive robots.

Abari Bhattacharya, Abhinav Kumar, Barbara Di Eugenio, Roderick Tabalba, Jillian Aurisano, Veronica Grosso, Andrew Johnson, Jason Leigh, Moira Zellner. Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue. 2023.

The process by which sections in a document are demarcated and labeled is known as section identification. Such sections are helpful to the reader when searching for information and contextualizing specific topics. The goal of this work is to segment the sections of clinical medical domain documentation. The primary contribution of this work is MedSecId, a publicly available set of 2,002 fully annotated medical notes from the MIMIC-III. We include several baselines, source code, a pretrained model and analysis of the data showing a relationship between medical concepts across sections using principal component analysis.

Health coaching helps patients identify and accomplish lifestyle-related goals, effectively improving the control of chronic diseases and mitigating mental health conditions. However, health coaching is cost-prohibitive due to its highly personalized and labor-intensive nature. In this paper, we propose to build a dialogue system that converses with the patients, helps them create and accomplish specific goals, and can address their emotions with empathy. However, building such a system is challenging since real-world health coaching datasets are limited and empathy is subtle. Thus, we propose a modularized health coaching dialogue with simplified NLU and NLG frameworks combined with mechanism-conditioned empathetic response generation. Through automatic and human evaluation, we show that our system generates more empathetic, fluent, and coherent responses and outperforms the state-of-the-art in NLU tasks while requiring less annotation. We view our approach as a key step towards building automated and more accessible health coaching systems.

With the onset of the COVID-19 pandemic, a number of public policy measures have been developed to curb the spread of the virus. However, little is known about the attitudes towards stay-at-home orders expressed on social media despite the fact that social media are central platforms for expressing and debating personal attitudes. To address this gap, we analyze the prevalence and framing of attitudes towards stay-at-home policies, as expressed on Twitter in the early months of the pandemic. We focus on three aspects of tweets: whether they contain an attitude towards stay-at-home measures, whether the attitude was for or against, and the moral justification for the attitude, if any. We collect and annotate a dataset of stay-at-home tweets and create classifiers that enable large-scale analysis of the relationship between moral frames and stay-at-home attitudes and their temporal evolution. Our findings suggest that frames of care are correlated with a supportive stance, whereas freedom and oppression signify an attitude against stay-at-home directives. There was widespread support for stay-at-home orders in the early weeks of lockdowns, followed by increased resistance toward the end of May and the beginning of June 2020. The resistance was associated with moral judgment that mapped to political divisions.

Human-human collaboration is characterized by a back-and-forth, where an action of one agent elicits the response of the other. This interaction is inherently multimodal and includes both high-level modalities such as language and low-level ones such as force exchanges. In this work, we investigate human collaborative manipulation: we show distinct patterns that can be identified in low-level physical data and that can be interpreted as primitives used by humans to negotiate about various aspects of the motion and to execute the motion. These primitives provide a high-level interpretation of the interaction and can be used to connect low-level behavior to language. We describe the human study used to collect the data, the data analysis process, and discuss how the identified primitives could be used by a robot’s interaction manager to mediate physical Human-Robot Interaction (pHRI).

Regular physical activity is associated with a reduced risk of chronic diseases such as type 2 diabetes and improved mental well-being.Yet, more than half of the US population is insufficiently active.Health coaching has been successful in promoting healthy behaviors.In this paper, we present our work towards assisting health coaches by extracting the physical activity goal the user and coach negotiate via text messages.We show that information captured by dialogue acts can help to improve the goal extraction results.We employ both traditional and transformer-based machine learning models for dialogue acts prediction and find them statistically indistinguishable in performance on our health coaching dataset.Moreover, we discuss the feedback provided by the health coaches when evaluating the correctness of the extracted goal summaries.This work is a step towards building a virtual assistant health coach to promote a healthy lifestyle.

Visual Question Answering (VQA) has been widely explored as a computer vision problem, however enhancing VQA systems with linguistic information is necessary for tackling the complexity of the task. The language understanding part can play a major role especially for questions asking about events or actions expressed via verbs. We hypothesize that if the question focuses on events described by verbs, then the model should be aware of or trained with verb semantics, as expressed via semantic role labels, argument types, and/or frame elements. Unfortunately, no VQA dataset exists that includes verb semantic information. We created a new VQA dataset annotated with verb semantic information called imSituVQA. imSituVQA is built by taking advantage of the imSitu dataset annotations. The imSitu dataset consists of images manually labeled with semantic frame elements, mostly taken from FrameNet.

Our goal is to develop an intelligent assistant to support users explore data via visualizations. We have collected a new corpus of conversations, CHICAGO-CRIME-VIS, geared towards supporting data visualization exploration, and we have annotated it for a variety of features, including contextualized dialogue acts. In this paper, we describe our strategies and their evaluation for dialogue act classification. We highlight how thinking aloud affects interpretation of dialogue acts in our setting and how to best capture that information. A key component of our strategy is data augmentation as applied to the training data, since our corpus is inherently small. We ran experiments with the Balanced Bagging Classifier (BAGC), Condiontal Random Field (CRF), and several Long Short Term Memory (LSTM) networks, and found that all of them improved compared to the baseline (e.g., without the data augmentation pipeline). CRF outperformed the other classification algorithms, with the LSTM networks showing modest improvement, even after obtaining a performance boost from domain-trained word embeddings. This result is of note because training a CRF is far less resource-intensive than training deep learning models, hence given a similar if not better performance, traditional methods may still be preferable in order to lower resource consumption.

In a collaborative task and the interaction that accompanies it, the participants often take on distinct roles, and dynamically switch the roles as the task requires. A domestic assistive robot thus needs to have similar capabilities. Using our previously proposed Multimodal Interaction Manager (MIM) framework, this paper investigates how role switching for a robot can be implemented. It identifies a set of primitive subtasks that encode common interaction patterns observed in our data corpus and that can be used to easily construct complex task models. It also describes an implementation on the NAO robot that, together with our original work, demonstrates that the robot can take on different roles. We provide a detailed analysis of the performance of the system and discuss the challenges that arise when switching roles in human-robot interactions.

Itika Gupta, Barbara Di Eugenio, Brian Ziebart, Aiswarya Baiju, Bing Liu, Ben Gerber, Lisa Sharp, Nadia Nabulsi, Mary Smart. Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2020.

Abstract This paper describes results from an observational, exploratory study of visual data exploration in a large, multi‐view, flexible canvas environment. Participants were provided with a set of data exploration sub‐tasks associated with a local crime dataset and were instructed to pose questions to a remote mediator who would respond by generating and organizing visualizations on the large display. We observed that participants frequently posed requests to cast a net around one or several subsets of the data or a set of data attributes. They accomplished this directly and by utilizing existing views in unique ways, including by requesting to copy and pivot a group of views collectively and posing a set of parallel requests on target views expressed in one command. These observed actions depart from multi‐view flexible canvas environments that typically provide interfaces in support of generating one view at a time or actions that operate on one view at a time. We describe how participants used these ‘cast‐a‐net’ requests for tasks that spanned more than one view and describe design considerations for multi‐view environments that would support the observed multi‐view generation actions.

In this paper, we will discuss our framework for summarizing goals discussed during health coaching dialogues. This can help coaches to recall patients' goals without reading the conversations. We build two supervised classification models, one for extracting the slot-values (goal attributes) and another to model the dialogue flow (stages-phases) of the conversation. Using these two models and heuristics, we build our goal extraction pipeline.

With rapid advances in social robotics, humanoids and autonomy, robot assistants appear to be within reach. However, robots are still unable to effectively interact with humans in activities of daily living. One of the challenges is in the frequent use of multiple communication modalities when humans engage in collaborative activities. In this paper, we propose a Multimodal Interaction Manager, a framework for an assistive robot that maintains an active multimodal interaction with a human partner while performing physical collaborative tasks. The heart of our framework is a Hierarchical Bipartite Action-Transition Network (HBATN), which allows the robot to infer the state of the task and the dialogue given spoken utterances and observed pointing gestures from a human partner, and to plan its next actions. Finally, we implemented this framework on a robot to provide preliminary evidence that the robot can successfully participate in a task-oriented multimodal interaction.

Sabita Acharya, Barbara Di Eugenio, Andrew Boyd, Richard Cameron, Karen Dunn Lopez, Pamela Martyn-Nemeth, Debaleena Chattopadhyay, Pantea Habibi, Carolyn Dickens, Haleh Vatani, Amer Ardati. Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue. 2019.

Pair programming is a practice where two coders work side by side at one computer. The practice has been linked to many benefits including increased student engagement, satisfaction, and course grades. We present a quantitative study comparing the fine-grained interactions of individual programmers versus pair programmers as they work to solve coding problems using an Intelligent Tutoring System. We collected data from over 115 students resulting in more than 53,000 log events. We discovered that while both individual and pair programmers had equivalent learning gains, pair programmers took significantly less time on most problems, consulted fewer examples, coded more efficiently, and showed more signs of engagement. Individuals adapted to problems requiring new and compounded concepts at a rate similar to pair programmers.

Our system generates summaries of hospital stays by combining information from two heterogenous sources: physician discharge notes and nursing plans of care.It extracts medical concepts from both sources; concepts that are identified as "complex" by our metric are explained by providing definitions obtained from three external knowledge sources.Finally, relevant concepts (with or without definition) are realized by SimpleNLG.

Analogies and worked out examples are effective means of instruction in a wide variety of learning environments. However, the extent of their effectiveness in Computer Science (CS) education has not been fully explored. We extended our intelligent tutoring system (ITS) for CS data structures, ChiQat-Tutor, to incorporate worked out examples and analogy as teaching strategies. We compare three versions of the system: one that uses standard worked out examples, one that uses analogical worked out examples, and one that uses a pure analogical explanation with separate worked out examples. A study with 66 students showed that students using the standard worked out examples had greater learning gains than students in both analogy conditions. We also found that analogy can be less effective for students with higher prior knowledge. Additionally, we show that some interaction patterns highly correlate with student gains. Overall, the system implementation and results represent a step towards exploring the use of well-established instructional strategies in a computer science ITS.

Abhinav Kumar, Jillian Aurisano, Barbara Di Eugenio, Andrew Johnson, Alberto Gonzalez, Jason Leigh. Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2016.

Computer Science is a difficult subject with many fundamentals to be taught, usually involving a steep learning curve for many students. It is some of these initial challenges that can turn students away from computer science. We have been developing a new Intelligent Tutoring System, ChiQat-Tutor, that focuses on tutoring of Computer Science fundamentals. Here, we outline the system under development, while bringing particular attention to its architecture and how it attains the primary goals of being easily extensible and providing a low barrier of entry to the end user. The system is broadly broken down into lessons, teaching strategies, and utilities, which work together to promote seamless integration of components. We also cover currently developed components in the form of a case study, as well as detailing our experience of deploying it to an undergraduate Computer Science classroom, leading to learning gains on par with prior work.

Barbara Di Eugenio, Andrew Boyd, Camillo Lugaresi, Abhinaya Balasubramanian, Gail Keenan, Mike Burton, Tamara Goncalves Rezende Macieira, Jianrong Li, Yves Lussier, Yves Lussier. Proceedings of the 8th International Natural Language Generation Conference (INLG). 2014.

In this paper we describe our progress towards understanding human communication through physical interaction. We describe a classification algorithm that can recognize four classes of actions that frequently occur during collaborative manipulation of planar objects. These actions were selected based on a user study involving dyads of elderly and care-giver in a realistic setting. Further user studies were conducted to collect the data necessary to develop the classification algorithm. As part of the data collection we also developed a sensory glove. The classification algorithm gives insight into human collaborative manipulation. More precisely, it identifies features in the data that are significant for classification. This information is particularly interesting as it only relies on physical aspects of the interaction and not on any particular sensor. As a result, the described work does not depend on any particular hardware and can be directly used by other researchers in human-robot interaction to develop further experiments and studies.

In recent years, the availability of too much information has become a fact of life for anybody connected with the Internet. The same is true for music: because of the penetration of portable devices and the availability of millions of tracks on the web, individual music collections have become unwieldy. Users need tools to help search their own song collections, and to recommend songs they may be interested in. Whereas recommendation systems have been developed for a variety of products, a music recommendation system presents special challenges, including the ability to recommend individual songs, as opposed to entire albums, even if only full album reviews are available on-line. SongRecommend, our music recommendation system, combines information extraction and generation techniques to produce summaries of reviews of individual songs from album reviews. We present a number of evaluations for SongRecommend: intrinsic evaluations of the extraction components, and of the informativeness of the summaries; and a user study of the impact of the song review summaries on users’ decision-making processes. When presented with the summary, users were able to make quicker decisions, and their choices were more varied. Whereas the smaller size of the summary has an impact on time-on-task, users do not appear to choose a specific recommendation only based on number of words. Our work demonstrates that state-of-the-art techniques in Natural Language Processing can be integrated into an effective end-to-end system.

In this paper, we analyze data gathered from students’ interactions with iList, an intelligent tutoring system that teaches linked lists to computer science (CS) undergraduates. A number of features have been extracted from the log files which were used to; a) build predictive models of students’ performance, b) analyze temporal aspects of students’ problem solving behavior. Our results suggest that it is possible to build predictive models of performance with an accuracy of 87% by using logistic regression. The results also show that it is more likely a student will perform a step correctly if s/he spends more time on it.

We developed a new model for iList, our system that helps students learn linked list. The model is automatically extracted from past student data, and allows iList to track students' problem-solving behavior in order to provide targeted feedback. We evaluated the new model both intrinsically and extrinsically. We show that the model can match most student actions after a relatively small sequence of observations, and that iList can effectively use the new student tracker to provide feedback and help students learn.

The aim of the project we discuss in this paper is to develop a computational model of peer learning. We present an extensive analysis of peer learning dialogues, analysis on which our computational model is based. Our model incorporates shifts of initiative as an identifier of knowledge co-construction. We have embedded this model in a peer-learning agent that collaborates with students to solve problems in the domain of computer science data structures.

To improve the interaction between students and an intelligent tutoring system, we developed two Natural Language generators, that we systematically evaluated in a three way comparison that included the original system as well.We found that the generator which intuitively produces the best language does engender the most learning.Specifically, it appears that functional aggregation is responsible for the improvement.

We discuss Feature Latent Semantic Analysis (FLSA), an extension to Latent Semantic Analysis (LSA). LSA is a statistical method that is ordinarily trained on words only; FLSA adds to LSA the richness of the many other linguistic features that a corpus may be labeled with. We applied FLSA to dialogue act classification with excellent results. We report results on three corpora: CallHome Spanish, MapTask, and our own corpus of tutoring dialogues.

In recent years, the kappa coefficient of agreement has become the de facto standard for evaluating intercoder agreement for tagging tasks. In this squib, we highlight issues that affect κ and that the community has largely neglected. First, we discuss the assumptions underlying different computations of the expected agreement component of κ. Second, we discuss how prevalence and bias affect the κ measure.

Centering theory is the best-known framework for theorizing about local coherence and salience; however, its claims are articulated in terms of notions which are only partially specified, such as “utterance,” “realization,” or “ranking.” A great deal of research has attempted to arrive at more detailed specifications of these parameters of the theory; as a result, the claims of centering can be instantiated in many different ways. We investigated in a systematic fashion the effect on the theory's claims of these different ways of setting the parameters. Doing this required, first of all, clarifying what the theory's claims are (one of our conclusions being that what has become known as “Constraint 1” is actually a central claim of the theory). Secondly, we had to clearly identify these parametric aspects: For example, we argue that the notion of “pronoun” used in Rule 1 should be considered a parameter. Thirdly, we had to find appropriate methods for evaluating these claims. We found that while the theory's main claim about salience and pronominalization, Rule 1—a preference for pronominalizing the backward-looking center (CB)—is verified with most instantiations, Constraint 1–a claim about (entity) coherence and CB uniqueness—is much more instantiation-dependent: It is not verified if the parameters are instantiated according to very mainstream views (“vanilla instantiation”), it holds only if indirect realization is allowed, and is violated by between 20% and 25% of utterances in our corpus even with the most favorable instantiations. We also found a trade-off between Rule 1, on the one hand, and Constraint 1 and Rule 2, on the other: Setting the parameters to minimize the violations of local coherence leads to increased violations of salience, and vice versa. Our results suggest that “entity” coherence—continuous reference to the same entities—must be supplemented at least by an account of relational coherence.

The focus of this paper is on an action representation formalism that encodes bothlinguistic andplanning knowledge about actions, and that supports the interpretation of complex Natural Language instructions and, in particular, of instructions containing Purpose Clauses. The representation uses linguistically motivated primitives, derived from Jackendoff's work on Conceptual Semantics, and is embedded in the description logic based system CLASSIC. I first motivate the characteristics of the formalism as needed to understand Natural Language instructions. I then describe the formalism itself, and I argue that the integration of a linguistically motivated lexical semantics formalism and of a description logic based system is beneficial to both. Finally, I show how the formalism is exploited by the algorithm that interprets Purpose Clauses. The output of the algorithm is used in theAnimation from NL project, that has as its goal the automatic creation of animated task simulations.

Our goal is to identify the features that predict the occurrence and placement of discourse cues in tutorial explanations in order to aid in the automatic generation of explanations. Previous attempts to devise rules for text generation were based on intuition or small numbers of constructed examples. We apply a machine learning program, C4.5, to induce decision trees for cue occurrence and placement from a corpus of data coded for a variety of features previously thought to affect cue usage. Our experiments enable us to identify the features with most predictive power, and show that machine learning can be used to induce decision trees useful for text generation.

Building text planning resources by hand is timeconsuming and difficult.Certainly, a number of planning architectures and their accompanying plan libraries have been implemented, but while the architectures themselves may be reused in a new domain, the library of plans typically cannot.One way to address this problem is to use machine learning techniques to automate the derivation of planning resources for new domains.In this paper, we apply this technique to build microplanning rules for preventative expressions in instructional text.

In this paper we will present our ongoing work on a plan-based discourse processor developed in the context of the Enthusiast Spanish to English translation system as part of the JANUS multi-lingual speech-to-speech translation system. We will demonstrate that theories of discourse which postulate a strict tree structure of discourse on either the intentional or attentional level are not totally adequate for handling spontaneous dialogues. We will present our extension to this approach along with its implementation in our plan-based discourse processor. We will demonstrate that the implementation of our approach outperforms an implementation based on the strict tree structure approach.

Based on an ongoing attempt to integrate Natural Language instructions with human figure animation, we demonstrate that agents' understanding and use of instructions can complement what they can derive from the environment in which they act. We focus on two attitudes that contribute to agents' behavior—their intentions and their expectations—and shown how Natural Language instructions contribute to such attitudes in ways that complement the environment. We also show that instructions can require more than one context of interpretation and thus that agents' understanding of instructions can evolve as their activity progresses. A significant consequence is that Natural Language understanding in the context of behavior cannot simply be treated as “front end” processing, but rather must be integrated more deeply into the processes that guide an agent's behavior and respond to its perceptions.

This paper presents an analysis of purpose clauses in the context of instruction understanding. Such analysis shows that goals affect the interpretation and / or execution of actions, lends support to the proposal of using generation and enablement to model relations between actions, and sheds light on some inference processes necessary to interpret purpose clauses.

No abstract available.