Explanations for autonomous vehicle (AV) decisions may build trust, however, explanations can contain errors. In a simulated driving study (n = 232), we tested how AV explanation errors, driving context characteristics (perceived harm and driving difficulty), and personal traits (prior trust and expertise) affected a passenger's comfort in relying on an AV, preference for control, confidence in the AV's ability, and explanation satisfaction. Errors negatively affected all outcomes. Surprisingly, despite identical driving, explanation errors reduced ratings of the AV's driving ability. Severity and potential harm amplified the negative impact of errors. Contextual harm and driving difficulty directly impacted outcome ratings and influenced the relationship between errors and outcomes. Prior trust and expertise were positively associated with outcome ratings. Results emphasize the need for accurate, contextually adaptive, and personalized AV explanations to foster trust, reliance, satisfaction, and confidence. We conclude with design, research, and deployment recommendations for trustworthy AV explanation systems.

The interdisciplinary field of Human-Computer Interaction (HCI) thrives on productive engagement with different domains, yet this engagement often breaks due to idiosyncratic writing styles and unfamiliar concepts. Inspired by the dialogic model of abstract metaphors, as well as the potential of Large Language Models (LLMs) to produce on-demand support, we investigate the use of metaphors to facilitate engagement between Science and Technology Studies (STS) and System HCI. Our reflective-style survey with early-career HCI researchers (N=48) reported that limited prior exposure to STS research can hinder perceived openness of the work, and ultimately interest in reading. The survey also revealed that metaphors enhance likelihood to continue reading STS papers, and alternative perspectives can build critical thinking skills to mitigate potential risks of LLM-generated metaphors. We lastly offer a specified model of metaphor exchange (within this generative context) that incorporates alternative perspectives to construct shared understanding in interdisciplinary engagement.

OBJECTIVE: Clinician implicit bias can impede patient-centered communication, leading to health care inequities. While the field of implicit bias education is evolving with advances in technology, clinicians' perspectives remain underexplored. This study investigated clinicians' perceptions of educational strategies to complement communication feedback technology in the implementation of an implicit bias education intervention. MATERIALS AND METHODS: We recruited primary care practitioners in remote interviews to brainstorm future technologies for improving clinician awareness of implicit bias in patient-provider communication. Participants completed an online survey in which they rated the priority of educational strategies that could complement the technology. We performed inductive-deductive thematic analysis of the interview data with Implicit Bias Recognition and Management (IBRM) domains as a priori codes and used descriptive statistics to summarize the survey data. RESULTS: Participants (n = 16) proposed how future technology could improve clinician awareness, such as recording visits to help clinicians be more self-aware of their communication; however, some providers expressed concerns regarding feedback fatigue and the potential impact of technology on reducing time spent with patients. Participants recommended incorporating feedback regularly into training, identifying organizational incentives, and debriefing with trusted colleagues and communication experts. DISCUSSION: Participants brainstormed technologies and identified educational strategies, such as discussion with a facilitator, that could promote clinician receptivity to feedback and inform IBRM approaches for clinical ambient intelligence. Yet, challenges remain to incentivizing participation for practicing clinicians, and Continuing Medical Education may be one effective approach. CONCLUSION: The proposed technologies and prioritized educational strategies have the potential to promote health equity by helping clinicians develop skills to manage implicit bias. In the future, these findings could inform IBRM interventions that leverage clinical ambient intelligence.

Improving accuracy, time spent, and ubiquity of delineation has been a long-standing design aim, yet many HCI works have overlooked high-stakes and complex healthcare annotation. We explore contouring, a critical workflow aimed at identifying and segmenting tumors, usually performed on immobile desktop computers in clinics, in which limited support for mobile access leads to prolonged and subpar treatment planning. Following interviews and think-aloud studies (N = 10 physicians), we report key contouring behaviors, and later design a novel cross-device prototype that enables contouring on everyday touch devices. We compared contouring via desktop and touch in a lab study (N = 8 residents) and found that mobile phones not only yielded similar accuracy, but also took significantly less time. Our results point to three broad design guidelines for cross-device solutions deployed within standalone healthcare workflows, and highlight how incorporating different device and input modalities can improve treatment delivery in today’s distributed healthcare environments.

Healthcare providers' implicit bias, based on patients' physical characteristics and perceived identities, negatively impacts healthcare access, care quality, and outcomes. Feedback tools are needed to help providers identify and learn from their biases. To incorporate providers' perspectives on the most effective ways to present such feedback, we conducted semi-structured design critique sessions with 24 primary care providers. We found that providers seek feedback designed with transparent metrics indicating the quality of their communication with a patient and trends in communication patterns across visits. Based on these metrics and trends, providers want this feedback presented in a dashboard paired with actionable, personalized tips about how to improve their communication behaviors. Our study provides new insights for interactive systems to help mitigate the impact of implicit biases in patient-provider communication. New systems that build upon these insights could support providers in making healthcare more equitable, particularly for patients from marginalized communities.

Patient-provider communication influences patient health outcomes, and analyzing such communication could help providers identify opportunities for improvement, leading to better care. Interpersonal communication can be assessed through "social-signals" expressed in non-verbal, vocal behaviors like interruptions, turn-taking, and pitch. To automate this assessment, we introduce a machine-learning pipeline that ingests audio-streams of conversations and tracks the magnitude of four social-signals: dominance, interactivity, engagement, and warmth. This pipeline is embedded into ConverSense, a web-application for providers to visualize their communication patterns, both within and across visits. Our user study with 5 clinicians and 10 patient visits demonstrates ConverSense's potential to provide feedback on communication challenges, as well as the need for this feedback to be contextualized within the specific underlying visit and patient interaction. Through this novel approach that uses data-driven self-reflection, ConverSense can help providers improve their communication with patients to deliver improved quality of care.

Apprenticeship is the predominant method for transferring specialized medical skills, yet the inter-dynamics between faculty and residents, including methods of feedback exchange are under-explored. We specifically investigate contouring: outlining tumors in preparation for radiotherapy, a critical skill that when performed subpar, severely degrades patient survival. Interviews and design-thinking workshops (N = four faculty; six residents) revealed misalignment between teaching methods and residents who desired timely, relevant, and diverse feedback. We further discuss reasons: overlapping learning content and strategies to ease tensions between clinical and teaching duties, and lack of support for exchange of cognitive processes. The follow-up survey study (N = 67 practitioners from 31 countries), which contained annotation and sketching tasks, provided diverse perspective over effective feedback elements. We lastly present sociotechnical implications in supporting faculty’s teaching duties and learners’ cognitive models, such as systematically leveraging senior learners in providing case-based guidance and supporting double-sided flow of cognitive information via in-situ video snippets.

Providing asynchronous feedback is a critical step in the 3D design workflow. A common approach to providing feedback is to pair textual comments with companion reference images, which helps illustrate the gist of text. Ideally, feedback providers should possess 3D and image editing skills to create reference images that can effectively describe what they have in mind. However, they often lack such skills, so they have to resort to sketches or online images that might not match well with the current 3D design. To address this, we introduce MemoVis , a text editor interface that assists feedback providers in creating reference images with generative AI driven by the feedback comments. First, a novel real-time viewpoint suggestion feature, based on a vision-language foundation model, helps feedback providers anchor a comment with a camera viewpoint. Second, given a camera viewpoint, we introduce three types of image modifiers based on pre-trained 2D generative models to turn a text comment into an updated version of the 3D scene from that viewpoint. We conducted a within-subjects study with \(14\) feedback providers, demonstrating the effectiveness of MemoVis. The quality and explicitness of the companion images were evaluated by another eight participants with prior 3D design experience.

While voice user interfaces offer increased accessibility due to hands-free and eyes-free interactions, older adults often have challenges such as constructing structured requests and perceiving how such devices operate. Voice-first user interfaces have the potential to address these challenges by enabling multimodal interactions. Standalone voice + touchscreen Voice Assistants (VAs), such as Echo Show, are specific types of devices that adopt such interfaces and are gaining popularity. However, the affordances of the additional touchscreen for older adults are unknown. Through a 40-day real-world deployment with older adults living independently, we present a within-subjects study (N = 16; age M = 82.5, SD = 7.77, min. = 70, max. = 97) to understand how a built-in touchscreen might benefit older adults during device setup, conducting self-report diary survey, and general uses. We found that while participants appreciated the visual outputs, they still preferred to respond via speech instead of touch. We identified six design implications that can inform future innovations of senior-friendly VAs for managing healthcare and improving quality of life.

Acquiring accessibility information about unfamiliar places in advance is essential for wheelchair users to make better decisions about physical visits. Today’s assessment approaches such as phone calls, photos/videos, or 360° virtual tours often fall short of providing the specific accessibility details needed for individual differences. For example, they may not reveal crucial information like whether the legroom underneath a table is spacious enough or if the spatial configuration of an appliance is convenient for wheelchair users. In response, we present Embodied Exploration, a Virtual Reality (VR) technique to deliver the experience of a physical visit while keeping the convenience of remote assessment. Embodied Exploration allows wheelchair users to explore high-fidelity digital replicas of physical environments with themselves embodied by avatars, leveraging the increasingly affordable VR headsets. With a preliminary exploratory study, we investigated the needs and iteratively refined our techniques. Through a real-world user study with six wheelchair users, we found Embodied Exploration is able to facilitate remote and accurate accessibility assessment. We also discuss design implications for embodiment, safety, and practicality.

Collaborative Mixed Reality (MR) systems that help extend expertise for physical tasks to remote environments often situate experts in an immersive view of the task environment to bring the collaboration closer to collocated settings. In this paper, we design UnMapped, an alternative interface for remote experts that combines a live 3D view of the active space within the novice’s environment with a static 3D recreation of the expert’s own workspace to leverage their existing spatial memories within it. We evaluate the impact of this approach on single and repeated use of collaborative MR systems for remote guidance through a comparative study. Our results indicate that despite having a limited understanding of the novice’s environment, using an UnMapped interface increased performance and communication efficiency while reducing experts’ task load. We also outline the various affordances of providing remote experts with a familiar and spatially-stable environment to assist novices.

While paper instructions are a mainstream medium for sharing knowledge, consuming such instructions and translating them into activities can be inefficient due to the lack of connectivity with the physical environment. We propose PaperToPlace, a novel workflow comprising an authoring pipeline, which allows the authors to rapidly transform and spatialize existing paper instructions into an MR experience, and a consumption pipeline, which computationally places each instruction step at an optimal location that is easy to read and does not occlude key interaction areas. Our evaluation of the authoring pipeline with 12 participants demonstrates the usability of our workflow and the effectiveness of using a machine learning based approach to help extract the spatial locations associated with each step. A second within-subjects study with another 12 participants demonstrates the merits of our consumption pipeline to reduce context-switching effort by delivering individual segmented instruction steps and offering hands-free affordances.

Population aging is an increasingly important consideration for health care in the 21th century, and continuing to have access and interact with digital health information is a key challenge for aging populations. Voice-based Intelligent Virtual Assistants (IVAs) are promising to improve the Quality of Life (QoL) of older adults, and coupled with Ecological Momentary Assessments (EMA) they can be effective to collect important health information from older adults, especially when it comes to repeated time-based events. However, this same EMA data is hard to access for the older adult: although the newest IVAs are equipped with a display, the effectiveness of visualizing time-series based EMA data on standalone IVAs has not been explored. To investigate the potential opportunities for visualizing time-series based EMA data on standalone IVAs, we designed a prototype system, where older adults are able to query and examine the time-series EMA data on Amazon Echo Show - a widely used commercially available standalone screen-based IVA. We conducted a preliminary semi-structured interview with a geriatrician and an older adult, and identified three findings that should be carefully considered when designing such visualizations.

Contouring is an indispensable step in Radiotherapy (RT) treatment planning. However, today’s contouring software is constrained to only work with a 2D display, which is less intuitive and requires high task loads. Virtual Reality (VR) has shown great potential in various specialties of healthcare and health sciences education due to the unique advantages of intuitive and natural interactions in immersive spaces. VR-based radiation oncology integration has also been advocated as a target healthcare application, allowing providers to directly interact with 3D medical structures. We present VRContour and investigate how to effectively bring contouring for radiation oncology into VR. Through an autobiographical iterative design, we defined three design spaces focused on contouring in VR with the support of a tracked tablet and VR stylus, and investigating dimensionality for information consumption and input (either 2D or 2D + 3D). Through a within-subject study (n = 8), we found that visualizations of 3D medical structures significantly increase precision, and reduce mental load, frustration, as well as overall contouring effort. Participants also agreed with the benefits of using such metaphors for learning purposes.

Accurately drawing non-planar 3D curves in immersive Virtual Reality (VR) is indispensable for many precise 3D tasks. However, due to lack of physical support, limited depth perception, and the non-planar nature of 3D curves, it is challenging to adjust mid-air strokes to achieve high precision. Instead of creating new interaction techniques, we investigated how task geometric shapes and input modalities affect precision-first drawing performance in a within-subject study (n=12) focusing on 3D target tracing in commercially available VR headsets. We found that compared to using bare hands, VR controllers and pens yield nearly 30% of precision gain, and that the tasks with large curvature, forward-backward or left-right orientations perform best. We finally discuss opportunities for designing novel interaction techniques for precise 3D drawing. We believe that our work will benefit future research aiming to create usable toolboxes for precise 3D drawing.

OBJECTIVE: People who experience marginalization, including Black, Indigenous, People of Color (BIPOC) and Lesbian, Gay, Bisexual, Transgender, Queer, Plus (ie, all other marginalized genders and sexual orientations) people (LGBTQ+) experience discrimination during healthcare interactions, which negatively impacts patient-provider communication and care. Yet, scarce research examines the lived experience of unfair treatment among patients from marginalized groups to guide patient-centered tools that improve healthcare equity. MATERIALS AND METHODS: We interviewed 25 BIPOC and/or LGBTQ+ people about their experiences of unfair treatment and discrimination when visiting healthcare providers. Through thematic analysis, we describe participants' immediate reactions and longer-term consequences of those experiences. RESULTS: We identified 4 ways that participants reacted to discrimination in the moment: Fighting, Fleeing, Excusing, and Working Around Bias. Long-term consequences reflect 6 ways they coped: Delaying or Avoiding Care, Changing Healthcare Providers, Self-prescribing, Covering Behaviors, Experiencing Health Complications, and Mistrusting Healthcare Institutions. DISCUSSION: By describing how patients react to experiences of unfair treatment and discrimination, our findings enhance the understanding of health disparities as patients cope and struggle to speak out.To combat these problems, we identify 3 future directions for informatics interventions that improve provider behavior, support patient advocacy, and address power dynamics in healthcare. CONCLUSIONS: BIPOC and LGBTQ+ patients' perspectives on navigating unfair treatment and discrimination in healthcare offers critical insight into their experiences and long-term consequences of those experiences. Understanding the circumstances and consequences of unfair treatment, discrimination, and the impact of bias through this patient-centered lens is crucial to inform informatics technologies that promote health equity.

Voice-based Intelligent Virtual Assistants (IVAs) promise to improve healthcare management and Quality of Life (QOL) by introducing the paradigm of hands-free and eye-free interactions. However, there has been little understanding regarding the challenges for designing such systems for older adults, especially when it comes to healthcare related tasks. To tackle this, we consider the processes of care delivery and QOL enhancements for older adults as a collaborative task between patients and providers. By interviewing 16 older adults living independently or semi-independently and 5 providers, we identified 12 barriers that older adults might encounter during daily routine and while managing health. We ultimately highlighted key design challenges and opportunities that might be introduced when integrating voice-based IVAs into the life of older adults. Our work will benefit practitioners who study and attempt to create full-fledged IVA-powered smart devices to deliver better care and support an increased QOL for aging populations.

Traumatic injuries require timely intervention, but medical expertise is not always available at the patient’s location. Despite recent advances in telecommunications, surgeons still have limited tools to remotely help inexperienced surgeons. Mixed Reality hints at a future where remote collaborators work side-by-side as if co-located; however, we still do not know how current technology can improve remote surgical collaboration. Through role-playing and iterative-prototyping, we identify collaboration practices used by expert surgeons to aid novice surgeons as well as technical requirements to facilitate these practices. We then introduce ARTEMIS, an AR-VR collaboration system that supports these key practices. Through an observational study with two expert surgeons and five novice surgeons operating on cadavers, we find that ARTEMIS supports remote surgical mentoring of novices through synchronous point, draw, and look affordances and asynchronous video clips. Most participants found that ARTEMIS facilitates collaboration despite existing technology limitations explored in this paper.

Mixed Reality has been shown to enhance remote guidance and is especially well-suited for physical tasks. Conversations during these tasks are heavily anchored around task objects and their spatial relationships in the real world, making referencing - the ability to refer to an object in a way that is understood by others - a crucial process that warrants explicit support in collaborative Mixed Reality systems. This paper presents a 2x2 mixed factorial experiment that explores the effects of providing spatial information and system-generated guidance to task objects. It also investigates the effects of such guidance on the remote collaborator’s need for spatial information. Our results show that guidance increases performance and communication efficiency while reducing the need for spatial information, especially in unfamiliar environments. Our results also demonstrate a reduced need for remote experts to be in immersive environments, making guidance more scalable, and expertise more accessible.

This paper investigates in-class interactions in synchronous online classrooms when the choice of modality is discretionary, such that students choose when and if they turn on their cameras and microphones. Instructor interviews (N = 7) revealed that most students preferred not to share videos and verbally participate. This hindered instructors’ ability to read their classrooms and make deeper connections with students. Survey results (N = 102) suggested that students felt a lacking sense of community in online vs. in-person lectures. Some students felt uncomfortable broadcasting their appearances to everyone in the class, and some were unaware of the benefits for instructors. Most students favored using the text chat to participate. Considering the needs of both instructors and students, we propose recommendations to mitigate the loss of classroom interactions by collecting and presenting less invasive social cues in an aggregated format, and incorporating opportunities for informal exchanges and individual control to spark peer bonding.

Data grouping is among the most frequently used operations in data visualization. It is the process through which relevant information is gathered, simplified, and expressed in summary form. Many popular visualization tools support automatic grouping of data (e.g., dividing up a numerical variable into bins). Although grouping plays a pivotal role in supporting data exploration, further adjustment and customization of auto-generated grouping criteria is non-trivial. Such adjustments are currently performed either programmatically or through menus and dialogues which require specific parameter adjustments over several steps. In response, we introduce Embedded Merge & Split (EMS), a new interaction technique for direct adjustment of data grouping criteria. We demonstrate how the EMS technique can be designed to directly manipulate width and position in bar charts and histograms, as a means for adjustment of data grouping criteria. We also offer a set of design guidelines for supporting EMS. Finally, we present the results of two user studies, providing initial evidence that EMS can significantly reduce interaction time compared to WIMP-based technique and was subjectively preferred by participants.

We introduce a mobile app for collecting in-the-wild data, including sensor measurements and self-reported labels describing people's behavioral context (e.g., driving, eating, in class, shower). Labeled data is necessary for developing context-recognition systems that serve health monitoring, aging care, and more. Acquiring labels without observers is challenging and previous solutions compromised ecological validity, range of behaviors, or amount of data. Our user interface combines past and near-future self-reporting of combinations of relevant context-labels. We deployed the app on the personal smartphones of 60 users and analyzed quantitative data collected in-the-wild and qualitative user-experience reports. The interface's flexibility was important to gain frequent, detailed labels, support diverse behavioral situations, and engage different users: most preferred reporting their past behavior through a daily journal, but some preferred reporting what they're about to do. We integrated insights from this work back into the app, which we make available to researchers for conducting in-the-wild studies.

Thermostats are the primary interface for occupants of office buildings to express their thermal comfort preferences. However, traditional thermostats are often ineffective due to physical inaccessibility, lack of information or limited responsiveness, which lead to occupant discomfort. Modern thermostat designs do overcome some of these limitations, but retrofitting them to existing buildings is prohibitively expensive. Software thermostats based on web or smartphone apps provide an alternate interaction mechanism with minimal deployment cost. However, their usage and effectiveness have not been studied extensively in real settings. We present Genie, a novel software thermostat that we designed and deployed in our university for over 21 months. We compare the use of Genie to traditional thermostats. Our data and user study show that due to the clarity of information and wider thermal control provided by Genie, users feel more comfortable in their offices. Furthermore, the improved comfort did not affect the overall energy consumption or lead to misuse of HVAC controls.

Early instruction plays a crucial role in allowing toddlers to develop social, cognitive, and sensory-motor skills. Free play is important in any early development program, but designing activities for free play is challenging. In this paper, we investigate the use of an interactive surface, BendableSound, a fabric-based interactive surface that enables young children to play piano sounds when touching the fabric, and its potential value in early education classrooms. We conducted a 9-week exploratory study in which 22 toddlers and 5 teachers used BendableSound during free play activities inside their classroom. Our qualitative results indicate that BendableSound was successfully adopted and integrated in toddler classrooms and could positively impact cognitive, social, and physical development. These results offer implications for the design of deformable surfaces and for their integration in activities to support the early development of toddlers.

This is a proposal for organizing a Multimodal Learning Analytics (MLA) data challenge as part of the workshop offering of the Learning Analytics and Knowledge (LAK) conference. It explains the motivation of the event, its objectives, target groups, expected format, organization, dissemination strategy and schedule.

VideoMob is an interactive video platform and an artwork that enables strangers visiting different installation locations to interact across time and space through a computer interface that detects their presence, video-records their actions while automatically removing the video background through computer vision, and co-situates visitors as part of the same digital environment. Through the combination of individual user videos to form a digital crowd, strangers are connected through the graphic display. Our work is inspired by the way distant people can interact with each other through technology and influenced by artists working in the realm of interactive art. We deployed VideoMob in a variety of settings, locations, and contexts to observe hundreds of visitors’ reactions. By analyzing behavioral data collected through depth cameras from our 1,068 recordings across eight venues, we studied how participants behave when given the opportunity to record their own video portrait into the artwork. We report the specific activity performed in front of the camera and the influences that existing crowds impose on new participants. Our analysis informs the integration of a series of possible novel interaction paradigms based on real-time analysis of the visitors’ behavior through specific computer vision and machine learning techniques that have the potential to increase the engagement of the artwork's visitors and to impact user experience.

In this paper, we describe how we developed an information display prototype for trauma resuscitation teams based on design ideas and feedback from clinicians. Our approach is grounded in participatory design, emphasizing the importance of gaining long-term commitment from clinicians in system development. Through a series of participatory design workshops, heuristic evaluation, and simulated resuscitation sessions, we identified the main information features to include on our display. Our results focus on how we balanced the design tensions that emerged when addressing the ad hoc, hierarchical, and multidisciplinary nature of trauma teamwork. We discuss the implications of balancing role-based differences for each information feature, as well as two major design tensions: process-based vs. state-based designs and role-based vs. team-based displays.

Color blindness is a highly prevalent vision impairment that inhibits people's ability to understand colors. Although classified as a mild disability, color blindness has important effects on the daily activity of people, preventing them from performing their tasks in the most natural and effective ways. In order to address this issue we developed Chroma, a wearable augmented-reality system based on Google Glass that allows users to see a filtered image of the current scene in real-time. Chroma automatically adapts the scene-view based on the type of color blindness, and features dedicated algorithms for color saliency. Based on interviews with 23 people with color blindness we implemented four modes to help colorblind individuals distinguish colors they usually can't see. Although Glass still has important limitations, initial tests of Chroma in the lab show that colorblind individuals using Chroma can improve their color recognition in a variety of real-world activities. The deployment of Chroma on a wearable augmented-reality device makes it an effective digital aid with the potential to augment everyday activities, effectively providing access to different color dimensions for colorblind people.

In this paper we describe the development of a novel paper-digital interface for recording movement trajectories, designed to assist ethnographers and ethologists in analysis of social movement. While we focus on development of a system to aid analysis of elephant movement, the resulting interaction techniques and facilities are quite general. The paper highlights how our design evolved to balance the goals of researchers, their current practices, and the challenges of integrating the relatively unconstrained world of pen and paper with the relatively constrained world of digital systems.

Photographs are powerful and pervasive media that facilitate communication and support reminiscence. Adding audio narrations to traditional paper photographs combines the familiarity of paper photos with the nuanced richness of voice. We enable creation of and interaction with audio-enhanced paper photographs through custom software deployed on a digital pen. This paper reports on use of a paper-digital photo album by an older adult (Ethel, age 105), her extended family, and nursing staff over a five-month field deployment. The interactive photo album was found to be easy to use and accessible, effectively engaged Ethel's and family members' interest, and provided a focal point for communication and social interaction. Family and caregivers report improvements in Ethel's social interaction. We discuss the properties of our audio-enhanced paper approach that make it a promising medium for engaging older adults and distill general design considerations for paper-digital photo albums.

Advances in learning analytics are contributing new empirical findings, theories, methods, and metrics for understanding how students learn. It also contributes to improving pedagogical support for students' learning through assessment of new digital tools, teaching strategies, and curricula. Multimodal learning analytics (MMLA)[1] is an extension of learning analytics and emphasizes the analysis of natural rich modalities of communication across a variety of learning contexts. This MMLA Grand Challenge combines expertise from the learning sciences and machine learning in order to highlight the rich opportunities that exist at the intersection of these disciplines. As part of the Grand Challenge, researchers were asked to predict: (1) which student in a group was the dominant domain expert, and (2) which problems that the group worked on would be solved correctly or not. Analyses were based on a combination of speech, digital pen and video data. This paper describes the motivation for the grand challenge, the publicly available data resources and results reported by the challenge participants. The results demonstrate that multimodal prediction of the challenge goals: (1) is surprisingly reliable using rich multimodal data sources, (2) can be accomplished using any of the three modalities explored, and (3) need not be based on content analysis.

This paper provides documentation on dataset resources for establishing a new research area called multimodal learning analytics (MMLA). Research on this topic has the potential to transform the future of educational practice and technology, as well as computational techniques for advancing data analytics. The Math Data Corpus includes high-fidelity time-synchronized multimodal data recordings (speech, digital pen, images) on collaborating groups of students as they work together to solve mathematics problems that vary in difficulty level. The Math Data Corpus resources include initial coding of problem segmentation, problem-solving correctness, and representational content of students' writing. These resources are made available to participants in the data-driven grand challenge for the Second International Workshop on Multimodal Learning Analytics. The primary goal of this event is to analyze coherent signal, activity, and lexical patterns that can identify domain expertise and change in domain expertise early, reliably, and objectively, as well as learning-oriented precursors. An additional aim is to build an international research community in the emerging area of multimodal learning analytics by organizing a series of workshops that bring together multidisciplinary scientists to work on MMLA topics.

Hybrid paper-digital interfaces are a promising approach for supporting language activities. The familiarity of pen and paper makes it a particularly attractive media for many user groups, including young children. Digital pens enhance interaction with traditional paper content by playing and recording audio and recognizing handwriting and gestures. Currently, generating custom interactive paper documents involves some programming, limiting its use by many user groups (e.g., educators and families) who might especially benefit from application of hybrid paper-digital interfaces in their practices. To address this need, we developed an end-user Toolkit for Authoring Pen and Paper Language Activities (TAP & PLAY). This paper describes the iterative development of the toolkit, its accessibility for novice non-technical users, and use in three different contexts for early language learning. We demonstrate and document the system's usability, generality, and utility for people who want to create and tailor their own custom interactive paper-based language activities.

Researchers from many disciplines are taking advantage of increasingly inexpensive digital video to capture extensive records of human activity in real-world settings. The ability to record and share such data has created a critical moment in the practice and scope of behavioral research. While recent work is beginning to develop techniques for visualizing and interacting with integrated multimodal information collected during field research, navigating and analyzing these large datasets remains challenging and tools are especially needed to support the early stages of data exploration.

As part of our research on multimodal analysis and visualization of activity dynamics, we are exploring the integration of data produced by a variety of sensor technologies within ChronoViz, a tool aimed at supporting the simultaneous visualization of multiple streams of time series data. This paper reports on the integration of a mobile eye-tracking system with data streams collected from HD video cameras, microphones, digital pens, and simulation environments. We focus on the challenging environment of the commercial airline flight deck, analyzing the use of mobile eye tracking systems in aviation human factors and reporting on techniques and methods that can be applied in this and other domains in order to successfully collect, analyze and visualize eye-tracking data in combination with the array of data types supported by ChronoViz.

Large-scale display walls, and the high-resolution visualizations they support, promise to become ubiquitous. Natural interaction with them, especially in collaborative environments, is increasingly important and yet remains an on-going challenge. Part of the problem is a resolution mismatch between low-resolution input devices and high-resolution display walls. In addition, enabling concurrent use by multiple users is difficult - for example, how would this large workspace be managed for multiple users and what novel collaborative interactions could occur? In this paper, we present an overlay interface element superimposed on wall-display applications to help constrain interaction, focus attention on subsections of a display wall, and facilitate a collaborative multi-user workflow.

The intertwining of everyday life and computation, along with a new generation of inexpensive digital recording devices and storage facilities, is revolutionizing our ability to collect and analyze human activity data. Such ubiquitous data collection has an exciting potential to augment human cognition and radically improve information-intensive work. In this paper we introduce a system to aid the process of data collection and analysis during observational research by providing non-intrusive automatic capture of paper-based annotations. The system exploits current note-taking practices and incorporates digital pen technology. We describe the development, deployment and use of the system for interactive visualization and annotation of multiple stream of video and other types of time-based data.

Pen and paper support the rapid production of sketches. However, the paper interface is not always optimal for collaborative sketching as seen in brainstorming sessions where multiple parties would often like to communicate and participate in the sketching synchronously. Novel interactive paper solutions may provide the answer by bridging the paper-digital divide and allowing users to sketch on paper simultaneously while capturing the actions digitally. We present an analysis of collaborative sketching activities in working environments with remote participation. After highlighting the importance of paper for natural interaction in these settings, we introduce PaperSketch, an interactive paper-digital tool for collaborative remote sketching. We discuss the collaborative development of ideas based on the prototype and outline how important feedback issues have been addressed by utilising spatial constraints and multimodal features.

Digital social media have transformed how we communicate and manage our relationships. Despite its portability, sketching as a social medium has been largely left behind. Given sketching's unique affordances for visual communication this absence is a real loss. Sketches convey visuo-spatial ideas directly, require minimal detail to render concepts, and show the peculiarities of handwriting. Sketching holds the promise to enrich how we communicate, and its ubiquity is critical for sharing information at opportune moments. We present the results of an exploratory field study of ubiquitous sketching for social media, documenting users' experiences with UbiSketch. This system integrates digital pens, paper, and mobile phones to support the transmission of paper sketches to online services. We learned that UbiSketch enabled participants to leverage sketching's unique affordances, that ubiquitous sketching creates a synergy with the practice of posting context-dependent information, and that it broadens and deepens social interaction.

After a stroke or brain injury, it may be more difficult to understand language and communicate with others. Speech-language therapy may help an individual regain language and cope with changes in their communication abilities. Our research examines the process of speech-language therapy with an emphasis on the practices of therapists working with adults with aphasia and apraxia of speech. This paper presents findings from field work undertaken to inform the design of a mixed paper-digital interface prototype using multimodal digital pens. We describe and analyze therapists' initial reactions to the system and present two case studies of use by older adults undergoing speech-language therapy. We discuss the utility of multimodal paper-digital interfaces to assist therapy and describe our vision of a system to help therapists independently create custom interactive paper materials for their clients.

The first steps towards bridging the paper-digital divide have been achieved with the development of a range of technologies that allow printed documents to be linked to digital content and services. However, the static nature of paper and limited structural information encoded in classical paginated formats make it difficult to map between parts of a printed instance of a document and logical elements of a digital instance of the same document, especially taking document revisions into account. We present a solution to this problem based on a model that combines metadata of the digital and printed instances to enable a seamless mapping between digital documents and their physical counterparts on paper. We also describe how the model was used to develop iDoc, a framework that supports the authoring and publishing of interactive paper documents.

Citations form the basis for a web of scientific publications. Search engines, embedded hyperlinks and digital libraries all simplify the task of finding publications of interest on the web and navigating to cited publications or web sites. However the actual reading of publications often takes place on paper and frequently on the move. We present a system Print-n-Link that uses technologies for interactive paper to enhance the reading process by enabling users to access digital information and/or searches for cited documents from a printed version of a publication using a digital pen for interaction. A special virtual printer driver automatically generates links from paper to digital services during the printing process based on an analysis of PDF documents. Depending on the user setting and interaction gesture, the system may retrieve metadata about the citation and inform the user through an audio channel or directly display the cited document on the user's screen.

Our demonstration is a paper-based interactive guide for visitors to the world's largest international arts festival that was developed as part of a project investigating new forms of context-aware information delivery and interaction in mobile environments. Information stored in a database is accessed from a set of interactive paper documents, including a printed festival brochure, a city map and a bookmark. Active areas are defined within the documents and selection of these using a special digital pen causes the corresponding query request along with context data to be sent to a festival application database and the response is returned to the visitor in the form of generated speech output. In addition to paper-based information browsing and transactions such as ticket booking, the digital pen can also be applied for data capture of event ratings and handwritten comments on events. The system integrates three main database components - a cross-media information platform, a content management framework for multi-channel context-aware publishing of data and the festival application database.