Abstract The increased prevalence of misinformation and inflammatory rhetoric online has amplified polarization on social media platforms in the United States, propelling a feedback loop resulting in the erosion of democratic norms. We conducted a study assessing how a social media platform employing appointed moderators would impact the polarization of its users compared to a peer-based digital jury moderation system, which may be better able to harness community knowledge and cultural nuances while fostering a sense of inclusion and trust in the moderation process. Although our study did not observe a significant impact on the polarization of moderators or users, moderators on average viewed the system as just, legitimate and effective at reducing harmful content. Furthermore, there were no significant differences between user perceptions of the content they were shown from either system, indicating that implementing such a peer-based system has the benefit of providing users agency in platform governance without adversely impacting user experience.

In order to gather data from the signing deaf community, efforts have been made to create online surveys in American Sign Language (ASL), despite a lack of user studies and UX/UI design guidelines informing the development of online survey tools featuring ASL. In this paper, we present SL-Surveys, an ASL-centric survey tool prototype showcasing a set of potential designs for multiple-choice, scalar, and multi-select questions. SL-Surveys was developed in an iterative process expressly for an exploratory think-aloud study investigating user experiences and perceptions of the designs. This preliminary study was conducted with seven deaf ASL-signing participants using a computer. The new design process, prototypes, user study and results make important strides towards a future where designs are not constrained by existing standards and practices based on written languages.

In this paper, we explore the use of brain-computer interface (BCI)-adapted visual effects to support atmosphere in a walking simulator, and investigate its impact on player-reported immersive experience. While players were using a keyboard or joystick controller to control the basic movement of a character, their mental state was accessed by a non-invasive BCI technique called functional near-infrared spectroscopy (fNIRS) to implicitly adjust the visual effects. Specifically, when less brain activity is detected, the players’ in-game vision becomes blurry and distorted, recreating the impression of losing focus. With this biological indication, we designed a BCI-controlled game, in which the vision becomes blurry and distorted when less brain activity is detected, recreating the impression of losing focus. To analyze the player’s experience, we conducted a within-subjects study where participants played both a BCI-controlled and non-BCI-controlled game and completed a questionnaire after each session. We then conducted a semi-structured interview to investigate player perceptions of the impact the BCI had on their experiences. The results showed that players had slightly improved immersion in the BCI-adaptive game, with the temporal dissociation score significantly different. Players also reported the BCI-adaptive visual effects are realistic and natural, and they enjoyed using BCI as a supplemental control.

Questionnaires are fundamental learning and research tools for gathering insights and information from individuals, and now can be created easily using online tools. However, existing resources for creating questionnaires are designed for written languages (e.g. English) and do not support sign languages (e.g. American Sign Language). Sign languages (SLs) have unique visual characteristics that do not fit into user interface paradigms designed for written, text-based languages. Through a series of formative studies with the ASL signing community, this paper takes steps towards understanding the viability, potential benefit, challenges, and user interest in SL-centric surveys, a novel approach for creating questionnaires that meet the needs of deaf individuals using sign languages, without obligatory reliance on a written language to complete a questionnaire.

Recent work has investigated the construction of touch-sensitive knitted fabrics, capable of being manufactured at scale, and having only two connections to external hardware. Additionally, several sensor design patterns and application prototypes have been introduced. Our aim is to start shaping the future of this technology according to user expectations. Through a formative focus group study, we explore users’ views of using these fabrics in different contexts and discuss potential concerns and application areas. Subsequently, we take steps toward addressing relevant questions, by first providing design guidelines for application designers. Furthermore, in one user study, we demonstrate that it is possible to distinguish different swipe gestures and identify accidental contact with the sensor, a common occurrence in everyday life. We then present experiments investigating the effect of stretching and laundering of the sensors on their resistance, providing insights about considerations necessary to include in computational models.

Technology advances and lower equipment costs are enabling non-invasive, convenient recording of brain data outside of clinical settings in more real-world environments, and by non-experts. Despite the growing interest in and availability of brain signal datasets, most analytical tools are made for experts in the specific device technology, and have rigid constraints on the type of analysis available. We developed BrainEx to support interactive exploration and discovery within brain signals datasets. BrainEx takes advantage of algorithms that enable fast exploration of complex, large collections of time series data, while being easy to use and learn. This system enables researchers to perform similarity search, explore feature data and natural clustering, and select sequences of interest for future searches and exploration, while also maintaining the usability of a visual tool. In addition to describing the distributed architecture and visual design for BrainEx, this paper reports on a benchmark experiment showing that it outperforms other existing systems for similarity search. Additionally, we report on a preliminary user study in which domain experts used the visual exploration interface and affirmed that it meets the requirements. Finally, it presents a case study using BrainEx to explore real-world, domain-relevant data.

In human-computer interaction (HCI), there has been a push towards open science, but to date, this has not happened consistently for HCI research utilizing brain signals due to unclear guidelines to support reuse and reproduction. To understand existing practices in the field, this paper examines 110 publications, exploring domains, applications, modalities, mental states and processes, and more. This analysis reveals variance in how authors report experiments, which creates challenges to understand, reproduce, and build on that research. It then describes an overarching experiment model that provides a formal structure for reporting HCI research with brain signals, including definitions, terminology, categories, and examples for each aspect. Multiple distinct reporting styles were identified through factor analysis and tied to different types of research. The paper concludes with recommendations and discusses future challenges. This creates actionable items from the abstract model and empirical observations to make HCI research with brain signals more reproducible and reusable.

Conducting human-centered research by, with, and for the ASL-signing Deaf community, requires rethinking current human-computer interaction processes in order to meet their linguistic and cultural needs and expectations. This paper highlights some key considerations that emerged in our work creating an ASL-based questionnaire, and our recommendations for handling them.

Frameworks such as Direct Manipulation or Instrumental Interaction have been an important force in HCI research. Evaluating the impact of frameworks can identify whether and how a framework was used, how it has evolved, and what trends have developed over time. However, studying the impact of such theoretical contributions requires consideration of various perspectives and level of impact. As a case study for investigating the impact of theoretical work in HCI, we present our evaluation of the impact of the Reality Based Interaction (RBI) framework, introduced by the authors in 2008. We provide our findings about the impact of the framework both on contemporary research, through content-based citation analysis, and in HCI education, through a survey we conducted on emerging interaction frameworks. The article contributes a comprehensive methodology for evaluating the impact of frameworks through our twofold approach: content-based citation analysis, including the design of a new citation typology; and a survey on the use of frameworks in education using a taxonomy of learning goals. We also consider the role of frameworks in HCI as well as the future of the RBI framework.

Multimodal signals allow us to gain insights into internal cognitive processes of a person, for example: speech and gesture analysis yields cues about hesitations, knowledgeability, or alertness, eye tracking yields information about a person's focus of attention, task, or cognitive state, EEG yields information about a person's cognitive load or information appraisal. Capturing cognitive processes is an important research tool to understand human behavior as well as a crucial part of a user model to an adaptive interactive system such as a robot or a tutoring system. As cognitive processes are often multifaceted, a comprehensive model requires the combination of multiple complementary signals. In this workshop at the ACM International Conference on Multimodal Interfaces (ICMI) conference in Boulder, Colorado, USA, we discussed the state-of-the-art in monitoring and modeling cognitive processes from multi-modal signals.

Collaborative ideation systems can help people generate more creative ideas by exposing them to ideas different from their own. However, there are competing theoretical views on whether and when such exposure is helpful. Associationist theory suggests that exposing ideators to ideas that are semantically far from their own maximizes novel combinations of ideas. In contrast, SIAM theory cautions that systems should offer far ideas only when ideators reach an impasse (a cognitive state in which they have exhausted ideas within a particular category), and offer near ideas during productive ideation (a cognitive state in which they are actively exploring ideas within a category), which maximizes exploration within categories. Our research compares these theoretical recommendations. In an online experiment, 245 participants generated ideas for a themed wedding; we detected and validated participants' cognitive states using a combination of behavioral and neuroimaging data. Receiving far ideas during productive ideation resulted in slower ideation and less within-category exploration, without significant benefits for novelty, compared to receiving no inspirations. Participants were also more likely to hit an impasse when receiving far ideas during productive ideation. These findings suggest that far inspirational ideas can harm creativity if received during productive ideation.

In this paper, we introduce the American Sign Language STEM Concept Learning Resource (ASL CLeaR), an educational application demo. The ASL CLeaR addresses a need for quality ASL STEM resources by featuring expertly presented STEM content in ASL, and employing an ASL-based search function and a visuocentric search interface. This paper discusses the main objectives of the ASL CLeaR, describes the components of the application, and suggests future work that could lead to improved educational outcomes for deaf and hard of hearing students in STEM topics.

With improving automation in many critical domains, operators will be expected to handle long periods of low task load while monitoring a system, and possibly responding to emergent situations. Monitoring the psychophysiological state of the operator during low task load may detect maladapted attention states in order to predict performance and facilitate a more effective workload transition during critical periods. This research explored the question of detecting anomalous attention states during transitions to high workload following extended periods of boredom using a non-invasive neuroimaging technique called functional near-infrared spectroscopy (fNIRS). Subjects at the point of lowest engagement and priming had a diminished hemodynamic response and performed worse on missile defense task, showing fNIRS may be useful for concurrent monitoring of the operator in such settings. RESEARCH HIGHLIGHTS Functional near-infrared spectroscopy brain sensing is feasible for use in long duration (3 h) tasks. Hemodynamic response was diminished during the middle of a long duration, low task load simulation when engagement and priming were lowest. fNIRS did not detect a change in workload, but did reflect temporal changes in event onset, which could be used to automatically adapt a system when an operator is in a degraded attention state.

A growing body of recent work has shown the feasibility of brain and body sensors as input to interactive systems. However, the interaction techniques and design decisions for their effective use are not well defined. We present a conceptual framework for considering implicit input from the brain, along with design principles and patterns we have developed from our work. We also describe a series of controlled, offline studies that lay the foundation for our work with functional near-infrared spectroscopy (fNIRS) neuroimaging, as well as our real-time platform that serves as a testbed for exploring brain-based adaptive interaction techniques. Finally, we present case studies illustrating the principles and patterns for effective use of brain data in human--computer interaction. We focus on signals coming from the brain, but these principles apply broadly to other sensor data and in domains such as aviation, education, medicine, driving, and anything involving multitasking or varying cognitive workload.

Dynamic difficulty adjustments can be used in human-computer systems in order to improve user engagement and performance. In this paper, we use functional near-infrared spectroscopy (fNIRS) to obtain passive brain sensing data and detect extended periods of boredom or overload. From these physiological signals, we can adapt a simulation in order to optimize workload in real-time, which allows the system to better fit the task to the user from moment to moment. To demonstrate this idea, we ran a laboratory study in which participants performed path planning for multiple unmanned aerial vehicles (UAVs) in a simulation. Based on their state, we varied the difficulty of the task by adding or removing UAVs and found that we were able to decrease error by 35% over a baseline condition. Our results show that we can use fNIRS brain sensing to detect task difficulty in real-time and construct an interface that improves user performance through dynamic difficulty adjustment.

Understanding the driver's cognitive load is important for evaluating in-vehicle user interfaces. This paper describes experiments to assess machine learning classification algorithms on their ability to automatically identify elevated cognitive workload levels in drivers, leading towards the development of robust tools for automobile user interface evaluation. We look at using both driver performance as well as physiological data. These measures can be collected in real-time and do not interfere with the primary task of driving the vehicle. We report classification accuracies of up to 90% for detecting elevated levels of cognitive load, and show that the inclusion of physiological data leads to higher classification accuracy than vehicle sensor data evaluated alone. Finally, we show results suggesting that models can be built to classify cognitive load across individuals, instead of building individual models for each per-son. By collecting data from drivers in two large field studies on the highway (20 drivers and 99 drivers), this work extends prior work and demonstrates feasibility and potential of such measures for HCI research in vehicles.

Simultaneously controlling multiple robots requires multiple operators working together as a team. Determining how to construct the team to promote performance and reduce workload are critical questions that must be answered in these settings. To this end, we investigated the effect of team structure and scheduling notification on operators' performance, subjective workload, work processes, and communication using a human-in-the-loop experiment. In an urban search and rescue setting, we compared a pooled condition, in which team members shared control of 24 robots, with a sector condition, in which each team member controlled half of all the robots. For scheduling notification, an alert was given when the operator spent too much time on one robot and either suggested or forced the operator to change to another robot. A discrete-event simulation model was constructed to model the teamwork in supervisory control of multiple robots. The model was significantly improved by the inclusion of a behavior termed as “backup.” Backup behavior is a critical coordination mechanism often observed in teams, but rarely explicitly modeled. Pooled teams showed an advantage when performing backup behaviors in both the experiment and the model. However, other factors must be considered when making a decision on what team structure to use.

There is an increasing requirement for advanced analytical methodologies to help military intelligence analysts cope with the growing amount of data they are saturated with on a daily basis. Specifically, within the context of terror network analysis, one of the largest problems is the transformation of raw tabular data into a visualization that is easily and effectively exploited by intelligence analysts. Currently, the primary method within the intelligence do-main is the node-link visualization, which encodes data sets by depicting the ties between nodes as lines between objects in a plane. This method, although useful, has limitations when the size and complexity of data grows. The matrix offers an alternate perspective because the two dimensions of the matrix are arrayed as an actors x actors matrix. This paper describes an experiment investigating node-link and matrix visualization techniques within social network analysis, and their effectiveness for the intelligence tasks of: 1) identifying leaders and 2) identifying clusters. The sixty participants in the experiment were all Air Force intelligence analysts and we provide recommendations for building visualization tools for this specialized group of users.

This paper describes the Brainput system, which learns to identify brain activity patterns occurring during multitasking. It provides a continuous, supplemental input stream to an interactive human-robot system, which uses this information to modify its behavior to better support multitasking. This paper demonstrates that we can use non-invasive methods to detect signals coming from the brain that users naturally and effortlessly generate while using a computer system. If used with care, this additional information can lead to systems that respond appropriately to changes in the user's state. Our experimental study shows that Brainput significantly improves several performance metrics, as well as the subjective NASA-Task Load Index scores in a dual-task human-robot activity.

Multitasking has become an integral part of work environments, even though people are not well-equipped cognitively to handle numerous concurrent tasks effectively. Systems that support such multitasking may produce better performance and less frustration. However, without understanding the user's internal processes, it is difficult to determine optimal strategies for adapting interfaces, since all multitasking activity is not identical. We describe two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Using functional near-infrared spectroscopy sensors, we differentiate four cognitive multitasking processes. These states cannot readily be distinguished using behavioral measures such as response time, accuracy, keystrokes or screen contents. We then present our human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. This prototype system serves as a platform to study interfaces that enable better task switching, interruption management, and multitasking.

A well designed user interface (UI) should be transparent, allowing users to focus their mental workload on the task at hand. We hypothesize that the overall mental workload required to perform a task using a computer system is composed of a portion attributable to the difficulty of the underlying task plus a portion attributable to the complexity of operating the user interface. In this regard, we follow Shneiderman's theory of syntactic and semantic components of a UI. We present an experiment protocol that can be used to measure the workload experienced by users in their various cognitive resources while working with a computer. We then describe an experiment where we used the protocol to quantify the syntactic workload of two user interfaces. We use functional near infrared spectroscopy, a new brain imaging technology that is beginning to be used in HCI. We also discuss extensions of our techniques to adaptive interfaces.

Much of the work done in the field of tangible interaction has focused on creating tools for learning; however, in many cases, little evidence has been provided that tangible interfaces offer educational benefits compared to more conventional interaction techniques. In this paper, we present a study comparing the use of a tangible and a graphical interface as part of an interactive computer programming and robotics exhibit that we designed for the Boston Museum of Science. In this study, we have collected observations of 260 museum visitors and conducted interviews with 13 family groups. Our results show that visitors found the tangible and the graphical systems equally easy to understand. However, with the tangible interface, visitors were significantly more likely to try the exhibit and significantly more likely to actively participate in groups. In turn, we show that regardless of the condition, involving multiple active participants leads to significantly longer interaction times. Finally, we examine the role of children and adults in each condition and present evidence that children are more actively involved in the tangible condition, an effect that seems to be especially strong for girls.

Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.

In this paper we describe the design and initial evaluation of a tangible computer programming exhibit for children on display at the Boston Museum of Science. We also discuss five design considerations for tangible interfaces in science museums that guided our development and evaluation. In doing so, we propose the notion of passive tangible interfaces. Passive tangibles serve as a way to address practical issues involving tangible interaction in public settings and as a design strategy to promote reflective thinking. Results from our evaluation indicate that passive tangibles can preserve many of the benefits of tangible interaction for informal science learning while remaining cost-effective and reliable.

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI, we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI provides insights for design and uncovers gaps or opportunities for future research.

We created Smart Blocks, an augmented mathematical manipulative that allows users to explore the concepts of volume and surface area of 3-dimensional (3D) objects. This interface supports physical manipulation for exploring spatial relationships and it provides continuous feedback for reinforcing learning. By leveraging the benefits of physicality with the advantages of digital information, this tangible interface provides an engaging environment for learning about surface area and volume of 3D objects.