Human-in-the-loop methods leverage human feedback to enhance machine learning and AI. Manual review of outputs can correct errors, identify model weaknesses, or expand labels to broaden model capabilities. Feedback collection methods range from simple flagging of outputs as correct or incorrect to more complex feature-level adjustments or natural language interpretations. This article presents a user study evaluating changes in user performance over time and explores the tradeoff between feedback quality and human effort. We compare four interactive input methods for reviewing and correcting outcomes in object detection and activity recognition in videos. Our findings indicate that while some complex input methods, such as free-text, require more time, the quality and impact of their feedback on model accuracy often surpass those of simpler methods that require less effort. However, more effort does not always lead to better-quality feedback, especially when aiming to improve the model. Our VLM experiments show that the most accurate models were trained using detailed natural language feedback or precise word-level corrections, while simple yes/no judgments also led to solid performance at a much lower annotation cost.

Augmented reality (AR) and virtual reality (VR) can enhance task guidance by overlaying visual information to improve efficiency and reduce errors. However, challenges remain in designing the appropriate presentation format and amount of information for real-time assistance. Prior research has shown benefits of visual cues in procedural tasks, but these findings are limited to simplified scenarios, highlighting a gap in understanding their effectiveness for complex, real-world applications. Therefore, we study visual design and cue effectiveness in the context of compound procedures encompassing subtasks and heterogeneous instructions. We present an experiment assessing different visual cues in VR to test a user’s ability to harness distinct information streams for different tasks, separating cues for object search and object placement for multi-step procedures. The results show that even for compound tasks requiring processing of multiple types of information, the addition of simple interaction cues for individual subtasks did significantly improved task performance for both time and errors. However, in contrast to prior studies showing successful precueing of future steps in more simplistic tasks, the study did not find evidence of precueing with the more complex tasks.

Knowledge graphs are often visualized using node-link diagrams that reveal relationships and structure. In many applications using graphs, it is desirable to allow users to edit graphs to ensure data accuracy or provides updates. Commonly in graph visualization, users can interact directly with the visual elements by clicking and typing updates to specific items through traditional interaction methods in the graphical user interface. However, it can become tedious to make many updates due to the need to individually select and change numerous items in a graph. Our research investigates natural language input as an alternative method for editing network graphs. We present a user study comparing GUI graph editing with two natural language alternatives to contribute novel empirical data of the trade-offs of the different interaction methods. The findings show natural language methods to be significantly more effective than traditional GUI interaction.

Existing video analysis models often lack explainability, perform poorly on long videos, and frequently hallucinate. Commercial solutions are closed-source and costly. We introduce CReLeRI, an open-source system for action detection in untrimmed videos. CReLeRI segments videos using scene and action transitions, detects actions and their arguments and grounds them in 3D space to improve interpretability and reduce hallucinations. The system promotes transparency and trust in AI-driven analysis of complex, real-world videos. A demonstration video is also available.

The prevalent evaluation methods used to estimate detection of redirected walking are based on methods from psychophysics that require users to know their virtual movements are being manipulated. However, this higher-than-normal level of attention toward their movements yields conservative detection thresholds. We find that participants who were unaware that redirected walking (translation gain) was applied detected the technique at a significantly higher gain than users who were aware (at gains of 1.73 and 1.38, respectively). We provide evidence that redirected walking-based navigation solutions may be able to leverage gain values that are larger than the current threshold guidelines would suggest.

For editing 3D spatial data, 3D interaction through virtual reality (VR) can be a viable alternative to 2D interfaces: research indicates that model editing in VR provides a more enjoyable experience and is fast to learn. These advantages make VR an appealing option for training new users’ spatial understanding before transitioning to standard 2D tools, like Blender and Maya. But how much does model editing in VR benefit the trained user? Our experiment compares the modeling accuracy of non-modelers, casual users, and formally trained artists for objects of varying complexity in desktop and VR. For users with no prior modeling experience, the study found significant improvements in qualitative accuracy and efficiency of aesthetic edits using VR. Importantly, improvements decreased with higher user experience and varied with types of editing for different surface features. The findings suggest that adaptation of traditional desktop modeling tools to VR should be situational decisions based on specific modeling scenarios.

Interpreting 3D information based on 2D slices of the data is notoriously difficult. Yet in the medical field makes intervention choices based on radiological scans on a daily basis. Volumetric reconstructions of these data sets, via voxel clouds and surface meshes can reduce the cognitive complexity of this task. However, creating these reconstructions requires extensive software training and time, often on the part of a technician separate from the treatment team. To shorten this process, we present a system designed for radiologists and surgeons on the care team. We integrate tools familiar to these experts and provide a stereoscopic environment for quick spatial comprehension and intuitive data curation. Based on the feedback from oncology collaborators on the system in its current state, it is not only helpful for communicating tumor progression but, additionally shows promise as a teaching tool to assist with building skills for traditional interpretation of radiology images.

Following step-by-step procedures is an essential component of various activities carried out by individuals in their daily lives. These procedures serve as a guiding framework that helps to achieve goals efficiently, whether it is assembling furniture or preparing a recipe. However, the complexity and duration of procedural activities inherently increase the likelihood of making errors. Understanding such procedural activities from a sequence of frames is a challenging task that demands an accurate interpretation of visual information and the ability to reason about the structure of the activity. To this end, we collect a new egocentric 4D dataset, CaptainCook4D, comprising 384 recordings (94.5 hours) of people performing recipes in real kitchen environments. This dataset consists of two distinct types of activity: one in which participants adhere to the provided recipe instructions and another in which they deviate and induce errors. We provide 5.3K step annotations and 10K fine-grained action annotations and benchmark the dataset for the following tasks: error recognition, multistep localization and procedure learning.

Input remapping techniques have been widely explored to allow users in virtual reality to exceed both their own physical abilities, the limitations of physical space, or to facilitate interactions with real-world objects. Often considered is how these techniques can be applied to achieve maximum utility, but still be undetectable to users to maintain a sense of immersion and presence. Existing psychophysical methods used to determine these detection thresholds have known limitations: they are highly conservative lower bounds for detection and do not account for complex usage of the technique. Our work describes and evaluates a method for estimating detection that reduces these limitations and yields meaningful upper bounds. We present the findings of our work where we apply this method to a well-explored hand motion scaling technique. In wholly unaware cases, we determined that users may detect their hand speed as abnormal at around 3.37 times the normal speed, compared to a scale factor of 1.47 that was estimated using traditional methods when users knew the motion scaling was occurring. A considerable number of participants in unaware cases (12 of 56) never detected their hand speed increasing at all, even at the maximum scale factor of 5.0. The study demonstrates just how conservative the thresholds generated by traditional psychophysical methods can be compared to detection during naive usage, and our method can be modified and applied easily to other techniques.

Head tracking is commonly used in VR applications to allow users to naturally view 3D content using physical head movement, but many applications also support turning with hand-held controllers. Controller and joystick controls are convenient for practical settings where full 360-degree physical rotation is not possible, such as when the user is sitting at a desk. Though controller-based rotation provides the benefit of convenience, previous research has demonstrated that virtual or joystick-controlled view rotation to have drawbacks of sickness and disorientation compared to physical turning. To combat such issues, researchers have considered various techniques such as speed adjustments or reduced field of view, but data is limited on how different variations for joystick rotation influences sickness and orientation perception. Our studies include different variations of techniques such as joystick rotation, resetting, and field-of-view reduction. We investigate trade-offs among different techniques in terms of sickness and the ability to maintain spatial orientation. In two controlled experiments, participants traveled through a sequence of rooms and were tested on spatial orientation, and we also collected subjective measures of sickness and preference. Our findings indicate a preference by users towards directly-manipulated joystick-based rotations compared to user-initiated resetting and minimal effects of technique on spatial awareness.

We consider the following video activity recognition (VAR) task: given a video, infer the set of activities being performed in the video and assign each frame to an activity. Although VAR can be solved accurately using existing deep learning techniques, deep networks are neither interpretable nor explainable and as a result their use is problematic in high stakes decision-making applications (in healthcare, experimental Biology, aviation, law, etc.). In such applications, failure may lead to disastrous consequences and therefore it is necessary that the user is able to either understand the inner workings of the model or probe it to understand its reasoning patterns for a given decision. We address these limitations of deep networks by proposing a new approach that feeds the output of a deep model into a tractable, interpretable probabilistic model called a dynamic conditional cutset network that is defined over the explanatory and output variables and then performing joint inference over the combined model. The two key benefits of using cutset networks are: (a) they explicitly model the relationship between the output and explanatory variables and as a result, the combined model is likely to be more accurate than the vanilla deep model and (b) they can answer reasoning queries in polynomial time and as a result, they can derive meaningful explanations by efficiently answering explanation queries. We demonstrate the efficacy of our approach on two datasets, Textually Annotated Cooking Scenes (TACoS), and wet lab, using conventional evaluation measures such as the Jaccard Index and Hamming Loss, as well as a human-subjects study.

Conducting data analysis tasks rarely occur in isolation. Especially in intelligence analysis scenarios where different experts contribute knowledge to a shared understanding, members must communicate how insights develop to establish common ground among collaborators. The use of provenance to communicate analytic sensemaking carries promise by describing the interactions and summarizing the steps taken to reach insights. Yet, no universal guidelines exist for communicating provenance in different settings. Our work focuses on the presentation of provenance information and the resulting conclusions reached and strategies used by new analysts. In an open-ended, 30-minute, textual exploration scenario, we qualitatively compare how adding different types of provenance information (specifically data coverage and interaction history) affects analysts' confidence in conclusions developed, propensity to repeat work, filtering of data, identification of relevant information, and typical investigation strategies. We see that data coverage (i.e., what was interacted with) provides provenance information without limiting individual investigation freedom. On the other hand, while interaction history (i.e., when something was interacted with) does not significantly encourage more mimicry, it does take more time to comfortably understand, as represented by less confident conclusions and less relevant information-gathering behaviors. Our results contribute empirical data towards understanding how provenance summarizations can influence analysis behaviors.

Information visualization uses various types of representations to encode data into graphical formats. Prior work on visualization techniques has evaluated the accuracy of perceived numerical data values from visual data encodings such as graphical position, length, orientation, size, and color. Our work aims to extend the research of graphical perception to the use of motion as data encodings for quantitative values. We present two experiments implementing multiple fundamental aspects of motion such as type, speed, and synchronicity that can be used for numerical value encoding as well as comparing motion to static visual encodings in terms of user perception and accuracy. We studied how well users can assess the differences between several types of motion and static visual encodings and present an updated ranking of accuracy for quantitative judgments. Our results indicate that non-synchronized motion can be interpreted more quickly and more accurately than synchronized motion. Moreover, our ranking of static and motion visual representations shows that motion, especially expansion and translational types, has great potential as a data encoding technique for quantitative value. Finally, we discuss the implications for the use of animation and motion for numerical representations in data visualization.

Redirected walking can effectively utilize a user’s physical space when traversing larger virtual environments by using virtual self-motion gains for a user’s physical motions. In particular, curvature gain presents unique advantages in redirection but can lead to suboptimal orientations. To prevent this and add additional utility in redirected walking, we formally present strafing gain. Strafing gain seeks to add incremental lateral movements to a user’s position causing the user to walk along a diagonal trajectory while maintaining the original orientation of the user. In a study with 27 participants, we tested 11 values to determine the detection thresholds of strafing gain. The study, which was modeled on prior detection threshold studies, found that strafing gain could successfully redirect participants to walk along a 5.57° diagonal to the right and a 4.68° diagonal to the left. Furthermore, a supplementary study with 10 participants was conducted, verifying that orientation was maintained throughout redirection and validating the obtained detection thresholds. We discuss the implications of these findings and potential ways of improving these quantities in real-world applications.

While EXplainable Artificial Intelligence (XAI) approaches aim to improve human-AI collaborative decision-making by improving model transparency and mental model formations, experiential factors associated with human users can cause challenges in ways system designers do not anticipate. In this article, we first showcase a user study on how anchoring bias can potentially affect mental model formations when users initially interact with an intelligent system and the role of explanations in addressing this bias. Using a video activity recognition tool in cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. However, those who encountered weaknesses earlier made significantly fewer errors, since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Motivated by these findings and similar existing work, we formalize and present a conceptual model of user’s past experiences that examine the relations between user’s backgrounds, experiences, and human factors in XAI systems based on usage time. Our work presents strong findings and implications, aiming to raise the awareness of AI designers toward biases associated with user impressions and backgrounds.

Avatars are often used to provide representations of users in 3D environments, such as desktop games or VR applications. While full-body avatars are often sought to be used in applications, low visibility avatars (i.e., head and hands) are often used in a variety of contexts, either as intentional design choices, for simplicity in contexts where full-body avatars are not needed, or due to external limitations. Avatar style can also vary from more simplistic and abstract to highly realistic depending on application context and user choices. We present the results of two desktop experiments that examine avatar visibility, style, and observer view on accuracy in a pointing interpretation task. Significant effects of visibility were found, with effects varying between horizontal and vertical components of error, and error amounts not always worsening as a result of lowering visibility. Error due to avatar visibility was much smaller than error resulting from avatar style or observer view. Our findings suggest that humans are reasonably able to understand pointing gestures with a limited observable body.

Research in interpretable machine learning proposes different computational and human subject approaches to evaluate model saliency explanations. These approaches measure different qualities of explanations to achieve diverse goals in designing interpretable machine learning systems. In this paper, we propose a benchmark for image and text domains using multi-layer human attention masks aggregated from multiple human annotators. We then present an evaluation study to compare model saliency explanations obtained using Grad-cam and LIME techniques to human understanding and acceptance. We demonstrate our benchmark’s utility for quantitative evaluation of model explanations by comparing it with human subjective ratings and ground-truth single-layer segmentation masks evaluations. Our study results show that our threshold agnostic evaluation method with the human attention baseline is more effective than single-layer object segmentation masks to ground truth. Our experiments also reveal user biases in the subjective rating of model saliency explanations.

EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools.

OBJECTIVE: While patients often contribute data for research, they want researchers to protect their data. As part of a participatory design of privacy-enhancing software, this study explored patients' perceptions of privacy protection in research using their healthcare data. MATERIALS AND METHODS: We conducted 4 focus groups with 27 patients on privacy-enhancing software using the nominal group technique. We provided participants with an open source software prototype to demonstrate privacy-enhancing features and elicit privacy concerns. Participants generated ideas on benefits, risks, and needed additional information. Following a thematic analysis of the results, we deployed an online questionnaire to identify consensus across all 4 groups. Participants were asked to rank-order benefits and risks. Themes around "needed additional information" were rated by perceived importance on a 5-point Likert scale. RESULTS: Participants considered "allowance for minimum disclosure" and "comprehensive privacy protection that is not currently available" as the most important benefits when using the privacy-enhancing prototype software. The most concerning perceived risks were "additional checks needed beyond the software to ensure privacy protection" and the "potential of misuse by authorized users." Participants indicated a desire for additional information with 6 of the 11 themes receiving a median participant rating of "very necessary" and rated "information on the data custodian" as "essential." CONCLUSIONS: Patients recognize not only the benefits of privacy-enhancing software, but also inherent risks. Patients desire information about how their data are used and protected. Effective patient engagement, communication, and transparency in research may improve patients' comfort levels, alleviate patients' concerns, and thus promote ethical research.

The need for interpretable and accountable intelligent systems grows along with the prevalence of artificial intelligence ( AI ) applications used in everyday life. Explainable AI ( XAI ) systems are intended to self-explain the reasoning behind system decisions and predictions. Researchers from different disciplines work together to define, design, and evaluate explainable systems. However, scholars from different disciplines focus on different objectives and fairly independent topics of XAI research, which poses challenges for identifying appropriate design and evaluation methodology and consolidating knowledge across efforts. To this end, this article presents a survey and framework intended to share knowledge and experiences of XAI design and evaluation methods across multiple disciplines. Aiming to support diverse design goals and evaluation methods in XAI research, after a thorough review of XAI related papers in the fields of machine learning, visualization, and human-computer interaction, we present a categorization of XAI design goals and evaluation methods. Our categorization presents the mapping between design goals for different XAI user groups and their evaluation methods. From our findings, we develop a framework with step-by-step design guidelines paired with evaluation methods to close the iterative design and evaluation cycles in multidisciplinary XAI teams. Further, we provide summarized ready-to-use tables of evaluation methods and recommendations for different goals in XAI research.

The visualization of hierarchically structured data over time is an ongoing challenge and several approaches exist trying to solve it. Techniques such as animated or juxtaposed tree visualizations are not capable of providing a good overview of the time series and lack expressiveness in conveying changes over time. Nested streamgraphs provide a better understanding of the data evolution, but lack the clear outline of hierarchical structures at a given timestep. Furthermore, these approaches are often limited to static hierarchies or exclude complex hierarchical changes in the data, limiting their use cases. We propose a novel visual metaphor capable of providing a static overview of all hierarchical changes over time, as well as clearly outlining the hierarchical structure at each individual time step. Our method allows for smooth transitions between treemaps and nested streamgraphs, enabling the exploration of the trade-off between dynamic behavior and hierarchical structure. As our technique handles topological changes of all types, it is suitable for a wide range of applications. We demonstrate the utility of our method on several use cases, evaluate it with a user study, and provide its full source code.

In many types of dynamic interactive visualizations, it is often desired to interact with moving objects. Stopping moving objects can make selection easier, but pausing animated content can disrupt perception and understanding of the visualization. To address such problems, we explore selection techniques that only pause a subset of all moving targets in the visualization. We present various designs for controlling pause regions based on cursor trajectory or cursor position. We then report a dual-task experiment that evaluates how different techniques affect both target selection performance and contextual awareness of the visualization. Our findings indicate that all pause techniques significantly improved selection performance as compared to the baseline method without pause, but the results also show that pausing the entire visualization can interfere with contextual awareness. However, the problem with reduced contextual awareness was not observed with our new techniques that only pause a limited region of the visualization. Thus, our research provides evidence that region-limited pause techniques can retain the advantages of selection in dynamic visualizations without imposing a negative effect on contextual awareness.

Analysis of streaming data often involves both real-time monitoring of incoming data as well as contextual awareness of data history. A focus-plus-context approach can support both goals, with variable levels of visual aggregation making it possible to provide a high level of detail for incoming and recent data while providing contextual information about recent history. Visual aggregation reduces data resolution in order to show the context of data over large periods of time within a limited display space. With a controlled experiment, we evaluated the effectiveness of different types of aggregation for four types of stream-analysis tasks. Overall, the results show that a focus-plus-context design has little negative impact on the ability to successfully monitor and analyze streaming data, making it possible to show longer periods of time than other approaches. However, visual aggregation can be problematic for trend recognition tasks. This research demonstrates how the effectiveness of the visualization depends on the specifics of the analysis task.

Mixed-initiative systems allow users to interactively provide feedback to potentially improve system performance. Human feedback can correct model errors and update model parameters to dynamically adapt to changing data. Additionally, many users desire the ability to have a greater level of control and fix perceived flaws in systems they rely on. However, how the ability to provide feedback to autonomous systems influences user trust is a largely unexplored area of research. Our research investigates how the act of providing feedback can affect user understanding of an intelligent system and its accuracy. We present a controlled experiment using a simulated object detection system with image data to study the effects of interactive feedback collection on user impressions. The results show that providing human-in-the-loop feedback lowered both participants’ trust in the system and their perception of system accuracy, regardless of whether the system accuracy improved in response to their feedback. These results highlight the importance of considering the effects of allowing end-user feedback on user trust when designing intelligent systems.

Domain-specific intelligent systems are meant to help system users in their decision-making process. Many systems aim to simultaneously support different users with varying levels of domain expertise, but prior domain knowledge can affect user trust and confidence in detecting system errors. While it is also known that user trust can be influenced by first impressions with intelligent systems, our research explores the relationship between ordering bias and domain expertise when encountering errors in intelligent systems. In this paper, we present a controlled user study to explore the role of domain knowledge in establishing trust and susceptibility to the influence of first impressions on user trust. Participants reviewed an explainable image classifier with a constant accuracy and two different orders of observing system errors (observing errors in the beginning of usage vs. in the end). Our findings indicate that encountering errors early-on can cause negative first impressions for domain experts, negatively impacting their trust over the course of interactions. However, encountering correct outputs early helps more knowledgable users to dynamically adjust their trust based on their observations of system performance. In contrast, novice users suffer from over-reliance due to their lack of proper knowledge to detect errors.

Virtual reality commonly makes use of tracked hand interactions for user input. Interaction techniques sometimes alter the mapping between the real and virtual coordinate systems to modify interaction possibilities. This paper studies fixed positional offsets applied to the location of the virtual hand. We present a controlled experiment in which users' hands were subject to fixed positional offsets of varying magnitudes while completing target-touching tasks. The study provides estimations for detection thresholds for positional hand offsets in six directions relative to the real-world location of the hand and provides evidence performance using offset virtual hands can vary based on offset parameters. Significant differences in offset detection were identified based on offset direction, indicating that positional adjustments made to virtual hands should consider directionality when limiting techniques rather than just a constant value. Hand offsets kept within the threshold value resulted in comparable performance to unmodified hand registration, while offsets beyond the threshold resulted in larger completion times.

In virtual reality (VR), natural physical hand interaction allows users to interact with virtual content using physical gestures. While the most straightforward use of tracked hand motion maintains a one-to-one mapping between the physical and virtual world, some cases might benefit from changing this mapping through scaled or redirected interactions that modify the mapping between user’s physical movements and the magnitude of corresponding virtual movements. However, large deviations in interaction fidelity may potentially provide distractions or a loss of perceived realism. Therefore, it is important to know the extent to which remapping techniques can be applied to scaled interactions in VR without users detecting the difference. In this paper, we extend prior research on redirected hand techniques by investigating user perception of scaled hand movements and estimating detection thresholds for different types of hand motion in VR. We conducted two experiments with a two-alternative forced-choice (2AFC) design to estimate the detection thresholds of remapped interaction. The first experiment tested the perception of motion scaling for simple hand movements, and the second experiment involved more complex reaching motions in a cognitively demanding game scenario. We present estimated detection thresholds for scale values that can be applied to virtual hand movements without users noticing the difference. Our findings show that detection thresholds differ significantly based on the type of hand movement (horizontal, vertical, and depth).

Various viewing and travel techniques are used in immersive virtual reality to allow users to see different areas or perspectives of 3D environments. Our research evaluates techniques for visually showing transitions between two viewpoints in head-tracked virtual reality. We present four experiments that focus on automated viewpoint changes that are controlled by the system rather than by interactive user control. The experiments evaluate three different transition techniques (teleportation, animated interpolation, and pulsed interpolation), different types of visual adjustments for each technique, and different types of viewpoint changes. We evaluated how differences in transition can influence a viewer's comfort, sickness, and ability to maintain spatial awareness of dynamic objects in a virtual scene. For instant teleportations, the experiments found participants could most easily track scene changes with rotational transitions without translational movements. Among the tested techniques, animated interpolations allowed significantly better spatial awareness of moving objects, but the animated technique was also rated worst in terms of sickness, particularly for rotational viewpoint changes. Across techniques, viewpoint transitions involving both translational and rotational changes together were more difficult to track than either individual type of change.

Machine learning and artificial intelligence algorithms can assist human decision making and analysis tasks. While such technology shows promise, willingness to use and rely on intelligent systems may depend on whether people can trust and understand them. To address this issue, researchers have explored the use of explainable interfaces that attempt to help explain why or how a system produced the output for a given input. However, the effects of meaningful and meaningless explanations (determined by their alignment with human logic) are not properly understood, especially with users who are non-experts in data science. Additionally, we wanted to explore how explanation inclusion and level of meaningfulness would affect the user’s perception of accuracy. We designed a controlled experiment using an image classification scenario with local explanations to evaluate and better understand these issues. Our results show that whether explanations are human-meaningful can significantly affect perception of a system’s accuracy independent of the actual accuracy observed from system usage. Participants significantly underestimated the system’s accuracy when it provided weak, less human-meaningful explanations. Therefore, for intelligent systems with explainable interfaces, this research demonstrates that users are less likely to accurately judge the accuracy of algorithms that do not operate based on human-understandable rationale.

This paper presents the design of Blocks4DS, a block-based environment for students to learn data structures. As a proof-of-concept, we designed custom blocks to allow students to build and visualize Binary Search Trees (BST). Blocks4DS is built on Blockly and uses vis.js to provide visualizations of the binary search tree and its operations. This paper describes the results from an initial evaluation of usability and student learning.

Immersive movies take advantage of virtual reality (VR) to bring new opportunities for storytelling that allow users to naturally turn their heads and bodies to view a 3D virtual world and follow the story in a surrounding space. However, while many designers often assume scenarios where viewers stand and are free to physically turn without constraints, this excludes many commonly desired usage settings where the user may wish to remain seated, such as the use of VR while relaxing on the couch or passing the time during a flight. For such situations, large amounts of physical turning may be uncomfortable due to neck strain or awkward twisting. Our research investigates a technique that automatically rotates the virtual scene to help redirect the viewer's physical rotation while viewing immersive narrative experiences. By slowly rotating the virtual content, viewers are encouraged to gradually turn physically to align their head positions to a more comfortable straight-ahead viewing direction in seated situations where physical turning is not ideal. We present our study of technique design and an evaluation of how the redirection approach affects user comfort, sickness, the amount of physical rotation, and likelihood of viewers noticing the rotational adjustments. Evaluation results show the rotation technique was effective at significantly reducing the amount of physical turning while watching immersive videos, and only 39% of participants noticed the automated rotation when the technique rotated at a speed of 3 degrees per second.

In this demo paper, we present the XFake system, an explainable fake news detector that assists end-users to identify news credibility. To effectively detect and interpret the fakeness of news items, we jointly consider both attributes (e.g., speaker) and statements. Specifically, MIMIC, ATTN and PERT frameworks are designed, where MIMIC is built for attribute analysis, ATTN is for statement semantic analysis and PERT is for statement linguistic analysis. Beyond the explanations extracted from the designed frameworks, relevant supporting examples as well as visualization are further provided to facilitate the interpretation. Our implemented system is demonstrated on a real-world dataset crawled from PolitiFact1, where thousands of verified political news have been collected.

Despite the best efforts of cyber security analysts, networked computing assets are routinely compromised, resulting in the loss of intellectual property, the disclosure of state secrets, and major financial damages. Anomaly detection methods are beneficial for detecting new types of attacks and abnormal network activity, but such algorithms can be difficult to understand and trust. Network operators and cyber analysts need fast and scalable tools to help identify suspicious behavior that bypasses automated security systems, but operators do not want another automated tool with algorithms they do not trust. Experts need tools to augment their own domain expertise and to provide a contextual understanding of suspicious behavior to help them make decisions. In this paper we present Situ, a visual analytics system for discovering suspicious behavior in streaming network data. Situ provides a scalable solution that combines anomaly detection with information visualization. The system's visualizations enable operators to identify and investigate the most anomalous events and IP addresses, and the tool provides context to help operators understand why they are anomalous. Finally, operators need tools that can be integrated into their workflow and with their existing tools. This paper describes the Situ platform and its deployment in an operational network setting. We discuss how operators are currently using the tool in a large organization's security operations center and present the results of expert reviews with professionals.

The scatterplot matrix (SPLOM) is a commonly used technique for visualizing multiclass multivariate data. However, multiclass SPLOMs have issues with overdraw (overlapping points), and most existing techniques for alleviating overdraw focus on individual scatterplots with a single class. This paper explores whether animation using flickering points is an effective way to alleviate overdraw in these multiclass SPLOMs. In a user study with 69 participants, we found that users not only performed better at identifying dense regions using animated SPLOMs, but also found them easier to interpret and preferred them to static SPLOMs. These results open up new directions for future work on alleviating overdraw for multiclass SPLOMs, and provide insights for applying animation to alleviate overdraw in other settings.

Effective use of data involving personal or sensitive information often requires different people to have access to personal information, which significantly reduces the personal privacy of those whose data is stored and increases risk of identity theft, data leaks, or social engineering attacks. Our research studies the tradeoffs between privacy and utility of personal information for human decision making. Using a record-linkage scenario, this paper presents a controlled study of how varying degrees of information availability influences the ability to effectively use personal information. We compared the quality of human decision-making using a visual interface that controls the amount of personal information available using visual markup to highlight data discrepancies. With this interface, study participants who viewed only 30% of data content had decision quality similar to those who had full 100% access. The results demonstrate that it is possible to greatly limit the amount of personal information available to human decision makers without negatively affecting utility or human effectiveness. However, the findings also show there is a limit to how much data can be hidden before negatively influencing the quality of judgment in decisions involving person-level data. Despite the reduced accuracy with extreme data hiding, the study demonstrates that with proper interface designs, many correct decisions can be made with even legally de-identified data that is fully masked (74.5% accuracy with fully-masked data compared to 84.1% with full access). Thus, when legal requirements only allow for de-identified data access, use of well-designed interface can significantly improve data utility.

On Reddit, users from tens of thousands of communities create and promote internet content, including pictures, videos, news, memes, and creative writing. However, like most social media feeds, subscribing to a very small subset of available content creates filter bubbles. These bubbles, while created unintentionally, skew perceptions of reality. This phenomena provides an impetus for researchers to design techniques breaking out of filter bubbles. Virtual reality provides opportunities for new environments that contextualize social media among multiple perspectives. We present one solution to the filter bubble problem: Blue Link City, which enables contextualized exploration of Reddit.

Virtual reality often uses motion tracking to incorporate physical hand movements into interaction techniques for selection and manipulation of virtual objects. To increase realism and allow direct hand interaction, real-world physical objects can be aligned with virtual objects to provide tactile feedback and physical grasping. However, unless a physical space is custom configured to match a specific virtual reality experience, the ability to perfectly match the physical and virtual objects is limited. Our research addresses this challenge by studying methods that allow one physical object to be mapped to multiple virtual objects that can exist at different virtual locations in an egocentric reference frame. We study two such techniques: one that introduces a static translational offset between the virtual and physical hand before a reaching action, and one that dynamically interpolates the position of the virtual hand during a reaching motion. We conducted two experiments to assess how the two methods affect reaching effectiveness, comfort, and ability to adapt to the remapping techniques when reaching for objects with different types of mismatches between physical and virtual locations. We also present a case study to demonstrate how the hand remapping techniques could be used in an immersive game application to support realistic hand interaction while optimizing usability. Overall, the translational technique performed better than the interpolated reach technique and was more robust for situations with larger mismatches between virtual and physical objects.

User interfaces for data visualization often consist of two main components: control panels for user interaction and visual representation. A recent trend in visualization is directly embedding user interaction into the visual representations. For example, instead of using control panels to adjust visualization parameters, users can directly adjust basic graphical encodings (e.g., changing distances between points in a scatterplot) to perform similar parameterizations. However, enabling embedded interactions for data visualization requires a strong understanding of how user interactions influence the ability to accurately control and perceive graphical encodings. In this paper, we study the effectiveness of these graphical encodings when serving as the method for interaction. Our user study includes 12 interactive graphical encodings. We discuss the results in terms of task performance and interaction effectiveness metrics.

Limited research has been performed attempting to handle multiuser storytelling environments in virtual reality. As such, a number of questions about handling story progression and maintaining user presence in a multi-user virtual environment have yet to be answered. We created a multi-user virtual reality story experience in which we intend to study a set of guided camera techniques and a set of gaze distractor techniques to determine how best to attract disparate users to the same story. Additionally, we describe our preliminary work and plans to study the effectiveness of these techniques, their effect on user presence, and generally how multiple users feel their actions affect the outcome of a story.

The use of self avatars in virtual reality (VR) can bring users a stronger sense of presence and produce a more compelling experience by providing additional visual feedback during interactions. Avatars also become increasingly more relevant in VR as they provide a user with an identity for social interactions in multi-user settings. However, with current consumer VR setups that include only a head mounted display and hand controllers, implementation of self avatars are generally limited in the ability to mimic actions performed in the real world. Our work explores the idea of simulating a wide range of upper body motions using motion and positional data from only the head and hand motion data. We present a method to differentiate head and hip motions using information from captured motion data and applying corresponding changes to a virtual avatar. We discuss our approach and initial results.

Many researchers have studied methods of effective travel in virtual environments, but little work has considered scene transitions, which may be important for virtual reality experiences like immersive 360 degree movies. In this research, we designed and evaluated three different scene transition techniques in two environments, conducted a pilot study, and collected metrics related to sickness, spatial orientation, and preference. Our preliminary results indicate that faster techniques are generally preferred by gamers and more gradual transitions are preferred by participants with less experience with 3D gaming and virtual reality.

Safety education is important in the construction industry. While research has been done on virtual environments for construction safety education, there is no set method for effectively contextualizing safety information and engaging students. In this research, we study the design of virtual environments to represent construction accident reports provided by the Occupational Health and Safety Administration (OSHA). We looked at different designs to contextualize the report data through space, visuals, and text. Users can explore the environment and interact through immersive virtual reality to learn more about a particular accident.

Traditionally in virtual reality systems, head tracking is used in head-mounted displays (HMDs) to allow users to control viewing using 360-degree head and body rotations. Our research explores interaction considerations that enable semi-natural methods of view control that will work for seated use of virtual reality with HMDs when physically turning all the way around is not ideal, such as when sitting on a couch or at a desk. We investigate the use of amplified head rotations so physically turning in a comfortable range can allow viewing of a 360-degree virtual range. Additionally, to avoid situations where the user's neck is turned in an uncomfortable position for an extended period, we also use redirection during virtual movement to gradually realign the user's head position back to the neutral, straight-ahead position. We ran a controlled experiment to evaluate guided head rotation and amplified head rotation without realignment during movement, and we compared both to traditional one-to-one head-tracked viewing as a baseline for reference. After a navigation task, overall errors on spatial orientation tasks were relatively low with all techniques, but orientation effects, sickness, and preferences varied depending on participants' 3D gaming habits. Using the guided rotation technique, participants who played 3D games performed better, reported higher preference scores, and demonstrated significantly lower sickness results compared to non-gamers.

Many types of virtual reality (VR) systems allow users to use natural, physical head movements to view a 3D environment. In some situations, such as when using systems that lack a fully surrounding display or when opting for convenient low-effort interaction, view control can be enabled through a combination of physical and virtual turns to view the environment, but the reduced realism could potentially interfere with the ability to maintain spatial orientation. One solution to this problem is to amplify head rotations such that smaller physical turns are mapped to larger virtual turns, allowing trainees to view the entire surrounding environment with small head movements. This solution is attractive because it allows semi-natural physical view control rather than requiring complete physical rotations or a fully-surrounding display. However, the effects of amplified head rotations on spatial orientation and many practical tasks are not well understood. In this paper, we present an experiment that evaluates the influence of amplified head rotation on 3D search, spatial orientation, and cybersickness. In the study, we varied the amount of amplification and also varied the type of display used (head-mounted display or surround-screen CAVE) for the VR search task. By evaluating participants first with amplification and then without, we were also able to study training transfer effects. The findings demonstrate the feasibility of using amplified head rotation to view 360 degrees of virtual space, but noticeable problems were identified when using high amplification with a head-mounted display. In addition, participants were able to more easily maintain a sense of spatial orientation when using the CAVE version of the application, which suggests that visibility of the user's body and awareness of the CAVE's physical environment may have contributed to the ability to use the amplification technique while keeping track of orientation.

While the primary goal of visual analytics research is to improve the quality of insights and findings, a substantial amount of research in provenance has focused on the history of changes and advances throughout the analysis process. The term, provenance, has been used in a variety of ways to describe different types of records and histories related to visualization. The existing body of provenance research has grown to a point where the consolidation of design knowledge requires cross-referencing a variety of projects and studies spanning multiple domain areas. We present an organizational framework of the different types of provenance information and purposes for why they are desired in the field of visual analytics. Our organization is intended to serve as a framework to help researchers specify types of provenance and coordinate design knowledge across projects. We also discuss the relationships between these factors and the methods used to capture provenance information. In addition, our organization can be used to guide the selection of evaluation methodology and the comparison of study outcomes in provenance research.

Visual history tools provide visual representations of the workflow during data analysis tasks. While there is an established need for reviewing analytic processes, and many visual history tools provide visualizations to do so, it is not well known how helpful the tools actually are for process recall. Through a controlled experiment, we evaluated how the presence of a visual history aid and varying levels of visual detail affect process memory. Participants conducted an analysis task using a visual text-document analysis tool. We evaluated their memories of the process both immediately after the analysis and then again one week later. Results showed that even visual history views with reduced data-resolution were effective for aiding process memory. Further, even without inclusion of any data in the visual history aids, the visual cues alone from the final workspace were enough to improve memory of the main themes of analyses.

Several researchers have investigated phantom tactile sensation (i.e., the perception of a nonexistent actuator between two real actuators) and apparent tactile motion (i.e., the perception of a moving actuator due to time delays between onsets of multiple actuations). Prior work has focused primarily on determining appropriate Durations of Stimulation (DOS) and Stimulus Onset Asynchronies (SOA) for simple touch gestures, such as a single finger stroke. To expand upon this knowledge, we investigated complex touch gestures involving multiple, simultaneous points of contact, such as a whole hand touching the arm. To implement complex touch gestures, we modified the Tactile Brush algorithm to support rectangular areas of tactile stimulation.

Virtual reality training systems are commonly used in a variety of domains, and it is important to understand how the realism of a training simulation influences training effectiveness. We conducted a controlled experiment to test the effects of display and scenario properties on training effectiveness for a visual scanning task in a simulated urban environment. The experiment varied the levels of field of view and visual complexity during a training phase and then evaluated scanning performance with the simulator's highest levels of fidelity and scene complexity. To assess scanning performance, we measured target detection and adherence to a prescribed strategy. The results show that both field of view and visual complexity significantly affected target detection during training; higher field of view led to better performance and higher visual complexity worsened performance. Additionally, adherence to the prescribed visual scanning strategy during assessment was best when the level of visual complexity during training matched that of the assessment conditions, providing evidence that similar visual complexity was important for learning the technique. The results also demonstrate that task performance during training was not always a sufficient measure of mastery of an instructed technique. That is, if learning a prescribed strategy or skill is the goal of a training exercise, performance in a simulation may not be an appropriate indicator of effectiveness outside of training-evaluation in a more realistic setting may be necessary.

Spatial judgments are important for many real-world tasks in engineering and scientific visualization. While existing research provides evidence that higher levels of display and interaction fidelity in virtual reality systems offer advantages for spatial understanding, few investigations have focused on small-scale spatial judgments or employed experimental tasks similar to those used in real-world applications. After an earlier study that considered a broad analysis of various spatial understanding tasks, we present the results of a follow-up study focusing on small-scale spatial judgments. In this research, we independently controlled field of regard, stereoscopy, and head-tracked rendering to study their effects on the performance of a task involving precise spatial inspections of complex 3D structures. Measuring time and errors, we asked participants to distinguish between structural gaps and intersections between components of 3D models designed to be similar to real underground cave systems. The overall results suggest that the addition of the higher fidelity system features support performance improvements in making small-scale spatial judgments. Through analyses of the effects of individual system components, the experiment shows that participants made significantly fewer errors with either an increased field of regard or with the addition of head-tracked rendering. The results also indicate that participants performed significantly faster when the system provided the combination of stereo and head-tracked rendering.

Visualizations often use spatial representations to aid understanding, but it is unclear what properties of a spatial information presentation are most important to effectively support cognitive processing. This research explores how spatial layout and view control impact learning and investigates the role of persistent visibility when working with large displays. We performed a controlled experiment with a learning activity involving memory and comprehension of a visually represented story. We compared performance between a slideshow-type presentation on a single monitor and a spatially distributed presentation among multiple monitors. We also varied the method of view control (automatic vs. interactive). Additionally, to separate effects due to location or persistent visibility with a spatially distributed layout, we controlled whether all story images could always be seen or if only one image could be viewed at a time. With the distributed layouts, participants maintained better memory of the associated locations where information was presented. However, learning scores were significantly better for the slideshow presentation than for the distributed layout when only one image could be viewed at a time.

Studying what design features are necessary and effective for educational virtual environments (VEs), we focused on two design issues: level of environmental detail and method of navigation. In a controlled experiment, participants studied animal facts distributed among different locations in an immersive VE. Participants viewed the information as either an automated tour through the environment or with full navigational control. The experiment also compared two levels of environmental detail: a sparse environment with only the animal fact cards and a detailed version that also included landmark items and ground textures. The experiment tested memory and understanding of the animal information. Though neither environmental detail nor navigation type significantly affected learning outcomes, the results suggest that manual navigation may have negatively affected the learning activity. Also, learning scores were correlated with both spatial ability and video game usage, suggesting that educational VEs may not be an appropriate presentation method for some learners.

Inaccurate perception of distances is a known problem within virtual environments. We hypothesize that the inclusion of virtual characters within these environments can improve an observer's ability to judge distances and achieve accurate spatial understanding. We have conducted an empirical study using a desktop display of a small scale environment to evaluate the validity of this concept. We investigated whether the presence and quantity of virtual human characters, as well as the naturalness of their locomotion animations, could improve egocentric and exocentric distance estimations. Preliminary results suggest that static or properly animated characters could improve exocentric estimations, and properly animated characters could reduce egocentric distance compression errors.

We propose the use of virtual environments to simulate augmented reality (AR) systems for the purposes of experimentation and usability evaluation. This method allows complete control in the AR environment, providing many advantages over testing with true AR systems. We also discuss some of the limitations to the simulation approach. We have demonstrated the use of such a simulation in a proof of concept experiment controlling the levels of registration error in the AR scenario. In this experiment, we used the simulation method to investigate the effects of registration error on task performance for a generic task involving precise motor control for AR object manipulation. Isolating jitter and latency errors, we provide empirical evidence of the relationship between accurate registration and task performance.