AI Uncertainty Quantification (UQ) has the potential to improve human decision-making beyond AI predictions alone by providing additional probabilistic information to users. The majority of past research on AI and human decision-making has concentrated on model explainability and interpretability, with little focus on understanding the potential impact of UQ on human decision-making. We evaluated the impact on human decision-making for instance-level UQ, calibrated using a strict scoring rule, in two online behavioral experiments. In the first experiment, our results showed that UQ was beneficial for decision-making performance compared to only AI predictions. In the second experiment, we found UQ had generalizable benefits for decision-making across a variety of representations for probabilistic information. These results indicate that implementing high quality, instance-level UQ for AI may improve decision-making with real systems compared to AI predictions alone.

X-ray vision, a technique that allows users to see through walls and other obstacles, is a popular technique for Augmented Reality (AR) and Mixed Reality (MR). In this paper, we demonstrate a dynamic X-ray vision window that is rendered in real-time based on the user’s current position and changes with movement in the physical environment. Moreover, the location and transparency of the window are also dynamically rendered based on the user’s eye gaze. We build this X-ray vision window for a current state-of-the-art MR Head-Mounted Device (HMD) – HoloLens 2 [5] by integrating several different features: scene understanding, eye tracking, and clipping primitive.

Explainable AI provides insights to users into the why for model predictions, offering potential for users to better understand and trust a model, and to recognize and correct AI predictions that are incorrect. Prior research on human and explainable AI interactions has focused on measures such as interpretability, trust, and usability of the explanation. There are mixed findings whether explainable AI can improve actual human decision-making and the ability to identify the problems with the underlying model. Using real datasets, we compare objective human decision accuracy without AI (control), with an AI prediction (no explanation), and AI prediction with explanation. We find providing any kind of AI prediction tends to improve user decision accuracy, but no conclusive evidence that explainable AI has a meaningful impact. Moreover, we observed the strongest predictor for human decision accuracy was AI accuracy and that users were somewhat able to detect when the AI was correct vs. incorrect, but this was not significantly affected by including an explanation. Our results indicate that, at least in some situations, the why information provided in explainable AI may not enhance user decision-making, and further research may be needed to understand how to integrate explainable AI into real systems.

In this paper we provide a critical analysis with metrics that will inform guidelines for designing distributed systems for Collective Situational Understanding (CSU). CSU requires both collective insight-i.e., accurate and deep understanding of a situation derived from uncertain and often sparse data and collective foresight-i.e., the ability to predict what will happen in the future. When it comes to complex scenarios, the need for a distributed CSU naturally emerges, as a single monolithic approach not only is unfeasible: it is also undesirable. We therefore propose a principled, critical analysis of AI techniques that can support specific tasks for CSU to derive guidelines for designing distributed systems for CSU.

We introduce attacklets, a novel approach to model the high dimensional interactions in cyberattacks. Attacklets are implemented using a real-world dataset of cyberattacks from the Verizon Data Breach Investigation Report. Whereas the commonly used attack graphs model the action sequences of attackers for specific exploits, attacklets model general attributes and states of each attack separately. Attacklets may inform the number and types of attributes across a wide range of cyberattacks. These structural properties can then be used in machine learning models to classify and predict future cyberattacks.

Massive amounts of data are being collected, stored, and analyzed for various business and marketing purposes. While such data analysis is critical for many applications, it could also violate the privacy of individuals. This paper describes the issues involved in designing a privacy aware data management framework for collecting, storing, and analyzing the data. We also discuss behavioral aspects of data sharing as well as aspects of a formal framework based on rewriting rules that encompasses the privacy aware data management framework.

Controlled natural language (CNL) has great potential to support human-machine interaction (HMI) because it provides an information representation that is both human readable and machine processable. We investigated the effectiveness of a CNL-based conversational interface for HMI in a behavioral experiment called simple human experiment regarding locally observed collective knowledge (SHERLOCK). In SHERLOCK, individuals acted in groups to discover and report information to the machine using natural language (NL), which the machine then processed into CNL. The machine fused responses from different users to form a common operating picture, a dashboard showing the level of agreement for distinct information. To obtain information to add to this dashboard, users explored the real world in a simulated crowdsourced sensing scenario. This scenario represented a simplified controlled analog for tactical intelligence (i.e., direct intelligence of the environment), which is key for rapidly planning military, law enforcement, and emergency operations. Overall, despite close to zero training, 74% of the users inputted NL that was machine interpretable and addressed the assigned tasks. An experimental manipulation aimed to increase user-machine interaction, however, did not improve performance as hypothesized. Nevertheless, results indicate that the conversational interface may be effective in assisting humans with collection and fusion of information in a crowdsourcing context.

We report on three experiments that investigate the efficacy of a new type of interface called a virtual verbal display (VVD) for nonvisual learning and navigation of virtual environments (VEs). Although verbal information has been studied for route-guidance, little is known about the use of context-sensitive, speech-based displays (e.g., the VVD) for supporting free exploration and wayfinding behavior. During training, participants used the VVD (Experiments I and II) or a visual display (Experiment III) to search the VEs and find four hidden target locations. At test, all participants performed a route-finding task in the corresponding real environment, navigating with vision (Experiments I and III) or from verbal descriptions (Experiment II). Training performance between virtual display modes was comparable, but wayfinding in the real environment was worse after VVD learning than visual learning, regardless of the testing modality. Our results support the efficacy of the VVD for searching computer-based environments but indicate a difference in the cognitive maps built up between verbal and visual learning, perhaps due to lack of physical movement in the VVD.