Exploring 3D surface data often involves navigating complex menus, creating challenges for both experts and new users who may struggle with overloaded interfaces. To address this, we introduce VocalVis , an open-source prototype that leverages real-time voice processing to simplify interaction and analysis of 3D surface data. By capitalizing on advancements in natural language processing, VocalVis enables users to interact with digital content through voice commands, addressing challenges associated with traditional navigation methods. Our study involves both data scientist novices and domain experts, offering insights into how voice interaction can streamline data exploration. The findings demonstrate the potential of voice as a powerful supplementary tool for visually analyzing complex datasets. This innovative approach opens up new possibilities for intuitive, accessible data exploration, making it convenient for a broader range of users to engage with 3D scientific data.

Using speech for hands-free interaction in Virtual Reality (VR) is gaining popularity, supported by advances in natural language processing that enable accurate speech-to-text transcription and intent inference. Prior work has shown that voice-based navigation is effective when hand use is restricted or occupied, particularly when visually salient points in the scene, known as landmarks, can be identified and articulated. However, in many virtual environments, such landmarks may be absent or not easily recognizable, limiting the applicability of voice commands. To address this, we propose three landmark-free, user-centered coordinate systems to support speech-based locomotion in VR: Cartesian, cylindrical, and spherical. Each system uses a distinct encoding and interaction style. The Cartesian system employs a simple three-digit code and requires low cognitive effort. The spherical system prioritizes precision but demands higher mental effort. The cylindrical system combines elements of both, offering a balance between usability and accuracy. We evaluated these systems in a quantitative user study with 24 participants, all with backgrounds in information technology. The study assessed the systems’ effectiveness for object positioning and navigation, comparing them to teleportation, the standard non-voice locomotion method in VR. Results show that the Cartesian system enables faster and more intuitive navigation compared to the spherical system. Distinct usage patterns were observed, with users predominantly focusing on the central field of view during navigation. Our findings indicate that user-centered coordinate systems are practical to implement and present a viable alternative for speech-driven, hands-free navigation in VR.

Generating visualizations from natural language queries is a useful extension to visualization libraries such as Vega-Lite. The goal of the NL2VIS task is to generate a valid Vega-Lite specification from a data frame and a natural language query as input, which can then be rendered as a visualization. To enable real-time interaction with the data, small model sizes and fast inferences are required. Previous work has introduced custom neural network solutions with custom visualization specifications and has not systematically tested pre-trained LMs to solve this problem. In this work, we opt for a more generic approach that (i) evaluates pre-trained LMs of different sizes and (ii) uses string encodings of data frames and visualization specifications instead of custom specifications. In our experiments, we show that these representations, in combination with pre-trained LMs, scale better than current state-of-the-art models. In addition, the small and base versions of the T5 architecture achieve real-time interaction, while LLMs far exceed latency thresholds suitable for visual exploration tasks. In summary, our models generate visualization specifications in real-time on a CPU and establish a new state of the art on the NL2VIS benchmark nvBench.

Uncertainty cues are inherent in natural human interaction, as they signal to communication partners how much they can rely on conveyed information. Humans subconsciously provide such signals both verbally (e.g., through expressions such as “maybe’’ or “I think’’) and non-verbally (e.g., by diverting their gaze). In contrast, artificial intelligence (AI)-based services and machine learning (ML) models such as ChatGPT usually do not disclose the reliability of answers to their users.

In various situations, such as clinical environments with sterile conditions or when hands are occupied with multiple devices, traditional methods of navigation and scene adjustment are impractical or even impossible. We explore a new solution by using voice control to facilitate interaction in virtual worlds to avoid the use of additional controllers. Therefore, we investigate three scenarios: Object Orientation, Visualization Customization, and Analytical Tasks and evaluate whether natural language interaction is possible and promising in each of these scenarios. In our quantitative user study participants were able to control virtual environments effortlessly using verbal instructions. This resulted in rapid orientation adjustments, adaptive visual aids, and accurate data analysis. In addition, user satisfaction and usability surveys showed consistently high levels of acceptance and ease of use. In conclusion, our study shows that the use of natural language can be a promising alternative for the improvement of user interaction in virtual environments. It enables intuitive interactions in virtual spaces, especially in situations where traditional controls have limitations. • Presenting an alternative, hands-free interaction system using real-time natural language processing. • Evaluation of VUIs in VR, spanning three primary categories: Orientation, Customization, and Analysis. • Exploring whether voice interaction can serve as a feasible mode of communication with visualizations. • Offering an open-source project to enhance accessibility for voice-based applications.

As locomotion is an important factor in improving Virtual Reality (VR) immersion and usability, research in this area has been and continues to be a crucial aspect for the success of VR applications. In recent years, a variety of techniques have been developed and evaluated, ranging from abstract control, vehicle, and teleportation techniques to more realistic techniques such as motion, gestures, and gaze. However, when it comes to hands-free scenarios, for example to increase the overall accessibility of an application or in medical scenarios under sterile conditions, most of the announced techniques cannot be applied. This is where the use of speech as an intuitive means of navigation comes in handy. As systems become more capable of understanding and producing speech, voice interfaces become a valuable alternative for input on all types of devices. This takes the quality of hands-free interaction to a new level. However, intuitive user-assisted speech interaction is difficult to realize due to semantic ambiguities in natural language utterances as well as the high real-time requirements of these systems. In this paper, we investigate steering-based locomotion and selection-based locomotion using three speech-based, hands-free methods and compare them with leaning as an established alternative. Our results show that landmark-based locomotion is a convenient, fast, and intuitive way to move between locations in a VR scene. Furthermore, we show that in scenarios where landmarks are not available, number grid-based navigation is a successful solution. Based on this, we conclude that speech is a suitable alternative in hands-free scenar-ios, and exciting ideas are emerging for future work focused on developing hands-free ad hoc navigation systems for scenes where landmarks do not exist or are difficult to articulate or recognize.

Henrik Voigt, Ozge Alacam, Monique Meuschke, Kai Lawonn, Sina Zarrieß. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

While certain types of instructions can be compactly expressed via images, there are situations where one might want to verbalise them, for example when directing someone.We investigate the task of Instruction Generation from Before/After Image Pairs which is to derive from images an instruction for effecting the implied change.For this, we make use of prior work on instruction following in a visual environment.We take an existing dataset, the BLOCKS data collected by Bisk et al. (2016) and investigate whether it is suitable for training an instruction generator as well.We find that it is, and investigate several simple baselines, taking these from the related task of image captioning.Through a series of experiments that simplify the task (by making image processing easier or completely side-stepping it; and by creating template-based targeted instructions), we investigate areas for improvement.We find that captioning models get some way towards solving the task, but have some difficulty with it, and future improvements must lie in the way the change is detected in the instruction.