We present ScaleFree, a GPU-accelerated adaptive Kernel Density Estimation (KDE) algorithm for scalable, interactive multiscale point cloud exploration. With this technique, we cater to the massive datasets and complex multiscale structures in advanced scientific computing, such as cosmological simulations with billions of particles. Effective exploration of such data requires a full 3D understanding of spatial structures, a capability for which immersive environments such as VR are particularly well suited. However, simultaneously supporting global multiscale context and fine-grained local detail remains a significant challenge. A key difficulty lies in dynamically generating continuous density fields from point clouds to facilitate the seamless scale transitions: while KDE is widely used, precomputed fields restrict the accuracy of interaction and omit fine-scale structures, while dynamic computation is often too costly for real-time VR interaction. We address this challenge by leveraging GPU acceleration with k-d-tree-based spatial queries and parallel reduction within a thread group for on-the-fly density estimation. With this approach, we can recalculate scalar fields dynamically as users shift their focus across scales. We demonstrate the benefits of adaptive density estimation through two data exploration tasks: adaptive selection and progressive navigation. Through performance experiments, we demonstrate that ScaleFree with GPU-parallel implementation achieves orders-of-magnitude speedups over sequential and multi-core CPU baselines. In a controlled experiment, we further confirm that our adaptive selection technique improves accuracy and efficiency in multiscale selection tasks.

Cross Reality (CR) is a new emerging field based on the current developments in Mixed Reality hardware, especially supported by the broad market penetration of video-based see-through Head-Mounted Displays. It refers to applications that span across different stages (real, Augmented Reality, Augmented Virtuality, Virtual Reality) of the reality-virtuality continuum, where users are interconnected between different stages and/or are able to transition between these stages. This publication follows the concept of other grand challenges publications and reflects the discussion of various researchers invested in CR. After an initial discussion at the 1stJoint Workshop on Cross Reality at IEEE ISMAR 2023, six topic groups have been identified, leading to 22 challenges, which were discussed in groups over the period of multiple months. The discussion of these challenges should act as a road map for future research in the area of CR.

High latency is often introduced due to limited computational capabilities and high hardware demands. It has proven to significantly impair user performance in target selection, a fundamental interaction task. Existing research has established that latency negatively impacts selection times and success rates in 2D interactive systems; however, the underlying behavioral mechanisms remain unclear. This article investigates the effects of latency on selection times, success rates, and endpoint distributions in Virtual Reality (VR) with controller-based raycasting and bare-hand direct touch-the two most common selection methods. Our results from a user study (N = 31) revealed distinct patterns between the two methods, leading to two novel mathematical models that account for latency, target width, and movement amplitude. These two models were validated via a new dataset collected from a second user study (N = 16) and were demonstrated to outperform the existing models. Our findings provide actionable recommendations to mitigate the negative impacts of latency and improve user experience in VR interface design.

Digital humanity visualization, as an innovative trend that combines historical and cultural studies with visualization, aims to provide the public with intuitive and engaging cultural exploration experiences. However, it is still challenging in this field due to the intrinsic complex and cumbersome textual data, such as how to intuitively and comprehensively present complex relationship networks and spatiotemporal evolution among the data. In this paper, we take the Jinling-related dataset as an example to design a composite visual storytelling tool that utilizes a narrative framework that smoothly changes between macro and micro perspectives. Simultaneously, AI-generated intuitive images are employed to represent the textual data, collectively narrating an engaging cultural story of Jinling. Through user studies, we validate the effectiveness and usability of the tool and demonstrate that the tool possesses excellent storytelling capabilities to improve users' cultural experiences.

We developed an AI-assisted interactive visual system that allow users to explore Journey to the West’s magical artifacts. Although prior studies have explored its narrative and symbolism, the embedded technological metaphors and speculative interaction concepts in the novel remain underexplored. Leveraging ChatGPT-4 for deep semantic analysis, we extracted mappings between principal characters and their signature artifacts and embedded interaction patterns reflecting mythical affordances. These affordances guided the creation of a progressive, narrative-driven interactive visual system built on large displays. In a user study with 20 participants, the system achieved high usability scores and strong preference with facial-matching and multimodal artifact exploration feature. Our work provides a framework for reinterpreting and showcasing mythological narratives. The Appendix is available at our GitHub repository.

Prior work has identified the shortcomings of color‐only encodings for maps with many categories, yet systematic comparisons of hybrid text–color strategies remain scarce. We therefore ran an 80‐participant crowdsourced study on choropleth maps with 8–13 categories—approaching the 10‐hue perceptual limit—to compare four annotation designs (Legend‐Aside, Label‐Fill, Label‐Fit, Colored Label‐Fill) across Count, Identify, Compare, and Rank tasks. Results show that the Label‐Fit Map—with a single, large in‐situ label—yields the highest accuracy and speed and ranks first in readability; Legend‐Aside excels in simple counting and side‐by‐side comparisons. These findings deliver clear, task‐specific guidelines for enhancing multiclass map readability and efficiency, informing the design of more effective map visualizations. All supplementary materials are available at our GitHub repository.

We introduce an interactive music system powered by AI-generated content (AIGC) that enables users to engage with music through multimodal interactions involving touch, sound, and spatial immersion. Motivated by the desire to enhance engagement and emotional connection with music, our system enables users to co-create and interact with musical content. Users upload a song and a descriptive text prompt, from which the system generates 3D visuals. During playback, users can embed their own audio inputs and trigger responsive visual effects such as color-driven point clouds using tangible controls. To explore how spatial scale and embodiment shape user experience, we implement the system across three increasing spatial scales and embodiments: (1) a handheld AR music box, (2) a table-sized stage box, and (3) a fully immersive VR environment. Through a user study, we investigate how different levels of immersion and interaction influence user engagement, emotional response, and sense of presence. Our findings demonstrate the potential of combining AIGC with embodied interaction to enrich creative expression and enhance immersive musical experiences.

This study examines the effects of two visual guidance techniques, Visual Enhancement and Visual Suppression, on user perception of contextual information in video content. Visual Enhancement introduces explicit visual cues to highlight target content, whereas Visual Suppression attenuates non-target elements, for example, by reducing their brightness. Both approaches aim to isolate specific objects from the background, directing attention to critical information within complex, dynamic scenes. Despite their growing usage, the relative effectiveness of these approaches in guiding attention and their impact on peripheral context awareness remain underexplored. To address this gap, we conducted a controlled user study with 27 participants. The results indicate that Visual Enhancement, through the addition of salient cues, more effectively directs user attention to target information than Visual Suppression. Our findings advance understanding of visual attention in dynamic environments and offer implications for designing visual guidance strategies.

Human-swarm interaction (HSI) is critical for scalable control of UAV swarm systems. Traditional interfaces struggle with generalization and user workload, especially in immersive environments. Hence, we present ChatHSI, a framework leveraging large language models (LLMs) for swarm task planning. ChatHSI integrates prompt engineering, action validation, and a human-in-the-loop mechanism to improve planning feasibility and executability. We implement ChatHSI in an immersive simulation to improve users’ spatial and situational awareness. Our method shows improved task efficiency, reduced workload, and higher usability in user studies. Ablation study proves the effectiveness of prompt context and action validation. The results show the feasibility of LLM-driven interaction for immersive swarm control and point toward adaptive, intuitive, and scalable HSI systems.

We present PuzzleSorter, a certainty-aware visual analytics system for cultural relic fragment restoration. Restoring cultural objects from broken fragments is a fundamental task in geometry and archaeology. Prior research proposes automatic models to classify fragments by types and assemble matched pairs successively. However, eroded fragments lead to erroneous results, posing two challenges for restorers to correct: (1) numerous fragments conceal errors within an overwhelming number of object appearances, and (2) the unknown difficulty of restoration hinders correction strategy development. To address these challenges, PuzzleSorter provides multi-criteria analysis that helps users identify certainties of current solutions and alternatives at the type, object, and fragment levels. Moreover, our system visualizes these certainties through a relation graph, which implies alternative assembly solutions with geometric context and indicates correction difficulties through neighbor proximity, number of neighbors, and path length. We demonstrate the feasibility and utility of our system through two case studies and expert interviews.

We propose and study a novel cross-reality environment that seamlessly integrates a monoscopic 2D surface (an interactive screen with touch and pen input) with a stereoscopic 3D space (an augmented reality HMD) to jointly host spatial data visualizations. This innovative approach combines the best of two conventional methods of displaying and manipulating spatial 3D data, enabling users to fluidly explore diverse visual forms using tailored interaction techniques. Providing such effective 3D data exploration techniques is pivotal for conveying its intricate spatial structures-often at multiple spatial or semantic scales-across various application domains and requiring diverse visual representations for effective visualization. To understand user reactions to our new environment, we began with an elicitation user study, in which we captured their responses and interactions. We observed that users adapted their interaction approaches based on perceived visual representations, with natural transitions in spatial awareness and actions while navigating across the physical surface. Our findings then informed the development of a design space for spatial data exploration in cross-reality. We thus developed cross-reality environments tailored to three distinct domains: for 3D molecular structure data, for 3D point cloud data, and for 3D anatomical data. In particular, we designed interaction techniques that account for the inherent features of interactions in both spaces, facilitating various forms of interaction, including mid-air gestures, touch interactions, pen interactions, and combinations thereof, to enhance the users' sense of presence and engagement. We assessed the usability of our environment with biologists, focusing on its use for domain research. In addition, we evaluated our interaction transition designs with virtual and mixed-reality experts to gather further insights. As a result, we provide our design suggestions for the cross-reality environment, emphasizing the interaction with diverse visual representations and seamless interaction transitions between 2D and 3D spaces.

Tangible interfaces in mixed reality (MR) environments allow for intuitive data interactions. Tangible cubes, with their rich interaction affordances, high maneuverability, and stable structure, are particularly well-suited for exploring multi-dimensional data types. However, the design potential of these cubes is underexplored. This study introduces a design space for tangible cubes in MR, focusing on interaction space, visualization space, sizes, and multiplicity. Using spatio-temporal data, we explored the interaction affordances of these cubes in a workshop (N=24). We identified unique interactions like rotating, tapping, and stacking, which are linked to augmented reality (AR) visualization commands. Integrating user-identified interactions, we created a design space for tangible-cube interactions and visualization. A prototype visualizing global health spending with small cubes was developed and evaluated, supporting both individual and combined cube manipulation. This research enhances our grasp of tangible interaction in MR, offering insights for future design and application in diverse data contexts.

Foot-based input can serve as a supplementary or alternative approach to text entry in virtual reality (VR). This work explores the feasibility and design of foot-based techniques that are hands-free. We first conducted a preliminary study to assess foot-based text entry in standing and seated positions with tap and swipe input approaches. The findings showed that foot-based text input was feasible, with the possibility for performance and usability improvements. We then developed three foot-based techniques, including two tap-based techniques (FeetSymTap and FeetAsymTap) and one swipe-based technique (FeetGestureTap), and evaluated their performance via another user study. The results show that the two tap-based techniques supported entry rates of 11.12 WPM and 10.80 WPM, while the swipe-based technique led to 9.16 WPM. Our findings provide a solid foundation for the future design and implementation of foot-based text entry in VR and have the potential to be extended to MR and AR.

With the exponential growth of digital information, ensuring text security, a fundamental component of information security, becomes increasingly paramount. While authentication remains a primary focus for data access control and protection, the rich sensor ecosystem and immersive experiences of virtual reality (VR) environments introduce new privacy risks, particularly with inconspicuous sensors like motion and location sensors. In this context, protecting the security of text entered by users poses a unique challenge. This paper explores the feasibility of enhancing text security by introducing variability in virtual input tools during typing processes. Specifically, we investigate the impact of introducing successive and random intermittent variations to the virtual ray (start point and direction) with controller-based raycasting techniques on text security and typing experience. The results demonstrate that introducing variability in virtual ray effectively protects regular text and passwords. Random intermittent introducing variability balances security and user experience for regular text. These findings provide insights into enhancing text security beyond authentication and defending against the potential risks in VR environments.

In team sports like basketball, understanding and executing tactics—coordinated plans of movements among players—are crucial yet complex, requiring extensive practice. These tactics require players to develop a keen sense of spatial and situational awareness. Traditional coaching methods, which mainly rely on basketball tactic boards and video instruction, often fail to bridge the gap between theoretical learning and the real-world application of tactics, due to shifts in view perspectives and a lack of direct experience with tactical scenarios. To address this challenge, we introduce VisCourt, a Mixed Reality (MR) tactic training system, in collaboration with a professional basketball team. To set up the MR training environment, we employed semi-automatic methods to simulate realistic 3D tactical scenarios and iteratively designed visual in-situ guidance. This approach enables full-body engagement in interactive training sessions on an actual basketball court and provides immediate feedback, significantly enhancing the learning experience. A user study with athletes and enthusiasts shows the effectiveness and satisfaction with VisCourt in basketball training and offers insights for the design of future SportsXR training systems.

Museum artifacts are the main way for visitors to experience and learn about cultural heritage. Augmented reality (AR) allows for high interactivity and is increasingly applied in museums to improve tourists’ experience and learning. It also supports the extension of museum experience to outside of the physical museum space, contributing to the visiting trajectory and takeaway experience. In this paper, we present our design of two tangible AR interfaces for cultural artifacts: Postcard AR and CubeMuseum AR, followed by three user studies that evaluate and optimize the design. In Study 1, we conducted a within-subjects study (N = 24) that compares the two AR interfaces with a baseline condition (Leaflet). Our results demonstrate the positive effects of tangible AR interfaces on users’ motivation and engagement in learning cultural heritage. In Study 2, we further explored how to optimize CubeMuseum AR by adopting a user-centered design approach. Through the analysis of expert interviews (N = 7) and an online survey (N = 207), the results specify a series of requirements and design guidelines for tangible AR interfaces to be used as a learning tool and a hybrid gift. Based on the findings, the design of the CubeMuseum AR was optimized and evaluated in Study 3. A between-subjects user study was conducted (N = 32) to compare the optimized design with the initial design. The results verified the positive effects of gamified tangible AR interfaces on users’ motivation, engagement, and performance in learning cultural heritage. We present our design and evaluation results, and discuss the implications of designing tangible AR interfaces for cultural heritage learning and museum gifting.

Object selection is essential in virtual reality (VR) head-mounted displays (HMDs). Prior work mainly focuses on enhancing and evaluating techniques for selecting a single object in VR, leaving a gap in the techniques for multi-object selection, a more complex but common selection scenario. To enable multi-object selection, the interaction technique should support group selection in addition to the default pointing selection mode for acquiring a single target. This composite interaction could be particularly challenging when using freehand gestural input. In this work, we present an empirical comparison of six freehand techniques, which are comprised of three mode-switching gestures (Finger Segment, Multi-Finger, and Wrist Orientation) and two group selection techniques (Cone-casting Selection and Crossing Selection) derived from prior work. Our results demonstrate the performance, user experience, and preference of each technique. The findings derive three design implications that can guide the design of freehand techniques for multi-object selection in VR HMDs.

We conduct two in-lab experiments (N = 93) to evaluate the effectiveness of Gantt charts, extended Gantt charts, and stringline charts for visualizing fixed-order event sequence data. We first formulate five types of event sequences and define three types of sequence elements: point events, interval events, and the temporal gaps between them. Our two experiments focus on event sequences with a pre-defined, fixed order and measure task error rates and completion time. The first experiment shows single sequences and assesses the three charts' performance in comparing event duration or gap. The second experiment shows multiple sequences and evaluates how well the charts reveal temporal patterns. The results suggest that when visualizing single fixed-order event sequences, 1) Gantt and extended Gantt charts lead to comparable error rates in the duration-comparing task; 2) Gantt charts exhibit either shorter or equal completion time than extended Gantt charts; 3) both Gantt and extended Gantt charts demonstrate shorter completion times than stringline charts; 4) however, stringline charts outperform the other two charts with fewer errors in the comparing task when event type counts are high. Additionally, when visualizing multiple point-based fixed-order event sequences, stringline charts require less time than Gantt charts for people to find temporal patterns. Based on these findings, we discuss design opportunities for visualizing fixed-order event sequences and discuss future avenues for optimizing these charts.

Steering is a fundamental task in interactive Virtual Reality (VR) systems. Prior work has demonstrated that movement direction can significantly influence user behavior in the steering task, and different interactive environments (VEs) can lead to various behavioral patterns, such as tablets and PCs. However, its impact on VR environments remains unexplored. Given the widespread use of steering tasks in VEs, including menu adjustment and object manipulation, this work seeks to understand and model the directional effect with a focus on barehand interaction, which is typical in VEs. This paper presents the results of two studies. The first study was conducted to collect behavioral data with four categories: movement time, average movement speed, success rate, and reenter times. According to the results, we examined the effect of movement direction and built the SθModel. We then empirically evaluated the model through the data collected from the first study. The results proved that our proposed model achieved the best performance across all the metrics (r2 > 0.95), with more than 15% improvement over the original Steering Law in terms of prediction accuracy. Next, we further validated the SθModel by another study with the change of device and steering direction. Consistent with previous assessments, the model continues to exhibit optimal performance in both predicting movement time and speed. Finally, based on the results, we formulated design recommendations for steering tasks in VEs to enhance user experience and interaction efficiency.

We propose three novel spatial data selection techniques for particle data in VR visualization environments. They are designed to be target- and context-aware and be suitable for a wide range of data features and complex scenarios. Each technique is designed to be adjusted to particular selection intents: the selection of consecutive dense regions, the selection of filament-like structures, and the selection of clusters-with all of them facilitating post-selection threshold adjustment. These techniques allow users to precisely select those regions of space for further exploration-with simple and approximate 3D pointing, brushing, or drawing input-using flexible point- or path-based input and without being limited by 3D occlusions, non-homogeneous feature density, or complex data shapes. These new techniques are evaluated in a controlled experiment and compared with the Baseline method, a region-based 3D painting selection. Our results indicate that our techniques are effective in handling a wide range of scenarios and allow users to select data based on their comprehension of crucial features. Furthermore, we analyze the attributes, requirements, and strategies of our spatial selection methods and compare them with existing state-of-the-art selection methods to handle diverse data features and situations. Based on this analysis we provide guidelines for choosing the most suitable 3D spatial selection techniques based on the interaction environment, the given data characteristics, or the need for interactive post-selection threshold adjustment.

Grid visualizations are widely used in many applications to visually explain a set of data and their proximity relationships. However, existing layout methods face difficulties when dealing with the inherent cluster structures within the data. To address this issue, we propose a cluster-aware grid layout method that aims to better preserve cluster structures by simultaneously considering proximity, compactness, and convexity in the optimization process. Our method utilizes a hybrid optimization strategy that consists of two phases. The global phase aims to balance proximity and compactness within each cluster, while the local phase ensures the convexity of cluster shapes. We evaluate the proposed grid layout method through a series of quantitative experiments and two use cases, demonstrating its effectiveness in preserving cluster structures and facilitating analysis tasks.

Target selection is a fundamental task in interactive Augmented Reality (AR) systems. Predicting the intended target of selection in such systems can provide users with a smooth, low-friction interaction experience. Our work aims to predict gaze-based target selection in AR headsets with eye and head endpoint distributions, which describe the probability distribution of eye and head 3D orientation when a user triggers a selection input. We first conducted a user study to collect users’ eye and head behavior in a gaze-based pointing selection task with two confirmation mechanisms (air tap and blinking). Based on the study results, we then built two models: a unimodal model using only eye endpoints and a multimodal model using both eye and head endpoints. Results from a second user study showed that the pointing accuracy is improved by approximately 32% after integrating our models into gaze-based selection techniques.

The support for multi-focus data exploration is vital in collaborative visualization. In these scenarios, which often involve multiple devices and large displays, users may focus on specific information on their individual screens while also sharing contextual views with others. While many visualization techniques developed for single-user applications can be adapted for use in collaborative settings, little research has been done on how to design adaptive versions of these techniques or how they may impact collaborative tasks involving large datasets. In this work, we perform a comparative study of three collaborative visualization techniques (Zooming, Interactive lenses and Overview+Detail) on large displays in three map-based visualization tasks (Exploration, Comparison and Spatial Memorizing). These three collaborative techniques draw on three different classical visualization techniques in a single-user setting. Our results show that these techniques have different impacts on users’ task performance and preferences. The collaborative Overview+Detail technique benefits users most in supporting Spatial Memorizing. Closely coupled groups prefer collaborative Zooming in Target Exploration. Based on these results, we further discuss the design of collaborative visualization techniques and propose suggestions for adapting classical single-user visualization techniques to a collaborative setting.

In recent years, with growing health consciousness, at-home workout has become increasingly popular for its convenience and safety. Most people choose to follow video guidance during exercising. However, our preliminary study revealed that fitness-minded people face challenges when watching exercise videos on handheld devices or fixed monitors, such as limited movement comprehension due to static camera angles and insufficient feedback. To address these issues, we reviewed popular workout videos, identified user requirements, and came up with an augmented reality (AR) solution. Following a user-centered iterative design process, we proposed a design space of AR visual cues for workouts and implemented an AR-based application. Specifically, we captured users’ exercise performance with pose-tracking technology and provided feedback via AR visual cues. Two user experiments showed that incorporating AR visual cues could improve movement comprehension and enable users to adjust their movements based on real-time feedback. Finally, we presented several suggestions to inspire future design and apply AR visual cues to sports training.

In the modern era of hectic lifestyles, individuals often struggle to take a momentary pause from their daily routines to connect with their own physiological changes. Additionally, many people face challenges in recognizing and effectively expressing their emotions, resulting in a lack of self-awareness. This lack of emotional understanding can hinder personal growth and well-being. To tackle this issue, we present a novel visualization approach that combines interactive devices to provide a more intuitive understanding of emotions. Our approach aims to bridge the gap between individuals and their emotional states by translating abstract emotions into visually perceivable physical forms. By leveraging audio-visual equipment, our system creates an environment where individuals can actively engage with and experience their emotions through interactive devices. Through the integration of synesthesia design principles, our visualization approach enables individuals to gain a deeper understanding of their feelings and represent their emotional changes in a more tangible and expressive manner. By facilitating a multisensory experience, individuals can establish a stronger connection with their emotions, promoting self-awareness and emotional well-being. This paper showcases the innovative design and implementation of our visualization, highlighting its potential to empower individuals in understanding and expressing their emotions.

This paper looks at how mixed reality can be used for the improvement and enhancement of Chinese acupuncture practice through the introduction of an acupuncture training simulator. A prototype system developed for our study allows practitioners to insert virtual needles using their bare hands into a full-scale 3D representation of the human body with labelled acupuncture points. This provides them with a safe and natural environment to develop their acupuncture skills simulating the actual physical process of acupuncture. It also helps them to develop their muscle memory for acupuncture and better develops their memory of acupuncture points through a more immersive learning experience. We describe some of the design decisions and technical challenges overcome in the development of our system. We also present the results of a comparative user evaluation with potential users aimed at assessing the viability of such a mixed reality system being used as part of their training and development. The results of our evaluation reveal the training system outperformed in the enhancement of spatial understanding as well as improved learning and dexterity in acupuncture practice. These results go some way to demonstrating the potential of mixed reality for improving practice in therapeutic medicine.

Glyph-based visualization achieves an impressive graphic design when associated with comprehensive visual metaphors, which help audiences effectively grasp the conveyed information through revealing data semantics. However, creating such metaphoric glyph-based visualization (MGV) is not an easy task, as it requires not only a deep understanding of data but also professional design skills. This paper proposes MetaGlyph, an automatic system for generating MGVs from a spreadsheet. To develop MetaGlyph, we first conduct a qualitative analysis to understand the design of current MGVs from the perspectives of metaphor embodiment and glyph design. Based on the results, we introduce a novel framework for generating MGVs by metaphoric image selection and an MGV construction. Specifically, MetaGlyph automatically selects metaphors with corresponding images from online resources based on the input data semantics. We then integrate a Monte Carlo tree search algorithm that explores the design of an MGV by associating visual elements with data dimensions given the data importance, semantic relevance, and glyph non-overlap. The system also provides editing feedback that allows users to customize the MGVs according to their design preferences. We demonstrate the use of MetaGlyph through a set of examples, one usage scenario, and validate its effectiveness through a series of expert interviews.

Immersive environments have become increasingly popular for visualizing and exploring large-scale, complex scientific data because of their key features: immersion, engagement, and awareness. Virtual reality offers numerous new interaction possibilities, including tactile and tangible interactions, gestures, and voice commands. However, it is crucial to determine the most effective combination of these techniques for a more natural interaction experience. In this paper, we present MEinVR, a novel multimodal interaction technique for exploring 3D molecular data in virtual reality. MEinVR combines VR controller and voice input to provide a more intuitive way for users to manip- ulate data in immersive environments. By using the VR controller to select locations and regions of interest and voice commands to perform tasks, users can efficiently perform complex data exploration tasks. Our findings provide suggestions for the design of multimodal interaction techniques in 3D data exploration in virtual reality.

Videos are well-received methods for storytellers to communicate various narratives. To further engage viewers, we introduce a novel visual medium where data visualizations are embedded into videos to present data insights. However, creating such data-driven videos requires professional video editing skills, data visualization knowledge, and even design talents. To ease the difficulty, we propose an optimization method and develop SmartShots, which facilitates the automatic integration of in-video visualizations. For its development, we first collaborated with experts from different backgrounds, including information visualization, design, and video production. Our discussions led to a design space that summarizes crucial design considerations along three dimensions: visualization, embedded layout, and rhythm. Based on that, we formulated an optimization problem that aims to address two challenges: (1) embedding visualizations while considering both contextual relevance and aesthetic principles and (2) generating videos by assembling multi-media materials. We show how SmartShots solves this optimization problem and demonstrate its usage in three cases. Finally, we report the results of semi-structured interviews with experts and amateur users on the usability of SmartShots.

Video moderation, which refers to remove deviant or explicit content from e-commerce livestreams, has become prevalent owing to social and engaging features. However, this task is tedious and time consuming due to the difficulties associated with watching and reviewing multimodal video content, including video frames and audio clips. To ensure effective video moderation, we propose VideoModerator, a risk-aware framework that seamlessly integrates human knowledge with machine insights. This framework incorporates a set of advanced machine learning models to extract the risk-aware features from multimodal video content and discover potentially deviant videos. Moreover, this framework introduces an interactive visualization interface with three views, namely, a video view, a frame view, and an audio view. In the video view, we adopt a segmented timeline and highlight high-risk periods that may contain deviant information. In the frame view, we present a novel visual summarization method that combines risk-aware features and video context to enable quick video navigation. In the audio view, we employ a storyline-based design to provide a multi-faceted overview which can be used to explore audio content. Furthermore, we report the usage of VideoModerator through a case scenario and conduct experiments and a controlled user study to validate its effectiveness.

The efficiency of warehouses is vital to e-commerce. Fast order processing at the warehouses ensures timely deliveries and improves customer satisfaction. However, monitoring, analyzing, and manipulating order processing in the warehouses in real time are challenging for traditional methods due to the sheer volume of incoming orders, the fuzzy definition of delayed order patterns, and the complex decision-making of order handling priorities. In this paper, we adopt a data-driven approach and propose OrderMonitor, a visual analytics system that assists warehouse managers in analyzing and improving order processing efficiency in real time based on streaming warehouse event data. Specifically, the order processing pipeline is visualized with a novel pipeline design based on the sedimentation metaphor to facilitate real-time order monitoring and suggest potentially abnormal orders. We also design a novel visualization that depicts order timelines based on the Gantt charts and Marey's graphs. Such a visualization helps the managers gain insights into the performance of order processing and find major blockers for delayed orders. Furthermore, an evaluating view is provided to assist users in inspecting order details and assigning priorities to improve the processing performance. The effectiveness of OrderMonitor is evaluated with two case studies on a real-world warehouse dataset.

Circular glyphs are used across disparate fields to represent multidimensional data. However, although these glyphs are extremely effective, creating them is often laborious, even for those with professional design skills. This paper presents GlyphCreator, an interactive tool for the example-based generation of circular glyphs. Given an example circular glyph and multidimensional input data, GlyphCreator promptly generates a list of design candidates, any of which can be edited to satisfy the requirements of a particular representation. To develop GlyphCreator, we first derive a design space of circular glyphs by summarizing relationships between different visual elements. With this design space, we build a circular glyph dataset and develop a deep learning model for glyph parsing. The model can deconstruct a circular glyph bitmap into a series of visual elements. Next, we introduce an interface that helps users bind the input data attributes to visual elements and customize visual styles. We evaluate the parsing model through a quantitative experiment, demonstrate the use of GlyphCreator through two use scenarios, and validate its effectiveness through user interviews.

Abstract We survey the state of the art of spatial interfaces for 3D visualization. Interaction techniques are crucial to data visualization processes and the visualization research community has been calling for more research on interaction for years. Yet, research papers focusing on interaction techniques, in particular for 3D visualization purposes, are not always published in visualization venues, sometimes making it challenging to synthesize the latest interaction and visualization results. We therefore introduce a taxonomy of interaction technique for 3D visualization. The taxonomy is organized along two axes: the primary source of input on the one hand and the visualization task they support on the other hand. Surveying the state of the art allows us to highlight specific challenges and missed opportunities for research in 3D visualization. In particular, we call for additional research in: (1) controlling 3D visualization widgets to help scientists better understand their data, (2) 3D interaction techniques for dissemination, which are under‐explored yet show great promise for helping museum and science centers in their mission to share recent knowledge, and (3) developing new measures that move beyond traditional time and errors metrics for evaluating visualizations that include spatial interaction.

Storyline visualizations are an effective means to present the evolution of plots and reveal the scenic interactions among characters. However, the design of storyline visualizations is a difficult task as users need to balance between aesthetic goals and narrative constraints. Despite that the optimization-based methods have been improved significantly in terms of producing aesthetic and legible layouts, the existing (semi-) automatic methods are still limited regarding 1) efficient exploration of the storyline design space and 2) flexible customization of storyline layouts. In this work, we propose a reinforcement learning framework to train an AI agent that assists users in exploring the design space efficiently and generating well-optimized storylines. Based on the framework, we introduce PlotThread, an authoring tool that integrates a set of flexible interactions to support easy customization of storyline visualizations. To seamlessly integrate the AI agent into the authoring process, we employ a mixed-initiative approach where both the agent and designers work on the same canvas to boost the collaborative design of storylines. We evaluate the reinforcement learning model through qualitative and quantitative experiments and demonstrate the usage of PlotThread using a collection of use cases.

Given a scatterplot with tens of thousands of points or even more, a natural question is which sampling method should be used to create a small but "good" scatterplot for a better abstraction. We present the results of a user study that investigates the influence of different sampling strategies on multi-class scatterplots. The main goal of this study is to understand the capability of sampling methods in preserving the density, outliers, and overall shape of a scatterplot. To this end, we comprehensively review the literature and select seven typical sampling strategies as well as eight representative datasets. We then design four experiments to understand the performance of different strategies in maintaining: 1) region density; 2) class density; 3) outliers; and 4) overall shape in the sampling results. The results show that: 1) random sampling is preferred for preserving region density; 2) blue noise sampling and random sampling have comparable performance with the three multi-class sampling strategies in preserving class density; 3) outlier biased density based sampling, recursive subdivision based sampling, and blue noise sampling perform the best in keeping outliers; and 4) blue noise sampling outperforms the others in maintaining the overall shape of a scatterplot.

Videos become prevalent for storytellers to inspire viewers' interests. To further enhance narrations, visualizations are integrated into videos to present data-driven insights. However, manually crafting such data-driven videos is difficult and time-consuming. Thus, we present SmartShots, a system that facilitates the automatic integration of in-video visualizations. Specifically, we propose a computational framework that integrates non-verbal video clips, images, a melody, and a data table to create a video with data visualizations embedded. The system automatically translates the multi-media material into shots and then combines the shots into a compelling video. In addition, we develop a set of post-editing interactions to incorporate users' design knowledge and help them re-edit the automatically-generated videos.

We present co-skeletons, a new method that computes consistent curve skeletons for 3D shapes from a given family. We compute co-skeletons in terms of sampling density and semantic relevance, while preserving the desired characteristics of traditional, per-shape curve skeletonization approaches. We take the curve skeletons extracted by traditional approaches for all shapes from a family as input, and compute semantic correlation information of individual skeleton branches to guide an edge-pruning process via skeleton-based descriptors, clustering, and a voting algorithm. Our approach achieves more concise and family-consistent skeletons when compared to traditional per-shape methods. We show the utility of our method by using co-skeletons for shape segmentation and shape blending on real-world data.

Abstract We discuss spatial selection techniques for three‐dimensional datasets. Such 3D spatial selection is fundamental to exploratory data analysis. While 2D selection is efficient for datasets with explicit shapes and structures, it is less efficient for data without such properties. We first propose a new taxonomy of 3D selection techniques, focusing on the amount of control the user has to define the selection volume. We then describe the 3D spatial selection technique Tangible Brush, which gives manual control over the final selection volume. It combines 2D touch with 6‐DOF 3D tangible input to allow users to perform 3D selections in volumetric data. We use touch input to draw a 2D lasso, extruding it to a 3D selection volume based on the motion of a tangible, spatially‐aware tablet. We describe our approach and present its quantitative and qualitative comparison to state‐of‐the‐art structure‐dependent selection. Our results show that, in addition to being dataset‐independent, Tangible Brush is more accurate than existing dataset‐dependent techniques, thus providing a trade‐off between precision and effort.

Storyline visualization techniques have progressed significantly to generate illustrations of complex stories automatically. However, the visual layouts of storylines are not enhanced accordingly despite the improvement in the performance and extension of its application area. Existing methods attempt to achieve several shared optimization goals, such as reducing empty space and minimizing line crossings and wiggles. However, these goals do not always produce optimal results when compared to hand-drawn storylines. We conducted a preliminary study to learn how users translate a narrative into a hand-drawn storyline and check whether the visual elements in hand-drawn illustrations can be mapped back to appropriate narrative contexts. We also compared the hand-drawn storylines with storylines generated by the state-of-the-art methods and found they have significant differences. Our findings led to a design space that summarizes 1) how artists utilize narrative elements and 2) the sequence of actions artists follow to portray expressive and attractive storylines. We developed iStoryline, an authoring tool for integrating high-level user interactions into optimization algorithms and achieving a balance between hand-drawn storylines and automatic layouts. iStoryline allows users to create novel storyline visualizations easily according to their preferences by modifying the automatically generated layouts. The effectiveness and usability of iStoryline are studied with qualitative evaluations.

We present a family of three interactive Context-Aware Selection Techniques (CAST) for the analysis of large 3D particle datasets. For these datasets, spatial selection is an essential prerequisite to many other analysis tasks. Traditionally, such interactive target selection has been particularly challenging when the data subsets of interest were implicitly defined in the form of complicated structures of thousands of particles. Our new techniques SpaceCast, TraceCast, and PointCast improve usability and speed of spatial selection in point clouds through novel context-aware algorithms. They are able to infer a user's subtle selection intention from gestural input, can deal with complex situations such as partially occluded point clusters or multiple cluster layers, and can all be fine-tuned after the selection interaction has been completed. Together, they provide an effective and efficient tool set for the fast exploratory analysis of large datasets. In addition to presenting Cast, we report on a formal user study that compares our new techniques not only to each other but also to existing state-of-the-art selection methods. Our results show that Cast family members are virtually always faster than existing methods without tradeoffs in accuracy. In addition, qualitative feedback shows that PointCast and TraceCast were strongly favored by our participants for intuitiveness and efficiency.

Data selection is a fundamental task in visualization because it serves as a pre-requisite to many follow-up interactions. Efficient spatial selection in 3D point cloud datasets consisting of thousands or millions of particles can be particularly challenging. We present two new techniques, TeddySelection and CloudLasso, that support the selection of subsets in large particle 3D datasets in an interactive and visually intuitive manner. Specifically, we describe how to spatially select a subset of a 3D particle cloud by simply encircling the target particles on screen using either the mouse or direct-touch input. Based on the drawn lasso, our techniques automatically determine a bounding selection surface around the encircled particles based on their density. This kind of selection technique can be applied to particle datasets in several application domains. TeddySelection and CloudLasso reduce, and in some cases even eliminate, the need for complex multi-step selection processes involving Boolean operations. This was confirmed in a formal, controlled user study in which we compared the more flexible CloudLasso technique to the standard cylinder-based selection technique. This study showed that the former is consistently more efficient than the latter - in several cases the CloudLasso selection time was half that of the corresponding cylinder-based selection.

We present the design and evaluation of FI3D, a direct-touch data exploration technique for 3D visualization spaces. The exploration of three-dimensional data is core to many tasks and domains involving scientific visualizations. Thus, effective data navigation techniques are essential to enable comprehension, understanding, and analysis of the information space. While evidence exists that touch can provide higher-bandwidth input, somesthetic information that is valuable when interacting with virtual worlds, and awareness when working in collaboration, scientific data exploration in 3D poses unique challenges to the development of effective data manipulations. We present a technique that provides touch interaction with 3D scientific data spaces in 7 DOF. This interaction does not require the presence of dedicated objects to constrain the mapping, a design decision important for many scientific datasets such as particle simulations in astronomy or physics. We report on an evaluation that compares the technique to conventional mouse-based interaction. Our results show that touch interaction is competitive in interaction speed for translation and integrated interaction, is easy to learn and use, and is preferred for exploration and wayfinding tasks. To further explore the applicability of our basic technique for other types of scientific visualizations we present a second case study, adjusting the interaction to the illustrative visualization of fiber tracts of the brain and the manipulation of cutting planes in this context.