Visualizations play a crucial part in effective communication of concepts and information. Recent advances in reasoning and retrieval augmented generation have enabled Large Language Models (LLMs) to perform deep research and generate comprehensive reports. Despite its progress, existing deep research frameworks primarily focus on generating text-only content, leaving the automated generation of interleaved texts and visualizations underexplored. This novel task poses key challenges in designing informative visualizations and effectively integrating them with text reports. To address these challenges, we propose Formal Description of Visualization (FDV), a structured textual representation of charts that enables LLMs to learn from and generate diverse, high-quality visualizations. Building on this representation, we introduce Multimodal DeepResearcher, an agentic framework that decomposes the task into four stages: (1) researching, (2) exemplar report textualization, (3) planning and (4) multimodal report generation. For the evaluation of the generated reports, we develop MultimodalReportBench which contains 100 diverse topics as inputs, and a set of dedicated metrics for report and chart evaluation. Extensive experiments across models and evaluation methods demonstrate the effectiveness of Multimodal DeepResearcher. Notably, utilizing the same Claude 3.7 Sonnet model, Multimodal DeepResearcher achieves an 82% overall win rate over the baseline method.

Zhuoyun Du, Runze Wang, Huiyu Bai, Zouying Cao, Xiaoyong Zhu, Yu Cheng, Bo Zheng, Wei Chen, Haochao Ying. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Yinghao Tang, Xueding Liu, Boyuan Zhang, Tingfeng Lan, Yupeng Xie, Jiale Lao, Yiyao Wang, Haoxuan Li, Tingting Gao, Bo Pan, Luoxuan Weng, Xiuqi Huang, Minfeng Zhu, Yingchaojie Feng, Yuyu Luo, Wei Chen. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Visual Language Models (VLMs) have become a robust foundation for document question answering.Processing long documents remains challenging due to limited context windows and computational budgets.Existing page-level retrieval methods offer a practical solution, typically encoding pages and queries into vectors and ranking them via cosine similarity.However, such embedding-based methods (i) lack query-page interaction before similarity scoring and (ii) usually require a large-scale dataset to align visual and textual embeddings.In this paper, we observe that the cross-modal attention maps of well-trained VLMs are able to highlight semantically relevant regions.Building on this insight, we present CAPS (Crossmodal Attention as Page Selector), a retrieval framework that utilizes attention mechanisms inside VLMs for page selection.Specifically, CAPS first enhances attention-based retrieval capability with a small amount of contrastive data, then identifies the most effective attention head through expert head selection, and finally employs an adaptive filtering mechanism to obtain an appropriate number of relevant page candidates.Extensive experiments on four long-document benchmarks demonstrate that CAPS outperforms state-of-the-art embedding-based methods in both retrieval precision and downstream DocQA accuracy.Notably, CAPS achieves these gains using less than 10% of the training data required by competing baselines, highlighting the data efficiency of attention-based page retrieval.

Massive Open Online Courses (MOOCs) have become increasingly popular worldwide. However, learners primarily rely on watching videos, easily losing knowledge context and reducing learning effectiveness. We propose HyperMOOC, a novel approach augmenting MOOC videos with concept-based embedded visualizations to help learners maintain knowledge context. Informed by expert interviews and literature review, HyperMOOC employs multi-glyph designs for different knowledge types and multi-stage interactions for deeper understanding. Using a timeline-based radial visualization, learners can grasp cognitive paths of concepts and navigate courses through hyperlink-based interactions. We evaluated HyperMOOC through a user study with 36 MOOC learners and interviews with two instructors. Results demonstrate that HyperMOOC enhances learners’ learning effect and efficiency on MOOCs, with participants showing higher satisfaction and improved course understanding compared to traditional video-based learning approaches.

The rapid growth of e-commerce has driven an increasing demand for robotized warehouses to handle large-scale logistics orders. Upon receiving orders, a warehouse engages in task planning that involves two crucial stages: matching the orders with racks that contain the required items, and planning the robot routes to deliver those racks for order fulfillment. Hence, effective task planning is essential for maximizing order throughput. However, while existing techniques perform well for orders that involve items from a single rack, they exhibit low efficiency and poor performance when dealing with complex orders that require multiple items from different racks. In this paper, we introduce the robotized warehouse complex task planning problem and propose a novel Complex Order Online Planning (COOP) framework to address the challenge. Specifically, the framework matches orders with racks using a maximal coverage matching method, optimized through vector similarity search and a residual matching strategy. Then, it adopts an effective progressive prioritized pathfinding algorithm to transport matched racks with minimal delivery cost. Finally, the framework introduces an enhanced pathfinding-aware rack selection model that considers rack delivery costs from the pathfinding stage to collaboratively optimize rack matching and overall planning scheme. Extensive experiments on real-world and synthetic datasets demonstrate that our approaches exhibit strong performance across various parameter configurations.

Perpetual view generation aims to synthesize a long-term video corresponding to an arbitrary camera trajectory solely from a single input image. Recent methods commonly utilize a pre-trained text-to-image diffusion model to synthesize new content of previously unseen regions along camera movement. However, the underlying 2D diffusion model lacks 3D awareness and results in distorted artifacts. Moreover, they are limited to generating views of static 3D scenes, neglecting to capture object movements within the dynamic 4D world. To alleviate these issues, we present DreamJourney, a two-stage framework that leverages the world simulation capacity of video diffusion models to trigger a new perpetual scene view generation task with both camera movements and object dynamics. Specifically, in stage I, DreamJourney first lifts the input image to 3D point cloud and renders a sequence of partial images from a specific camera trajectory. A video diffusion model is then utilized as generative prior to complete the missing regions and enhance visual coherence across the sequence, producing a cross-view consistent video adheres to the 3D scene and camera trajectory. Meanwhile, we introduce two simple yet effective strategies (early stopping and view padding) to further stabilize the generation process and improve visual quality. Next, in stage II, DreamJourney leverages a multimodal large language model to produce a text prompt describing object movements in current view, and uses video diffusion model to animate current view with object movements. Stage I and II are repeated recurrently, enabling perpetual dynamic scene view generation. Extensive experiments demonstrate the superiority of our DreamJourney over state-of-the-art methods both quantitatively and qualitatively. Our project page:https://dream-journey.vercel.app/.

Large language models (LLMs) have achieved remarkable performance across a wide range of natural language tasks. Understanding how LLMs internally represent knowledge remains a significant challenge. Despite Sparse Autoencoders (SAEs) have emerged as a promising technique for extracting interpretable features from LLMs, SAE features do not inherently align with human-understandable concepts, making their interpretation cumbersome and labor-intensive. To bridge the gap between SAE features and human concepts, we present ConceptViz, a visual analytics system designed for exploring concepts in LLMs. ConceptViz implements a novel Identification ⇒ Interpretation ⇒Validation pipeline, enabling users to query SAEs using concepts of interest, interactively explore concept-to-feature alignments, and validate the correspondences through model behavior verification. We demonstrate the effectiveness of ConceptViz through two usage scenarios and a user study. Our results show that ConceptViz enhances interpretability research by streamlining the discovery and validation of meaningful concept representations in LLMs, ultimately aiding researchers in building more accurate mental models of LLM features. Our code and user guide are publicly available at https://github.com/Happy-Hippo209/conceptViz.

The advent of Retrieval-Augmented Generation (RAG) has significantly enhanced the ability of Large Language Models (LLMs) to produce factually accurate and up-to-date responses. However, the performance of a RAG system is not determined by a single component but emerges from a complex interplay of modular choices, such as embedding models and retrieval algorithms. This creates a vast and often opaque configuration space, making it challenging for developers to understand performance trade-offs and identify optimal designs. To address this challenge, we present RAGExplorer, a visual analytics system for the systematic comparison and diagnosis of RAG configurations. RAGExplorer guides users through a seamless macro-to-micro analytical workflow. Initially, it empowers developers to survey the performance landscape across numerous configurations, allowing for a high-level understanding of which design choices are most effective. For a deeper analysis, the system enables users to drill down into individual failure cases, investigate how differences in retrieved information contribute to errors, and interactively test hypotheses by manipulating the provided context to observe the resulting impact on the generated answer. We demonstrate the effectiveness of RAGExplorer through detailed case studies and user studies, validating its ability to empower developers in navigating the complex RAG design space. Our code and user guide are publicly available at https://github.com/Thymezzz/RAGExplorer.

The unauthorized acquisition of 3D assets by means of advanced techniques in 3D reconstruction is a major but often overlooked threat to publishers of videos. Preventing such threats is challenging due to the uninterpretable nature of 3D reconstruction and the diversity in requirements of 3D model demonstration. In this paper, we introduce VidGuard3D-a visual risk analysis approach that quantifies and locates the sources of risk for video-based 3D asset reconstruction attacks. Our approach uses attack simulation to support users in formulating a comprehensive understanding of 3D asset leakage risks, with a particular focus on the correlations between video segments and exposure risks of user-specified areas in the asset. We also proposed a prototype system that integrates this approach to facilitate video editing according to the knowledge of correlations. Two operations of video editing can be swiftly applied by users to form editing plans and minimize detected leakage risks. Finally, we conducted a user study and case studies that demonstrated the practicality and effectiveness of our approach.

The emergence of quantum computers heralds a new frontier in computational power, empowering quantum algorithms to address challenges that defy classical computation. However, the design of quantum algorithms is challenging as it largely requires the manual efforts of quantum experts to transit mathematical expressions to quantum circuit diagrams. To ease this process, particularly for prototyping, educational, and modular design workflows, we propose to bridge the textual and visual contexts between mathematics and quantum circuits through visual linking and transitions. We contribute a design space for quantum algorithm design, focusing on the textual and visual elements, interactions, and design patterns throughout the quantum algorithm design process. Informed by the design space, we introduce QuRAFT, a visual interface that facilitates a seamless transition from abstract mathematical expressions to concrete quantum circuits. QuRAFT incorporates a suite of eight integrated visual and interaction designs tailored to support users in the formulation, implementation, and validation process of the quantum algorithm design. Through two detailed case studies and a user evaluation, this paper demonstrates the effectiveness of QuRAFT. Feedback from quantum computing experts highlights the practical utility of QuRAFT in algorithm design and provides valuable implications for future advancements in visualization and interaction design within the quantum computing domain.

Recent advances in large language models (LLMs) have shown great potential in automating the process of visualization authoring through simple natural language utterances. However, instructing LLMs using natural language is limited in precision and expressiveness for conveying visualization intent, leading to misinterpretation and time-consuming iterations. To address these limitations, we conduct an empirical study to understand how LLMs interpret ambiguous or incomplete text prompts in the context of visualization authoring, and the conditions making LLMs misinterpret user intent. Informed by the findings, we introduce visual prompts as a complementary input modality to text prompts, which help clarify user intent and improve LLMs' interpretation abilities. To explore the potential of multimodal prompting in visualization authoring, we design VisPilot, which enables users to easily create visualizations using multimodal prompts, including text, sketches, and direct manipulations on existing visualizations. We evaluate VisPilot through a controlled user study and an expert evaluation. The results suggest that multimodal prompts facilitate users in communicating spatial constraints, local references, and design preferences while maintaining comparable task efficiency to text-only prompting. We further discuss when text, visual, and hybrid prompts are beneficial for visualization authoring, and summarize design implications for future human-AI authoring systems.

Urban visual querying leverages visual representations and interactions to depict the domain of interest and express related requests for exploring complex datasets, which is usually an iterative process. One main challenge of this process is the vast search space in terms of identifying querying conditions, observing querying results, and making the subsequent queries. This paper proposes a novel acceleration scheme that intelligently recommends a small set of querying results subject to previous queries. Central to our approach is a reinforcement learning based approach that trains a recommendation agent by simulating user behavior and characterizing the search space. We propose a mixed-initiative urban visual query scheme to enhance the exploration process additionally. We evaluate our approach by performing qualitative and quantitative experiments on a real-world scenario. The experimental results demonstrate the capability of reducing user workload, achieving optimized querying, and improving analysis efficiency.

Understanding and interpreting a script is essential for effective acting. Existing visualization methods, however, primarily focus on general narrative comprehension and often neglect actors' specific cognitive and expressive needs. To address this gap, we propose an actor-oriented visual analytics approach that automatically extracts key script features, including implicit emotions, causal relationships, and critical events, to support rapid comprehension of coherent storylines. These extracted features are integrated into a coordinated multi-view visualization, enabling actors to analyze and explore scripts comprehensively and gain deeper insights into the narrative. Furthermore, we introduce a method for structuring performance guidance, converting script events into interpretable, multi-stage recommendations that adapt to individual acting styles. Actor feedback is incorporated to iteratively refine these recommendations, ensuring alignment with personal performance traits. To demonstrate the practicality of our approach, we implement ActorVis, a system that operationalizes these methods and provides actors with interactive visualizations and adaptive performance guidance. Case studies and user evaluations confirm the approach's effectiveness in facilitating rapid script understanding and enhancing personalized performance support.

The security of individual privacy is paramount for trajectory publication, while preserving trajectory utility is also essential to serve analysis tasks such as urban planning and transportation development. To assess and maintain trajectory utility, existing studies consider geographic context. However, innate semantic characteristics of trajectories (e.g., origin, destination, stay point, path) have been overlooked, which prevents data owners from specifying task-specific utility measurements and, consequently, from achieving a delicate balance between privacy and utility. This paper proposes an interactive trajectory publishing approach driven by flexible utility considerations, which processes trajectory points according to their semantics to fulfill diverse utility requirements. Concretely, we decouple trajectories into origin-destination (OD) and path components: ODs are generalized into regions to satisfy $k$k-anonymity, and paths are sanitized within each OD group using road-network-aware differential privacy under predefined privacy constraints. We also develop a visual interface to support exploration and comprehension of privacy-preserving solutions, through which we incorporate human knowledge into the privacy scheme. Experiments on real-world urban datasets demonstrate the effectiveness of our approach.

Pathology images are crucial for cancer diagnosis and treatment. Although artificial intelligence has driven rapid advancements in pathology image analysis, the interpretation of ultra-large and multi-scale pathology images in clinical practice still heavily relies on physicians' experience. Clinicians need to repeatedly zoom in and out on individual slides to compare and assess pathological details - a process that is both time-consuming and prone to visual fatigue. The system first employs a diffusion model to perform tissue segmentation on pathology images, then calculates pathological tissue proportions and morphological metrics. Finally, through multi-scale dynamic comparison and multi-level visual evaluation, the system facilitates comprehensive and precise analysis of pathology images. The system provides clinicians with an intelligent and interactive tool for pathology image interpretation, enabling efficient visualization and precise analysis of pathological details, thereby reducing the effort require for detailed analysis.

Figurative art, as a culturally embedded medium, encodes narrative, symbolic, and emotional meanings that reflect artistic choices and historical realities. The recent availability of large-scale digital collections of figurative artworks creates opportunities for computational analysis, but existing methods mostly focus on classification or style detection, lacking structured modeling of high-level figurative elements and integration of cultural context. We present DAVA, a visual analytics system that supports interdisciplinary exploration of figurative art. First, we model paintings across three structural levels: facial expressions (micro), posture features (meso), and object co-occurrence (macro). Second, we employ a vision-language model to discover latent patterns from these features and present them through novel visualization designs. Third, we introduce domain-informed AI agents that simulate interdisciplinary research teams to interpret artworks in cultural and historical context. To evaluate DAVA, we first conducted a quantitative evaluation demonstrating the accuracy and consistency of the multi-agent interpretation mechanism. Case studies and expert interviews then confirmed the system's utility and support for semantically and historically informed exploration of figurative art.

Specifying visual encodings and interactions is exhausting but essential for authoring interactive visualizations. In this paper, we present G2, a customizable framework designed to support rapid generation of interactive visualizations with uniform specifications. G2 employs a data-driven grammar of graphics and defines a uniform set of interaction specifications. We discuss the design and implementation of G2 with rich examples, and a user interview to demonstrate its effectiveness. Since its first release in March 2016, G2 has undergone 360 iterations, received 12,000 stars, and supported over 22,600 related projects on Github.

Dependency-aware spatial crowdsourcing (DASC) addresses the unique challenges posed by subtask dependencies in spatial task assignment. This paper investigates the task assignment problem in DASC and proposes a two-stage Recommend and Match Optimization (RMO) framework, leveraging multi-agent reinforcement learning for subtask recommendation and a multi-dimensional utility function for subtask matching. The RMO framework primarily addresses two key challenges: credit assignment for subtasks with interdependencies and maintaining overall coherence between subtask recommendation and matching. Specifically, we employ meta-gradients to construct auxiliary policies and establish a gradient connection between two stages, which can effectively address credit assignment and joint optimization of subtask recommendation and matching, while concurrently accelerating network training. We further establish a unified gradient descent process through gradient synchronization across recommendation networks, auxiliary policies, and the matching utility evaluation function. Experiments on two real-world datasets validate the effectiveness and feasibility of our proposed approach.

Zhuoyun Du, LujieZheng LujieZheng, Renjun Hu, Yuyang Xu, Xiawei Li, Ying Sun, Wei Chen, Jian Wu, Haolei Cai, Haochao Ying. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Yingchaojie Feng, Yiqun Sun, Yandong Sun, Minfeng Zhu, Qiang Huang, Anthony Kum Hoe Tung, Wei Chen. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

With the proliferation of mobile devices, spatial crowdsourcing has emerged as a promising paradigm for facilitating location-based services, encompassing various applications across academia and industries. Recently, pioneering works have attempted to infer workers' mobility patterns from historical data to improve the quality of task assignment. However, these studies have overlooked or under-examined issues such as the dynamic mobility patterns of crowd workers, especially in the context of newcomers, the misalignment between the objectives of mobility prediction and task assignment, and the effective utilization of predicted mobility patterns. In this paper, we investigate a problem we term Task Assignment in Mobility Prediction-aware Spatial Crowdsourcing (TAMP). To address the TAMP problem, we first propose a task-adaptive meta-learning algorithm, which trains a set of specific meta-knowledge for workers' mobility prediction models through game theory-based learning task clustering and meta-training within each cluster. Then, we design a task assignment-oriented loss function and develop a task assignment algorithm that incorporates prediction performance, prioritizing assignments with higher confidence of completion. Extensive experiments on real-world datasets validate that our proposed methods can effectively improve the quality of task assignment.

Business intelligence (BI) transforms large volumes of data within modern organizations into actionable insights for informed decision-making. Recently, large language model (LLM)-based agents have streamlined the BI workflow by automatically performing task planning, reasoning, and actions in executable environments based on natural language (NL) queries. However, existing approaches primarily focus on individual BI tasks such as NL2SQL and NL2VIS. The fragmentation of tasks across different data roles and tools lead to inefficiencies and potential errors due to the iterative and collaborative nature of BI. In this paper, we introduce DataLab, a unified BI platform that integrates a one-stop LLM-based agent framework with an augmented computational notebook interface. DataLab supports various BI tasks for different data roles in data preparation, analysis, and visualization by seamlessly combining LLM assistance with user customization within a single environment. To achieve this unification, we design a domain knowledge incorporation module tailored for enterprise-specific BI tasks, an inter-agent communication mechanism to facilitate information sharing across the BI workflow, and a cell-based context management strategy to enhance context utilization efficiency in BI notebooks. Extensive experiments demonstrate that DataLab achieves state-of-the-art performance on various BI tasks across popular research benchmarks. Moreover, DataLab maintains high effectiveness and efficiency on real-world datasets from Tencent, achieving up to a 58.58% increase in accuracy and a 61.65 % reduction in token cost on enterprise-specific BI tasks.

Sample selection improves the efficiency and effectiveness of machine learning models by providing informative and representative samples. Typically, samples can be modeled as a sample graph, where nodes are samples and edges represent their similarities. Most existing methods are based on local information, such as the training difficulty of samples, thereby overlooking global information, such as connectivity patterns. This oversight can result in suboptimal selection because global information is crucial for ensuring that the selected samples well represent the structural properties of the graph. To address this issue, we employ structural entropy to quantify global information and losslessly decompose it from the whole graph to individual nodes using the Shapley value. Based on the decomposition, we present $\textbf{S}$tructural-$\textbf{E}$ntropy-based sample $\textbf{S}$election ($\textbf{SES}$), a method that integrates both global and local information to select informative and representative samples. SES begins by constructing a $k$NN-graph among samples based on their similarities. It then measures sample importance by combining structural entropy (global metric) with training difficulty (local metric). Finally, SES applies importance-biased blue noise sampling to select a set of diverse and representative samples. Comprehensive experiments on three learning scenarios --- supervised learning, active learning, and continual learning --- clearly demonstrate the effectiveness of our method.

Collaborative spatial crowdsourcing leverages distributed workers' collective intelligence to accomplish spatial tasks. A central challenge is to efficiently assign suitable workers to collaborate on these tasks. Although mainstream reinforcement learning (RL) methods have proven effective in task allocation, they face two key obstacles: delayed reward feedback and non-stationary data distributions, both hindering optimal allocation and collaborative efficiency. To address these limitations, we propose CAFE (credit assignment and fine-tuning enhanced), a novel multi-agent RL framework for spatial crowdsourcing. CAFE introduces a credit assignment mechanism that distributes rewards based on workers' contributions and spatiotemporal constraints, coupled with bi-level meta-optimization to jointly optimize credit assignment and RL policy. To handle non-stationary spatial task distributions, CAFE employs an adaptive fine-tuning procedure that efficiently adjusts credit assignment parameters while preserving collaborative knowledge. Experiments on two real-world datasets validate the effectiveness of our framework, demonstrating superior performance in terms of task completion and equitable reward redistribution.

Recent research has increasingly utilized policy gradient methods, a class of reinforcement learning algorithms, to optimize continuous-variable diffusion models for downstream objectives, which is also crucial in domains like graph generation. However, these approaches fail when applied directly to graph generation with discrete diffusion models. The primary challenge arises from the fact that directly applying such optimization complicates the denoising strategy of the models, leading to highly erratic and random outputs. To address this issue, we propose denoising trajectory complexity regularization (DTCR), a novel method that regularizes the optimization process of graph diffusion models and allows a simpler and more effective denoising strategy. Experimental results demonstrate that our method significantly improves prior approaches in multiple downstream task-specific objectives.

Understanding cultural heritage through technology faces challenges in connecting with diverse audiences, especially when interpreting art across cultures. In this work, we present CultiVerse, a visual analytics system that leverages Large Language Models (LLMs) to support cross-cultural appreciation of Traditional Chinese Paintings (TCPs). CultiVerse operates within a mixed-initiative framework and guides users through three stages: extracting cultural context, aligning cross-cultural symbols, and extrapolating meaning in the viewer's cultural frame. By combining an interactive interface with LLM-powered analysis, the system enables deeper engagement with symbolic meanings and encourages serendipitous cross-cultural discoveries. Our approach bridges AI interpretation and human insight to foster mutual understanding in a multicultural setting. A curated TCP dataset supports exploration, while empirical evaluations confirm that CultiVerse enhances user understanding, interpretation accuracy, and cultural empathy.

Graph-based Retrieval-Augmented Generation (RAG) has shown great capability in enhancing Large Language Model (LLM)’s answer with an external knowledge base. Compared to traditional RAG, it introduces a graph as an intermediate representation to capture better structured relational knowledge in the corpus, elevating the precision and comprehensiveness of generation results. However, developers usually face challenges in analyzing the effectiveness of GraphRAG on their dataset due to GraphRAG’s complex information processing pipeline and the overwhelming amount of LLM invocations involved during graph construction and query, which limits GraphRAG interpretability and accessibility. This research proposes a visual analysis framework that helps RAG developers identify critical recalls of GraphRAG and trace these recalls through the GraphRAG pipeline. Based on this framework, we develop XGraphRAG, a prototype system incorporating a set of interactive visualizations to facilitate users’ analysis process, boosting failure cases collection and improvement opportunities identification. Our evaluation demonstrates the effectiveness and usability of our approach. Our work is open-sourced and available at https://github.com/Gk0Wk/XGraphRAG.

Previous efforts have managed to generate production-ready 3D assets from text or images. However, these methods primarily employ NeRF or 3D Gaussian representations, which are not adept at producing smooth, high-quality geometries required by modern rendering pipelines. In this paper, we propose LDM, a L arge tensorial S D F M odel, which introduces a novel feed-forward framework capable of generating high-fidelity, illumination-decoupled textured mesh from a single image or text prompts. We firstly utilize a multi-view diffusion model to generate sparse multi-view inputs from single images or text prompts, and then a transformer-based model is trained to predict a tensorial SDF field from these sparse multi-view image inputs. Finally, we employ a gradient-based mesh optimization layer to refine this model, enabling it to produce an SDF field from which high-quality textured meshes can be extracted. Extensive experiments demonstrate that our method can generate diverse, high-quality 3D mesh assets with corresponding decomposed RGB textures within seconds. The project code is available at https://github.com/rgxie/LDM .

Machine learning (ML) has become a cornerstone of key applications at ByteDance. As model complexity and data volumes surge, data management for large-scale ML workloads faces substantial challenges, particularly with recent advances in large recommendation models (LRMs) and large multimodal models (LMMs). Traditional approaches exhibit limitations in storage efficiency, metadata scalability, update mechanisms, and integration with ML frameworks. To address these challenges, we propose Magnus, a holistic data management system built upon Apache Iceberg. Magnus integrates innovative optimizations across resource-efficient storage formats optimized for large wide tables and multimodal data, built-in support for vector and inverted indexes to accelerate data retrieval, scalable metadata planning with Git-like branching and tagging capabilities, and high-performance update/upsert based on lightweight merge-on-read (MOR) strategies. Additionally, Magnus provides native support and specialized enhancement for LRM and LMM training workloads. Experimental results demonstrate significant performance gains in real-world ML scenarios. Magnus has been deployed at ByteDance for over five years, enabling robust and efficient data infrastructure for large-scale ML workloads.

Voronoi diagram is commonly used to visualize geographical point dataset with a collection of plane-partitioned facets. As the size of the geographical point dataset increases, facets are densely distributed, and present different sizes and irregular shapes, leading to overdrawing and confusion problems, and hampering the visual perception of Voronoi diagram and insightful exploration of geographical point data. In this paper, we propose a novel Voronoi diagram generation framework to visualize and explore large-scale geographical point datasets. Firstly, an attribute-based blue noise sampling model is designed to select a subset of points to generate the simplified Voronoi diagram, retaining both the spatial distribution and attribute relationship of the original large-scale geographical points. Then a couple of optimization schemes are integrated into the sampling model to replace the representative points, aiming to enhance the visual perception of Voronoi diagram, such as shape balance and color characterization. Furthermore, we implement an interactive online Voronoi diagram generation tool, GeoVoronoi, enabling users to generate meaningful facets according to their requirements. Quantitative comparisons, case studies and user studies based on real-world datasets have demonstrated the effectiveness of our proposed method in the generation of credible Voronoi diagram and in-depth exploration of geographical point datasets.

In the context of the exponentially increasing volume of narrative texts such as novels and news, readers struggle to extract and consistently remember storylines from these intricate texts due to the constraints of human working memory and attention span. To tackle this issue, we propose a visualization approachStoryExplorer, which facilitates the process of knowledge externalization of narrative texts and further makes the form of mental models more coherent. Through the formative study and close collaboration with two domain experts, we identified key challenges for the extraction of the storyline. Guided by the distilled requirements, we then propose a set of workflow (i.e., insight finding-scripting-storytelling) to enable users to interactively generate fragments of narrative structures. We then propose a visualization systemStoryExplorerthat combines stroke annotation and GPT-based visual hints to quickly extract story fragments and interactively construct storylines. To evaluate the effectiveness and usefulness ofStoryExplorer, we conducted two case studies and in-depth user interviews with 16 target users. The result shows that users can conveniently and effectively extract the storyline by usingStoryExploreralong with the proposed workflow.

During the machining process of parts, choosing appropriate machining strategies optimizes production costs effectively. However, when the amount of data is limited, existing neural networks often struggle to fit the data accurately. Meanwhile, existing neural networks suffer from information dilution and lack effective mechanisms for direct information transfer between nonadjacent surfaces. This article proposes a method to extract General Machining Unit data. This data improves few-shot training performance. We investigate the distribution and information flow within General Machining Units and design a new way of data augmentation. In addition, to address the information dilution, a novel wormhole mechanism is proposed to aggregate information that spans the topological connections. In the backbone, we propose the Brep-WH layer that integrates wormhole mechanisms and attention pool layers. Both the Brep-WH network and the General Machining Unit data successfully improve the accuracy of the milling strategy dataset and the Fusion 360 Gallery segmentation dataset.

The proliferation of large language models (LLMs) has underscored concerns regarding their security vulnerabilities, notably against jailbreak attacks, where adversaries design jailbreak prompts to circumvent safety mechanisms for potential misuse. Addressing these concerns necessitates a comprehensive analysis of jailbreak prompts to evaluate LLMs' defensive capabilities and identify potential weaknesses. However, the complexity of evaluating jailbreak performance and understanding prompt characteristics makes this analysis laborious. We collaborate with domain experts to characterize problems and propose an LLM-assisted framework to streamline the analysis process. It provides automatic jailbreak assessment to facilitate performance evaluation and support analysis of components and keywords in prompts. Based on the framework, we design JailbreakLens, a visual analysis system that enables users to explore the jailbreak performance against the target model, conduct multi-level analysis of prompt characteristics, and refine prompt instances to verify findings. Through a case study, technical evaluations, and expert interviews, we demonstrate our system's effectiveness in helping users evaluate model security and identify model weaknesses.

Financial cluster analysis allows investors to discover investment alternatives and avoid undertaking excessive risks. However, this analytical task faces substantial challenges arising from many pairwise comparisons, the dynamic correlations across time spans, and the ambiguity in deriving implications from business relational knowledge. We propose Prismatic, a visual analytics system that integrates quantitative analysis of historical performance and qualitative analysis of business relational knowledge to cluster correlated businesses interactively. Prismatic features three clustering processes: dynamic cluster generation, knowledge-based cluster exploration, and correlation-based cluster validation. Utilizing a multi-view clustering approach, it enriches data-driven clusters with knowledge-driven similarity, providing a nuanced understanding of business correlations. Through well-coordinated visual views, Prismatic facilitates a comprehensive interpretation of intertwined quantitative and qualitative features, demonstrating its usefulness and effectiveness via case studies on formulating concept stocks and extensive interviews with domain experts.

Topology optimization can maximally leverage the high DOFs and mechanical potentiality of porous foams but faces challenges in adapting to free-form outer shapes, maintaining full connectivity between adjacent foam cells, and achieving high simulation accuracy. Utilizing the concept of Voronoi tessellation may help overcome the challenges owing to its distinguished properties on highly flexible topology, natural edge connectivity, and easy shape conforming. However, a variational optimization of the so-called Voronoi foams has not yet been fully explored. In addressing the issue, a concept of explicit topology optimization of open-cell Voronoi foams is proposed that can efficiently and reliably guide the foam's topology and geometry variations under critical physical and geometric requirements. Taking the site (or seed) positions and beam radii as the DOFs, we explore the differentiability of the open-cell Voronoi foams w.r.t. its seed locations, and propose a highly efficient local finite difference method to estimate the derivatives. During the gradient-based optimization, the foam topology can change freely, and some seeds may even be pushed out of shape, which greatly alleviates the challenges of prescribing a fixed underlying grid. The foam's mechanical property is also computed with a much-improved efficiency by an order of magnitude, in comparison with benchmark FEM, via a new material-aware numerical coarsening method on its highly heterogeneous density field counterpart. We show the improved performance of our Voronoi foam in comparison with classical topology optimization approaches and demonstrate its advantages in various settings.

The rapid development of large language models (LLMs), such as ChatGPT, has revolutionized the efficiency of creating programming tutorials. LLMs can be instructed with text prompts to generate comprehensive text descriptions for code snippets provided by users. However, the lack of transparency in the end-to-end generation process has hindered the understanding of model behavior and limited user control over the generated results. To tackle this challenge, we introduce a novel approach that breaks down the programming tutorial creation task into actionable steps. By employing the tree-of-thought method, LLMs engage in an exploratory process to generate diverse and faithful programming tutorials. We then present SPROUT, an authoring tool equipped with a series of interactive visualizations that empower users to have greater control and understanding of the programming tutorial creation process. A formal user study demonstrated the effectiveness of SPROUT, showing that our tool assists users to actively participate in the programming tutorial creation process, leading to more reliable and customizable results. By providing users with greater control and understanding, SPROUT enhances the user experience and improves the overall quality of programming tutorial.

Recently, Large Language Model based Autonomous System (LLMAS) has gained great popularity for its potential to simulate complicated behaviors of human societies. One of its main challenges is to present and analyze the dynamic events evolution of LLMAS. In this work, we present a visualization approach to explore the detailed statuses and agents' behavior within LLMAS. Our approach outlines a general pipeline that organizes raw execution events from LLMAS into a structured behavior model. We leverage a behavior summarization algorithm to create a hierarchical summary of these behaviors, arranged according to their sequence over time. Additionally, we design a cause trace method to mine the causal relationship between agent behaviors. We then develop AgentLens, a visual analysis system that leverages a hierarchical temporal visualization for illustrating the evolution of LLMAS, and supports users to interactively investigate details and causes of agents' behaviors. Two usage scenarios and a user study demonstrate the effectiveness and usability of our AgentLens.

The proliferation of large language models (LLMs) has revolutionized the capabilities of natural language interfaces (NLIs) for data analysis. LLMs can perform multi-step and complex reasoning to generate data insights based on users' analytic intents. However, these insights often entangle with an abundance of contexts in analytic conversations such as code, visualizations, and natural language explanations. This hinders efficient recording, organization, and navigation of insights within the current chat-based LLM interfaces. In this paper, we first conduct a formative study with eight data analysts to understand their general workflow and pain points of insight management during LLM-powered data analysis. Accordingly, we introduce InsightLens, an interactive system to overcome such challenges. Built upon an LLM-agent-based framework that automates insight recording and organization along with the analysis process, InsightLens visualizes the complex conversational contexts from multiple aspects to facilitate insight navigation. A user study with twelve data analysts demonstrates the effectiveness of InsightLens, showing that it significantly reduces users' manual and cognitive effort without disrupting their conversational data analysis workflow, leading to a more efficient analysis experience.

Graph visualization techniques provide a foundational framework for offering comprehensive overviews and insights into cloud computing systems, facilitating efficient management and ensuring their availability and reliability. Despite the enhanced computational and storage capabilities of larger-scale cloud computing architectures, they introduce significant challenges to traditional graph-based visualization due to issues of hierarchical heterogeneity, scalability, and data incompleteness. This paper proposes a novel abstraction approach to visualize large multi-layer networks. Our method leverages graphons, a probabilistic representation of network layers, to encompass three core steps: an inner-layer summary to identify stable and volatile substructures, an inter-layer mixup for aligning heterogeneous network layers, and a context-aware multi-layer joint sampling technique aimed at reducing network scale while retaining essential topological characteristics. By abstracting complex network data into manageable weighted graphs, with each graph depicting a distinct network layer, our approach renders these intricate systems accessible on standard computing hardware. We validate our methodology through case studies, quantitative experiments and expert evaluations, demonstrating its effectiveness in managing large multi-layer networks, as well as its applicability to broader network types such as transportation and social networks.

Chart images, such as bar charts, pie charts, and line charts, are explosively produced due to the wide usage of data visualizations. Accordingly, knowledge mining from chart images is becoming increasingly important, which can benefit downstream tasks like chart retrieval and knowledge graph completion. However, existing methods for chart knowledge mining mainly focus on converting chart images into raw data and often ignore their visual encodings and semantic meanings, which can result in information loss for many downstream tasks. In this paper, we propose ChartKG, a novel knowledge graph (KG) based representation for chart images, which can model the visual elements in a chart image and semantic relations among them including visual encodings and visual insights in a unified manner. Further, we develop a general framework to convert chart images to the proposed KG-based representation. It integrates a series of image processing techniques to identify visual elements and relations, e.g., CNNs to classify charts, yolov5 and optical character recognition to parse charts, and rule-based methods to construct graphs. We present four cases to illustrate how our knowledge-graph-based representation can model the detailed visual elements and semantic relations in charts, and further demonstrate how our approach can benefit downstream applications such as semantic-aware chart retrieval and chart question answering. We also conduct quantitative evaluations to assess the two fundamental building blocks of our chart-to-KG framework, i.e., object recognition and optical character recognition. The results provide support for the usefulness and effectiveness of ChartKG.

Massive Open Online Courses (MOOCs) platforms are becoming increasingly popular in recent years. Online learners need to watch the whole course video on MOOC platforms to learn the underlying new knowledge, which is often tedious and time-consuming due to the lack of a quick overview of the covered knowledge and their structures. In this article, we propose ConceptThread, a visual analytics approach to effectively show the concepts and the relations among them to facilitate effective online learning. Specifically, given that the majority of MOOC videos contain slides, we first leverage video processing and speech analysis techniques, including shot recognition, speech recognition and topic modeling, to extract core knowledge concepts and construct the hierarchical and temporal relations among them. Then, by using a metaphor of thread, we present a novel visualization to intuitively display the concepts based on video sequential flow, and enable learners to perform interactive visual exploration of concepts. We conducted a quantitative study, two case studies, and a user study to extensively evaluate ConceptThread. The results demonstrate the effectiveness and usability of ConceptThread in providing online learners with a quick understanding of the knowledge content of MOOC videos.

Programming tutorials serve a crucial role in teaching coding and programming techniques. Creating high-quality programming tutorials remains a laborious task. Authors devote effort in writing step-by-step solutions, creating examples, and editing existing tutorials. We explore the potential of using the text-code connection to improve the authoring experience of programming tutorials. We proposed a mixed-initiative approach to infer, establish, and maintain the latent text-code connections. With a series of interactions, the STEP-LINK ( STEP- by-Step Tutorial Editing with Programmable LINK ages ) prototype leverages text-code connections to assist users in authoring tutorials. The results of our experiment demonstrate the effectiveness of our system in supporting users in the authoring of step-by-step code explanations, the creation of examples, and the iteration of tutorials.

Mutual funds are one of the most important and popular investment ways for ordinary investors to maintain and increase the value of their assets. However, it is challenging for ordinary investors to select optimal mutual funds from thousands of fund choices managed by different managers. Various investors often have different personal investment preferences and it is difficult to characterize their preferences quickly. Also, mutual fund performance relies on various factors (e.g., the economic market and the management of fund managers), and most of these factors are dynamically changing, making it difficult to efficiently compare different mutual funds in detail. To address these challenges, we propose FundSelector, an interactive multi-view visual analytics system that quantifies user preferences to rank mutual funds and allows ordinary investors to explore mutual fund performance in terms of multiple factors and scales. Two novel visual designs are proposed to enable detailed comparisons of mutual funds. Rank-informed bipartite contribution bar chart provides interpretable fund ranking results by explicitly showing both positive and negative factors. Elastic trend chart allows investors to analyze and compare the temporal evolution of the mutual funds’ performances in a customizable way. We evaluated FundSelector through two case studies and interviews with eight ordinary investors. The results highlight its effectiveness and utility.

Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, Qian Liu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Model stealing (MS) involves querying and observing the output of a machine learning model to steal its capabilities. The quality of queried data is crucial, yet obtaining a large amount of real data for MS is often challenging. Recent works have reduced reliance on real data by using generative models. However, when high-dimensional query data is required, these methods are impractical due to the high costs of querying and the risk of model collapse. In this work, we propose using sample gradients (SG) to enhance the utility of each real sample, as SG provides crucial guidance on the decision boundaries of the victim model. However, utilizing SG in the model stealing scenario faces two challenges: 1. Pixel-level gradient estimation requires ex-tensive query volume and is susceptible to defenses. 2. The estimation of sample gradients has a significant variance. This paper proposes Superpixel Sample Gradient stealing (SPSG) for model stealing under the constraint of limited real samples. With the basic idea of imitating the victim model's low-variance patch-level gradients instead ofpixel-level gradients, SPSG achieves efficient sample gradient es-timation through two steps. First, we perform patch-wise perturbations on query images to estimate the average gradient in different regions of the image. Then, we filter the gradients through a threshold strategy to reduce variance. Exhaustive experiments demonstrate that, with the same number of real samples, SPSG achieves accuracy, agreements, and adversarial success rate significantly surpassing the current state-of-the-art MS methods. Codes are available at https://github.com/zyI123456aBISPSG_attack.

In causal inference, estimating heterogeneous treatment effects (HTE) is critical for identifying how different subgroups respond to interventions, with broad applications in fields such as precision medicine and personalized advertising. Although HTE estimation methods aim to improve accuracy, how to provide explicit subgroup descriptions remains unclear, hindering data interpretation and strategic intervention management. In this paper, we propose CURLS, a novel rule learning method leveraging HTE, which can effectively describe subgroups with significant treatment effects. Specifically, we frame causal rule learning as a discrete optimization problem, finely balancing treatment effect with variance and considering the rule interpretability. We design an iterative procedure based on the minorize-maximization algorithm and solve a submodular lower bound as an approximation for the original. Quantitative experiments and qualitative case studies verify that compared with state-of-the-art methods, CURLS can find subgroups where the estimated and true effects are 16.1% and 13.8% higher and the variance is 12.0% smaller, while maintaining similar or better estimation accuracy and rule interpretability. Code is available at https://osf.io/zwp2k/.

Recent research has made significant progress in optimizing diffusion models for downstream objectives, which is an important pursuit in fields such as graph generation for drug design. However, directly applying these models to graph presents challenges, resulting in suboptimal performance. This paper introduces graph diffusion policy optimization (GDPO), a novel approach to optimize graph diffusion models for arbitrary (e.g., non-differentiable) objectives using reinforcement learning. GDPO is based on an eager policy gradient tailored for graph diffusion models, developed through meticulous analysis and promising improved performance. Experimental results show that GDPO achieves state-of-the-art performance in various graph generation tasks with complex and diverse objectives. Code is available at https://github.com/sail-sg/GDPO.

In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions, however, directly learning with which may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100M long caption oriented text-image pairs. Our method achieves superior performance in long-text-image retrieval tasks. The project page is available at https://wuw2019.github.io/lot-lip.

Authoring graph visualization requires advanced programming skills, expert domain knowledge, and significant workload. Existing authoring tools either support limited templates of graph visualization, or suffer from a high learning cost. We analyze the design requirements on a tool for graph visualizations, and contribute Nuwa, a user-friendly declarative authoring tool for the interactive specification of graph visualizations in terms of data, entity, change, and encoding. Our implementation empowers users to conveniently create, compare, and modulate comprehensive graph visualizations with a wide range of styles. We showcase various examples to verify the expressiveness of Nuwa. Via an expert interview and the analysis on cognitive dimensions we evaluate the usability of Nuwa.

This paper presents GraphFederator, a novel approach to construct federated representations of multi-party graphs and supports privacy-preserving visual analysis of graphs. Inspired by the concept of federated learning, we reformulate the analysis of multi-party graphs into a decentralization process. The new federation framework consists of a shared module that is responsible for federated modeling and analysis, and a set of local modules that run on respective graph data. Specifically, we propose a Federated Graph Representation Model (FGRM) that is learned from encrypted characteristics of multi-party graphs in local modules. We also design multiple visualization tools for federated visualization, exploration, and analysis of multi-party graphs. Experimental results on two datasets demonstrate the effectiveness of our approach.

Social networks are the fabric of society and the subject of frequent visual analysis. Closed triads represent triangular relationships between three people in a social network and are significant for understanding inherent interconnections and influence within the network. The most common methods for representing social networks (node-link diagrams and adjacency matrices) are not optimal for understanding triangles. We propose extending the adjacency matrix form to 3D for better visualization of network triads. We design a 3D matrix reordering technique and implement an immersive interactive system to assist in visualizing and analyzing closed triads in social networks. The evaluations demonstrate that our method provides substantial added value over node-link diagrams in improving the efficiency and accuracy of manipulating and understanding the social network triads.

We propose a visual query system for risky enterprises. Based on a small number of input samples of risky enterprises, users can interactively analyze and obtain SQL statements that express the query results, facilitating subsequent task operations and understanding risk criteria. Due to the diversity of enterprises and the complexity of economic indicators, understanding, evaluating, and querying risky enterprises is a challenging task for ordinary users. To solve this problem, our visual query system integrates multicriteria decision-making techniques and data mining through a visual analysis interface. The system facilitates efficient identification of risky enterprises, leveraging knowledge gained from a small number of known risk company samples. Our system includes an analytic hierarchy model to express domain knowledge, a query tree construction algorithm, and a well-designed visual interface. Two real cases demonstrate the effectiveness of our system.

Abstract Experimental economics is an important branch of economics to study human behaviours in a controlled laboratory setting or out in the field. Scientific experiments are conducted in experimental economics to collect what decisions people make in specific circumstances and verify economic theories. As a significant couple of variables in the virtual experimental environment, decisions and outcomes change with the subjective factors of participants and objective circumstances, making it a difficult task to capture human behaviour patterns and establish correlations to verify economic theories. In this paper, we present a visual analytics system, EBPVis, which enables economists to visually explore human behaviour patterns and faithfully verify economic theories, e.g. the vicious cycle of poverty and poverty trap. We utilize a Doc2Vec model to transform the economic behaviours of participants into a vectorized space according to their sequential decisions, where frequent sequences can be easily perceived and extracted to represent human behaviour patterns. To explore the correlation between decisions and outcomes, an Outcome View is designed to display the outcome variables for behaviour patterns. We also provide a Comparison View to support an efficient comparison between multiple behaviour patterns by revealing their differences in terms of decision combinations and time‐varying profits. Moreover, an Individual View is designed to illustrate the outcome accumulation and behaviour patterns of subjects. Case studies, expert feedback and user studies based on a real‐world dataset have demonstrated the effectiveness and practicability of EBPVis in the representation of economic behaviour patterns and certification of economic theories.

Advanced quantum computers have the capability to perform practical quantum computing to address specific problems that are intractable for classical computers. Nevertheless, these computers are susceptible to noise, leading to unexpectable errors in outcomes, which makes them less trustworthy. To address this challenge, we propose HammingVis, a visual analytics approach that helps identify and understand errors in quantum outcomes. Given that these errors exhibit latent structural patterns within Hamming space, we introduce two graph visualizations to reveal these patterns from distinct perspectives. One highlights the overall structure of errors, while the other focuses on the impact of errors within important subspaces. We further develop a prototype system for interactively exploring and discerning the correct outcomes within Hamming space. A novel design is presented to distinguish the neighborhood patterns between error and correct outcomes. The effectiveness of our approach is demonstrated through case studies involving two classic quantum algorithms’ outcome data.

Large language models (LLMs) like those that power OpenAI’s ChatGPT and Google’s Gemini have played a major part in the recent wave of machine learning and artificial intelligence advancements. However, interpreting LLMs and visualizing their components is extremely difficult due to the incredible scale and high dimensionality of model data. NeuronautLLM introduces a visual analysis system for identifying and visualizing influential neurons in transformer-based language models as they relate to user-defined prompts. Our approach combines simple, yet information-dense visualizations as well as neuron explanation and classification data to provide a wealth of opportunities for exploration. NeuronautLLM was reviewed by two experts to verify its efficacy as a tool for practical model interpretation. Interviews and usability tests with five LLM experts demonstrated NeuronautLLM’s exceptional usability and its readiness for real-world application. Furthermore, two in-depth case studies on model reasoning and social bias highlight NeuronautLLM’s versatility in aiding the analysis of a wide range of LLM research problems.

Interface icons are often scaled to adapt to different displays in cross-device collaborations. However, adaptive scaling of icons may cause perceptual bias in how icon arrays are visually perceived, which reduces usability and coherent user experience. This article presents an empirical study that evaluates the perceptual bias in the consistency of icon spacing and size caused by adaptive scaling. Then the impact of various visual features of icons (i.e., the border shape, polarity, and composition) on the perceptual bias are investigated. In this study, we found that cross-device scaling of icons causes a perceptual bias in the consistency of icon spacing and size. The shape of the icon border has a significant difference in the perceived spacing of icons, and the bias of the round shape is smaller than that of the square shape. Moreover, changing the icon polarity can affect the perceptual bias of consistency in icon size. These findings are expected to propose scaling recommendations for improving the visual consistency of icon arrays across.

Smart city situational awareness has recently emerged as a hot topic in research societies, industries, and governments because of its potential to integrate cutting-edge information technology and solve urgent challenges that modern cities face. For example, in the latest five-year plan, the Chinese government has highlighted the demand to empower smart city management with new technologies such as big data and Internet of Things, for which situational awareness is normally the crucial first step. While traditional static surveillance data on cities have been available for decades, this review reports a type of relatively new yet highly important urban data source, i.e., the big mobile data collected by devices with various levels of mobility representing the movement and distribution of public and private agents in the city. We especially focus on smart city situational awareness enabled by synthesizing the localization of hundreds of thousands of mobile software Apps using the Global Positioning System (GPS). This technique enjoys advantages such as a large penetration rate (∼50% urban population covered), uniform spatiotemporal coverage, and high localization precision. We first discuss the pragmatic requirements for smart city situational awareness and the challenges faced. Then we introduce two suites of empowering technologies that help fulfill the requirements of (1) cybersecurity insurance for smart cities and (2) spatiotemporal modeling and visualization for situational awareness, both via big mobile data. The main contributions of this review lie in the description of a comprehensive technological framework for smart city situational awareness and the demonstration of its feasibility via real-world applications.

Data imputation is an essential pre-processing task for data governance, aimed at filling in incomplete data. However, conventional data imputation methods can only partly alleviate data incompleteness using isolated tabular data, and they fail to achieve the best balance between accuracy and efficiency. In this paper, we present a novel visual analysis approach for data imputation. We develop a multi-party tabular data association strategy that uses intelligent algorithms to identify similar columns and establish column correlations across multiple tables. Then, we perform the initial imputation of incomplete data using correlated data entries from other tables. Additionally, we develop a visual analysis system to refine data imputation candidates. Our interactive system combines the multi-party data imputation approach with expert knowledge, allowing for a better understanding of the relational structure of the data. This significantly enhances the accuracy and efficiency of data imputation, thereby enhancing the quality of data governance and the intrinsic value of data assets. Experimental validation and user surveys demonstrate that this method supports users in verifying and judging the associated columns and similar rows using their domain knowledge.

Top- k Publish/Subscribe (TkPS) service is widely studied in spatial database, with various cache-based methods proposed to address its efficiency challenge in top- k result maintenance. These methods require in-depth exploration of relationships between cache updates and different factors (e.g., data distribution) to optimize cache performance. However, there is currently no system available that assists developers in conducting comprehensive cache analyses within TkPS services. We therefore introduce Catcher , a multi-functional cache analysis system designed for TkPS services. It not only enables users to intuitively analyze the entire maintenance process of top- k results but also aids in identifying bottlenecks and potential optimization spaces of caches. Catcher provides two user-friendly interfaces that allow users to employ simple and easy-to-use consoles to perform statistical analysis. Furthermore, Catcher offers the real-time evaluation of cache-based methods, providing users with instant analysis. We have demonstrated the usability of Catcher on real-world datasets. A short video of our demonstration can be found at https://youtu.be/qI81HoypB0w.

Drug Target Interaction (DTI) prediction plays a crucial role in in-silico drug discovery, especially for deep learning (DL) models. Along this line, existing methods usually first extract features from drugs and target proteins, and use drug-target pairs to train DL models. However, these DL-based methods essentially rely on similar structures and patterns defined by the homologous proteins from a large amount of data. When few drug-target interactions are known for a newly discovered protein and its homologous proteins, prediction performance can suffer notable reduction. In this paper, we propose a novel Protein-Context enhanced Master/Slave Framework (PCMS), for zero-shot DTI prediction. This framework facilitates the efficient discovery of ligands for newly discovered target proteins, addressing the challenge of predicting interactions without prior data. Specifically, the PCMS framework consists of two main components: a Master Learner and a Slave Learner. The Master Learner first learns the target protein context information, and then adaptively generates the corresponding parameters for the Slave Learner. The Slave Learner then perform zero-shot DTI prediction in different protein contexts. Extensive experiments verify the effectiveness of our PCMS compared to state-of-the-art methods in various metrics on two public datasets.

With the acceleration of economic globalization, a large amount of research studies have been conducted for the exploration of industrial economics. In the visualization community, common visualization tools present potential features of industrial economics. They hardly meet the various and complex user requirements for insightful analysis and decision-making. In this article, we design VIEA, a web-based visualization system that integrates a rich set of views and tailored interactions, enabling users to easily perceive economic features, such as geographical distributions, trade relationships, and pattern comparisons. Case studies and user studies based on real-world datasets have been conducted to demonstrate the effectiveness of our system in the exploration of industrial economics.

Recently, implicit neural representation has been widely used to learn the appearance of human bodies in the canonical space, which can be further animated using a parametric human model. However, how to decompose the material properties from the implicit representation for relighting has not yet been investigated thoroughly. We propose to address this problem with a novel framework, ReN Human, that takes sparse or even monocular input videos collected in unconstrained lighting to produce a 3D human representation that can be rendered with novel views, poses, and lighting. Our method represents humans as deformable implicit neural representation and decomposes the geometry, material of humans as well as environment illumination for capturing a relightable and animatable human model. Moreover, we introduce a volumetric lighting grid consisting of spherical Gaussian mixtures to learn the spatially varying illumination and animatable visibility probes to model the dynamic self-occlusion caused by human motion. Specifically, we learn the material property fields and illumination using a physically-based rendering layer that uses Monte Carlo importance sampling to facilitate differentiation of the complex rendering integral. We demonstrate that our approach outperforms recent novel views and poses synthesis methods in a challenging benchmark with sparse videos, enabling high-fidelity human relighting.

We focus on the challenging problem of learning an unbiased classifier from a large number of potentially relevant but noisily labeled web images given only a few clean labeled images. This problem is particularly practical because it reduces the expensive annotation costs by utilizing freely accessible web images with noisy labels. Typically, prototypes are representative images or features used to classify or identify other images. However, in the few clean and many noisy scenarios, the class prototype can be severely biased due to the presence of irrelevant noisy images. The resulting prototypes are less compact and discriminative, as previous methods do not take into account the diverse range of images in the noisy web image collections. On the other hand, the relation modeling between noisy and clean images is not learned for the class prototype generation in an end-to-end manner, which results in a suboptimal class prototype. In this article, we introduce a similarity maximization loss named SimNoiPro. Our SimNoiPro first generates noise-tolerant hybrid prototypes composed of clean and noise-tolerant prototypes and then pulls them closer to each other. Our approach considers the diversity of noisy images by explicit division and overcomes the optimization discrepancy issue. This enables better relation modeling between clean and noisy images and helps extract judicious information from the noisy image set. The evaluation results on two extended few-shot classification benchmarks confirm that our SimNoiPro outperforms prior methods in measuring image relations and cleaning noisy data.

In the field of digital humanities, color research aims to discover explanations for painting history and color usage habits. However, researchers analyzing color relationships is challenging and time-consuming, as it requires color extraction and a detailed review of many painting images for reference and comparison of color relationships. In our work, we propose ColorNetVis, an interactive color network analysis tool that enables researchers to explore color relationships through color networks. The core of ColorNetVis is a bipartite network model that establishes a bipartite relationship between colors and Chinese painting within a scope based on color difference measurement. It constructs a one-mode color network through projection algorithms and similarity calculation methods to discover the relationship between colors. We propose a coordinated set of views to demonstrate the combination of determined color networks with painting types and real-world attributes. We use color space view, color attribute distribution view, and single color query components to assist researchers in conducting detailed color analysis and validation. Through case studies, researcher reviews, and user studies, we demonstrate that ColorNetVis can effectively help researchers discover knowledge of color relationships and potential color research directions.

Natural language interfaces (NLIs) enable users to flexibly specify analytical intentions in data visualization. However, diagnosing the visualization results without understanding the underlying generation process is challenging. Our research explores how to provide explanations for NLIs to help users locate the problems and further revise the queries. We present XNLI, an explainable NLI system for visual data analysis. The system introduces a Provenance Generator to reveal the detailed process of visual transformations, a suite of interactive widgets to support error adjustments, and a Hint Generator to provide query revision hints based on the analysis of user queries and interactions. Two usage scenarios of XNLI and a user study verify the effectiveness and usability of the system. Results suggest that XNLI can significantly enhance task accuracy without interrupting the NLI-based analysis process.

Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.

The transfer function is crucial for direct volume rendering (DVR) to create an informative visual representation of volumetric data. However, manually adjusting the transfer function to achieve the desired DVR result can be time-consuming and unintuitive. In this paper, we propose Differentiable Design Galleries, an image-based transfer function design approach to help users explore the design space of transfer functions by taking advantage of the recent advances in deep learning and differentiable rendering. Specifically, we leverage neural rendering to learn a latent design space, which is a continuous manifold representing various types of implicit transfer functions. We further provide a set of interactive tools to support intuitive query, navigation, and modification to obtain the target design, which is represented as a neural-rendered design exemplar. The explicit transfer function can be reconstructed from the target design with a differentiable direct volume renderer. Experimental results on real volumetric data demonstrate the effectiveness of our method.

A computational graph in a deep neural network (DNN) denotes a specific data flow diagram (DFD) composed of many tensors and operators. Existing toolkits for visualizing computational graphs are not applicable when the structure is highly complicated and large-scale (e.g., BERT (Devlin et al. 2019)). To address this problem, we propose leveraging a suite of visual simplification techniques, including a cycle-removing method, a module-based edge-pruning algorithm, and an isomorphic subgraph stacking strategy. We design and implement an interactive visualization system that is suitable for computational graphs with up to 10 thousand elements. Experimental results and usage scenarios demonstrate that our tool reduces 60% elements on average and hence enhances the performance for recognizing and diagnosing DNN models. Our contributions are integrated into an open-source DNN visualization toolkit, namely, MindInsight [2].

Diagnosing the cluster-based performance of large-scale deep neural network (DNN) models during training is essential for improving training efficiency and reducing resource consumption. However, it remains challenging due to the incomprehensibility of the parallelization strategy and the sheer volume of complex data generated in the training processes. Prior works visually analyze performance profiles and timeline traces to identify anomalies from the perspective of individual devices in the cluster, which is not amenable for studying the root cause of anomalies. In this article, we present a visual analytics approach that empowers analysts to visually explore the parallel training process of a DNN model and interactively diagnose the root cause of a performance issue. A set of design requirements is gathered through discussions with domain experts. We propose an enhanced execution flow of model operators for illustrating parallelization strategies within the computational graph layout. We design and implement an enhanced Marey's graph representation, which introduces the concept of time-span and a banded visual metaphor to convey training dynamics and help experts identify inefficient training processes. We also propose a visual aggregation technique to improve visualization efficiency. We evaluate our approach using case studies, a user study and expert interviews on two large-scale models run in a cluster, namely, the PanGu- α 13B model (40 layers), and the Resnet model (50 layers).

Quantum computing is a rapidly evolving field that enables exponential speed-up over classical algorithms. At the heart of this revolutionary technology are quantum circuits, which serve as vital tools for implementing, analyzing, and optimizing quantum algorithms. Recent advancements in quantum computing and the increasing capability of quantum devices have led to the development of more complex quantum circuits. However, traditional quantum circuit diagrams suffer from scalability and readability issues, which limit the efficiency of analysis and optimization processes. In this research, we propose a novel visualization approach for large-scale quantum circuits by adopting semantic analysis to facilitate the comprehension of quantum circuits. We first exploit meta-data and semantic information extracted from the underlying code of quantum circuits to create component segmentations and pattern abstractions, allowing for easier wrangling of massive circuit diagrams. We then develop Quantivine, an interactive system for exploring and understanding quantum circuits. A series of novel circuit visualizations is designed to uncover contextual details such as qubit provenance, parallelism, and entanglement. The effectiveness of Quantivine is demonstrated through two usage scenarios of quantum circuits with up to 100 qubits and a formal user evaluation with quantum experts. A free copy of this paper and all supplemental materials are available at https://osf.io/2m9yh/?view_only=0aa1618c97244f5093cd7ce15f1431f9.

The study of cultural artifact provenance, tracing ownership and preservation, holds significant importance in archaeology and art history. Modern technology has advanced this field, yet challenges persist, including recognizing evidence from diverse sources, integrating sociocultural context, and enhancing interactive automation for comprehensive provenance analysis. In collaboration with art historians, we examined the handscroll, a traditional Chinese painting form that provides a rich source of historical data and a unique opportunity to explore history through cultural artifacts. We present a three-tiered methodology encompassing artifact, contextual, and provenance levels, designed to create a "Biography" for handscroll. Our approach incorporates the application of image processing techniques and language models to extract, validate, and augment elements within handscroll using various cultural heritage databases. To facilitate efficient analysis of non-contiguous extracted elements, we have developed a distinctive layout. Additionally, we introduce ScrollTimes, a visual analysis system tailored to support the three-tiered analysis of handscroll, allowing art historians to interactively create biographies tailored to their interests. Validated through case studies and expert interviews, our approach offers a window into history, fostering a holistic understanding of handscroll provenance and historical significance.

Differential privacy is an essential approach for privacy preservation in data queries. However, users face a significant challenge in selecting an appropriate privacy scheme, as they struggle to balance the utility of query results with the preservation of diverse individual privacy. Customizing a privacy scheme becomes even more complex in dealing with queries that involve multiple data attributes. When adversaries attempt to breach privacy firewalls by conducting multiple regular data queries with various attribute values, data owners must arduously discern unpredictable disclosure risks and construct suitable privacy schemes. In this paper, we propose a visual analysis approach for formulating privacy schemes of differential privacy. Our approach supports the identification and simulation of potential privacy attacks in querying statistical results of multi-dimensional databases. We also developed a prototype system, called DPKnob, which integrates multiple coordinated views. DPKnob not only allows users to interactively assess and explore privacy exposure risks by browsing high-risk attacks, but also facilitates an iterative process for formulating and optimizing privacy schemes based on differential privacy. This iterative process allows users to compare different schemes, refine their expectations of privacy and utility, and ultimately establish a well-balanced privacy scheme. The effectiveness of this study is verified by a user study and two case studies with real-world datasets.

With the incredible growth of the scale and complexity of datasets, creating proper visualizations for users becomes more and more challenging in large datasets. Though several visualization recommendation systems have been proposed, so far, the lack of practical engineering inputs is still a major concern regarding the usage of visualization recommendations in the industry. In this paper, we proposed AVA, an open-sourced web-based framework for Automated Visual Analytics. AVA contains both empiric-driven and insight-driven visualization recommendation methods to meet the demands of creating aesthetic visualizations and understanding expressible insights respectively. The code is available at https://github.com/antvis/AVA.

Prompt tuning and adapter tuning have shown great potential in transferring pre-trained vision-language models (VLMs) to various downstream tasks. In this work, we design a new type of tuning method, termed as regularized mask tuning, which masks the network parameters through a learnable selection. Inspired by neural pathways, we argue that the knowledge required by a downstream task already exists in the pre-trained weights but just gets concealed in the upstream pre-training stage. To bring the useful knowledge back into light, we first identify a set of parameters that are important to a given downstream task, then attach a binary mask to each parameter, and finally optimize these masks on the downstream data with the parameters frozen. When updating the mask, we introduce a novel gradient dropout strategy to regularize the parameter selection, in order to prevent the model from forgetting old knowledge and overfitting the downstream data. Experimental results on 11 datasets demonstrate the consistent superiority of our method over previous alternatives. It is noteworthy that we manage to deliver 18.73% performance improvement compared to the zero-shot CLIP via masking an average of only 2.56% parameters. Furthermore, our method is synergistic with most existing parameter-efficient tuning methods and can boost the performance on top of them. Project page can be found here.

As labor shortages and safety regulations become more prominent, the need for human-in-the-loop automation of excavators is increasing. To meet this demand, we have developed a comprehensive modeling method for the excavator arm using nonlinear optimization approaches, including a simplified model that maps the task space to the joint space, as well as an equivalent model that maps the joint space to the actuator space. These models were then used to build a feedforward-PID joint velocity controller and a joint trajectory controller combined with position feedback, which forms the core of our proposed semi-automatic control system for the excavator arm. Our deployment scheme is simple and efficient, and has been deployed on two excavators of different makes and sizes. Experiments show that our deployment scheme performs well on both excavators, with an average error of 0.05 rad/s for the velocity controller and less than 5 cm for the trajectory controller. Using our semi-automatic system, we have completed demonstration experiments for precise digging and grading operations. A demonstration video can be found at https://youtu.be/N6I0WZGSF68.

Node-link diagrams are the most popular form for graph visualization. Yet, salient information of a node-link diagram cannot be fully depicted by solely presenting the visualization. We propose to augment node-link diagrams by creating textual descriptions for interested information. We conduct an expert review and a user interview to identify six requirements of generated interpretations, including three requirements for connection extraction and three requirements for visual expression. Our solution, GraphDescriptor, generates textual descriptions with two stages: feature extraction and description generation. The first one identifies and extracts features of node-link diagrams, like node connections, visual designs, and types of graph layouts. The second stage creates a group of hierarchical sentences based on a pre-defined schema. To the best of our knowledge, our approach is the first attempt to generate textual descriptions automatically. Three use cases and the in-lab user study confirm the superiority of our approach.

A non-fungible token (NFT) is a data unit stored on the blockchain. Nowadays, more and more investors and collectors (NFT traders), who participate in transactions of NFTs, have an urgent need to assess the performance of NFTs. However, there are two challenges for NFT traders when analyzing the performance of NFT. First, the current rarity models have flaws and are sometimes not convincing. In addition, NFT performance is dependent on multiple factors, such as images (high-dimensional data), history transactions (network), and market evolution (time series). It is difficult to take comprehensive consideration and analyze NFT performance efficiently. To address these challenges, we propose NFTVis, a visual analysis system that facilitates assessing individual NFT performance. A new NFT rarity model is proposed to quantify NFTs with images. Four well-coordinated views are designed to represent the various factors affecting the performance of the NFT. Finally, we evaluate the usefulness and effectiveness of our system using two case studies and user studies.

Abstract Visualization and artificial intelligence (AI) are well-applied approaches to data analysis. On one hand, visualization can facilitate humans in data understanding through intuitive visual representation and interactive exploration. On the other hand, AI is able to learn from data and implement bulky tasks for humans. In complex data analysis scenarios, like epidemic traceability and city planning, humans need to understand large-scale data and make decisions, which requires complementing the strengths of both visualization and AI. Existing studies have introduced AI-assisted visualization as AI4VIS and visualization-assisted AI as VIS4AI. However, how can AI and visualization complement each other and be integrated into data analysis processes are still missing. In this paper, we define three integration levels of visualization and AI. The highest integration level is described as the framework of VIS+AI, which allows AI to learn human intelligence from interactions and communicate with humans through visual interfaces. We also summarize future directions of VIS+AI to inspire related studies.

In wireless sensor networks, unmanned aerial vehicles (UAVs) can be employed to collect data from sensor nodes (SNs) efficiently. In this article, we consider a dual-UAV-enabled (long-distance) data collection system, where one UAV is dispatched to collect data from distributed SNs, while the other UAV is employed to relay data from the data-collection UAV to a fusion center (FC) that locates far from the SNs. To shorten the time duration for the FC to collect all data, we propose to minimize the completion time of the data collection task by jointly optimizing the transmit power and bandwidth of all SNs and the UAVs, as well as the three-dimensional trajectories of the two UAVs. Instead of assuming the simplified line-of-sight UAV-ground channel model as in most existing works, we model the channels between the UAVs and SNs as well as that between the UAVs and FC by applying the practically more accurate elevation-angle-dependent Rician fading channel model. The resulting optimization problem is nonconvex and thus is difficult to solve in general. Nevertheless, we propose an algorithm to solve it efficiently by using the techniques of block coordinate descent, slack variable substitution, and successive convex approximation. Simulation results show that our proposed algorithm can achieve higher communication efficiency than other benchmark schemes and greatly reduce the task completion time for data collection.

Automatic visualization generates meaningful visualizations to support data analysis and pattern finding for novice or casual users who are not familiar with visualization design. Current automatic visualization approaches adopt mainly aggregation and filtering to extract patterns from the original data. However, these limited data transformations fail to capture complex patterns such as clusters and correlations. Although recent advances in feature engineering provide the potential for more kinds of automatic data transformations, the auto-generated transformations lack explainability concerning how patterns are connected with the original features. To tackle these challenges, we propose a novel explainable recommendation approach for extended kinds of data transformations in automatic visualization. We summarize the space of feasible data transformations and measures on explainability of transformation operations with a literature review and a pilot study, respectively. A recommendation algorithm is designed to compute optimal transformations, which can reveal specified types of patterns and maintain explainability. We demonstrate the effectiveness of our approach through two cases and a user study.

Nature has been a plentiful source of materials, replenishment, inspiration, and creativity. Nature collage, as a crafting technique, is a fun and educational activity for children to explore nature and engage their creativity. However, the raw material collection is limited to static things such as leaves, ignoring inspiration from nature sounds and dynamic elements such as babbling creeks. Using a mobile application, we hope to encourage children's creativity by renewing collage materials collection and careful observation in nature. To explore this, we conducted a formative study with children (N=20) and a design workshop with experts (N=6) to formulate NaCanva, an AI-assisted multi-modal collage creation system for children. Drawing on the interactivity between children and nature, NaCanva enables the multi-modal material collection, including images, sound, and videos, which differs our system from traditional collages. We validated this system with a between-subject user study (N=30), and the results suggested that NaCanva unleashes children's creativity in nature collage creation by enhancing children's multidimensional observation and engagement in nature.

Numerous graph clustering methods have been proposed to explore aggregation structures across multiple graphs. In these methods, single-graph features are merely considered or multigraph features are simply weighted, which are insufficient for the construction of reasonable multiple graph clustering features, since the association information between pairwise graphs is ignored and the varied local correlations might influence the clustering preference. Thus, we propose an interactive multiple graph clustering model, iMGC, in this article, to achieve reasonable multiple graph clustering features, which cannot only express multiple relationships, but also preserve associations of nodes across multiple graphs. First, a unified graph matrix is constructed with the combination of structural differences quantified by graph representation learning, which is further optimized by minimizing the difference of structural characteristics between it and each single graph matrix. Thus, multiple relationships are well integrated and expressed, while the varied local correlations within different graphs are also balanced in the unified graph matrix. Then, a constrained Laplacian rank is applied on the unified graph matrix to generate the unified clustering result directly, which is able to preserve association features across multiple graphs. Furthermore, we provide a set of visualization and interaction interfaces, enabling users to intuitively optimize and evaluate the multiple graph clustering features, and interactively explore the multiple graphs. Case studies and quantitative comparisons based on real-world datasets have demonstrated the effectiveness of iMGC in the clustering performance from various perspectives and exploration of multiple graphs.

Numerous sampling strategies have been proposed to reduce the visual clutter of large-scale geographical point data visualization, which focus on the preservation of original data features, such as randomness, spatial distribution, and associated relationship. However, user preferences and demands are not taken into account in the course of sampling, which will lead to the sampled results deviating from user requirements and impede personalized geospatial analysis in specific application scenarios. In this article, we propose a user-driven sampling model for the visual abstraction of the large-scale geographical point data based on convolutional neural networks (CNN). First, a blue noise sampling model is applied to partition the geographical space into local areas, and a set of visual interfaces are designed to present the data features of those points in the local areas, enabling users to visually select representative points according to their requirements. Then, user preferences are quantified with a CNN model based on the eigenvectors of the representative points, which are further utilized to guide the sampling courses of the other local areas. Thus, all the sampled points will retain the spatial distribution of original data points and fulfill the user preference as far as possible. In addition, we implement a visualization framework to integrate manual point selection, CNN training, automatic point sampling, and visual comparison, allowing users to easily obtain and evaluate the sampled points from the perspectives of data analysis and user requirements. Quantitative comparisons and case studies based on real-world datasets are conducted to demonstrate the effectiveness of our sampling model in the preservation of user preferences and visual exploration of large-scale geospatial point data.

Patterns in charts refer to interesting visual features or forms. Identifying patterns not only helps analysts understand the ‘shape’ of the data but also supports better and faster decision-making. Existing solutions for identifying patterns in charts require a large number of labeled data instances, making it intractable without user supervision. In this paper, we propose ChartNavigator, an interactive pattern identification and annotation framework for unlabeled visualization charts. ChartNavigator leverages a novel chart-sensitive deep factor model to map patterns into a low-dimensional factor representation space, and facilitates rich analysis with the derived representations. We design and implement a visual interface to support efficient identification and annotation of potential patterns in charts. Evaluations with multiple datasets show that our approach outperforms the baseline models in identifying and annotating patterns

Recent years have witnessed rapid advances in autonomous driving technologies. Autonomous vehicles are more likely to be accepted if they drive comfortably to avoid potholes, bumps, and other road damage conditions, especially when there are elderly and disabled passengers on board. Traditionally, sensing road damage conditions is either labor-intensive by field investigation and reporting, or inaccurate by surveillance cameras and driving recorders due to limited perspective. In this paper, we propose GoComfort, a crowdsensing-based framework to provide low-cost and fine-grained comfortable navigation for autonomous vehicles with road damage identification leveraging high-precision road sensing data. First, we propose to exploit city-wide autonomous vehicle fleets as crowdsensing participants, and employ an edge-cloud-hybrid computing paradigm to efficiently collect high-precision road damage-related data, including 3D LiDAR point clouds and street view images. Second, we design an accurate road damage identification model fusing spatial structures of point clouds and texture features of street view images, and use an active learning-based method to address the sparse labels issue. Finally, we devise two comfortable navigation scenarios, i.e., fine-grained road damage avoidance and coarse-grained city-wide navigation, and propose a hierarchical road damage assessment diagram for comfortable route planning. Experiments using real-world road sensing data in Xiamen, China show that our approach identifies road damage conditions with an accuracy of 87.5%, and achieves a user acceptance rate of 93.8% in riding comfort evaluation, outperforming the state-of-the-art baselines.

We present a novel privacy preservation strategy for aggregated visual query of decentralized data. The key idea is to imitate the flowchart of the federated learning framework, and reformulate the visualization process within a federated infrastructure. The federation of visualization is fulfilled by leveraging a shared global module that composes the encrypted externalizations of transformed visual features of data pieces in local modules. We design two implementations of federated visualization: a prediction-based scheme, and a query-based scheme. We demonstrate the effectiveness of our approach with a set of visual forms, and verify its robustness with evaluations. We report the value of federated visualization in real scenarios with an expert review.

Horizontal federated learning (HFL) enables distributed clients to train a shared model and keep their data privacy. In training high-quality HFL models, the data heterogeneity among clients is one of the major concerns. However, due to the security issue and the complexity of deep learning models, it is challenging to investigate data heterogeneity across different clients. To address this issue, based on a requirement analysis we developed a visual analytics tool, HetVis, for participating clients to explore data heterogeneity. We identify data heterogeneity through comparing prediction behaviors of the global federated model and the stand-alone model trained with local data. Then, a context-aware clustering of the inconsistent records is done, to provide a summary of data heterogeneity. Combining with the proposed comparison techniques, we develop a novel set of visualizations to identify heterogeneity issues in HFL. We designed three case studies to introduce how HetVis can assist client analysts in understanding different types of heterogeneity issues. Expert reviews and a comparative study demonstrate the effectiveness of HetVis.

Multiple-view (MV) representations enabling multi-perspective exploration of large and complex data are often employed on 2D displays. The technique also shows great potential in addressing complex analytic tasks in immersive visualization. However, although useful, the design space of MV representations in immersive visualization lacks in deep exploration. In this paper, we propose a new perspective to this line of research, by examining the effects of view layout for MV representations on situated analytics. Specifically, we disentangle situated analytics in perspectives of situatedness regarding spatial relationship between visual representations and physical referents, and analytics regarding cross-view data analysis including filtering, refocusing, and connecting tasks. Through an in-depth analysis of existing layout paradigms, we summarize design trade-offs for achieving high situatedness and effective analytics simultaneously. We then distill a list of design requirements for a desired layout that balances situatedness and analytics, and develop a prototype system with an automatic layout adaptation method to fulfill the requirements. The method mainly includes a cylindrical paradigm for egocentric reference frame, and a force-directed method for proper view-view, view-user, and view-referent proximities and high view visibility. We conducted a formal user study that compares layouts by our method with linked and embedded layouts. Quantitative results show that participants finished filtering- and connecting-centered tasks significantly faster with our layouts, and user feedback confirms high usability of the prototype system.

We propose a contrastive dimensionality reduction approach (CDR) for interactive visual cluster analysis. Although dimensionality reduction of high-dimensional data is widely used in visual cluster analysis in conjunction with scatterplots, there are several limitations on effective visual cluster analysis. First, it is non-trivial for an embedding to present clear visual cluster separation when keeping neighborhood structures. Second, as cluster analysis is a subjective task, user steering is required. However, it is also non-trivial to enable interactions in dimensionality reduction. To tackle these problems, we introduce contrastive learning into dimensionality reduction for high-quality embedding. We then redefine the gradient of the loss function to the negative pairs to enhance the visual cluster separation of embedding results. Based on the contrastive learning scheme, we employ link-based interactions to steer embeddings. After that, we implement a prototype visual interface that integrates the proposed algorithms and a set of visualizations. Quantitative experiments demonstrate that CDR outperforms existing techniques in terms of preserving correct neighborhood structures and improving visual cluster separation. The ablation experiment demonstrates the effectiveness of gradient redefinition. The user study verifies that CDR outperforms t-SNE and UMAP in the task of cluster identification. We also showcase two use cases on real-world datasets to present the effectiveness of link-based interactions.

Data workers use various scripting languages for data transformation, such as SAS, R, and Python. However, understanding intricate code pieces requires advanced programming skills, which hinders data workers from grasping the idea of data transformation at ease. Program visualization is beneficial for debugging and education and has the potential to illustrate transformations intuitively and interactively. In this article, we explore visualization design for demonstrating the semantics of code pieces in the context of data transformation. First, to depict individual data transformations, we structure a design space by two primary dimensions, i.e., key parameters to encode and possible visual channels to be mapped. Then, we derive a collection of 23 glyphs that visualize the semantics of transformations. Next, we design a pipeline, named Somnus, that provides an overview of the creation and evolution of data tables using a provenance graph. At the same time, it allows detailed investigation of individual transformations. User feedback on Somnus is positive. Our study participants achieved better accuracy with less time using Somnus, and preferred it over carefully-crafted textual description. Further, we provide two example applications to demonstrate the utility and versatility of Somnus.

The development of digitized humanity information provides a new perspective on data-oriented studies of history. Many previous studies have ignored uncertainty in the exploration of historical figures and events, which has limited the capability of researchers to capture complex processes associated with historical phenomena. We propose a visual reasoning system to support visual reasoning of uncertainty associated with spatio-temporal events of historical figures based on data from the China Biographical Database Project. We build a knowledge graph of entities extracted from a historical database to capture uncertainty generated by missing data and error. The proposed system uses an overview of chronology, a map view, and an interpersonal relation matrix to describe and analyse heterogeneous information of events. The system also includes uncertainty visualization to identify uncertain events with missing or imprecise spatio-temporal information. Results from case studies and expert evaluations suggest that the visual reasoning system is able to quantify and reduce uncertainty generated by the data.

In history research, cohort analysis seeks to identify social structures and figure mobilities by studying the group-based behavior of historical figures. Prior works mainly employ automatic data mining approaches, lacking effective visual explanation. In this paper, we present CohortVA, an interactive visual analytic approach that enables historians to incorporate expertise and insight into the iterative exploration process. The kernel of CohortVA is a novel identification model that generates candidate cohorts and constructs cohort features by means of pre-built knowledge graphs constructed from large-scale history databases. We propose a set of coordinated views to illustrate identified cohorts and features coupled with historical events and figure profiles. Two case studies and interviews with historians demonstrate that CohortVA can greatly enhance the capabilities of cohort identifications, figure authentications, and hypothesis generation.

Data privacy is an essential issue in publishing data visualizations. However, it is challenging to represent multiple data patterns in privacy-preserving visualizations. The prior approaches target specific chart types or perform an anonymization model uniformly without considering the importance of data patterns in visualizations. In this paper, we propose a visual analytics approach that facilitates data custodians to generate multiple private charts while maintaining user-preferred patterns. To this end, we introduce pattern constraints to model users' preferences over data patterns in the dataset and incorporate them into the proposed Bayesian network-based Differential Privacy (DP) model PriVis. A prototype system, DPVisCreator, is developed to assist data custodians in implementing our approach. The effectiveness of our approach is demonstrated with quantitative evaluation of pattern utility under the different levels of privacy protection, case studies, and semi-structured expert interviews.

Collusive fraud, in which multiple fraudsters collude to defraud health insurance funds, threatens the operation of the healthcare system. However, existing statistical and machine learning-based methods have limited ability to detect fraud in the scenario of health insurance due to the high similarity of fraudulent behaviors to normal medical visits and the lack of labeled data. To ensure the accuracy of the detection results, expert knowledge needs to be integrated with the fraud detection process. By working closely with health insurance audit experts, we propose FraudAuditor, a three-stage visual analytics approach to collusive fraud detection in health insurance. Specifically, we first allow users to interactively construct a co-visit network to holistically model the visit relationships of different patients. Second, an improved community detection algorithm that considers the strength of fraud likelihood is designed to detect suspicious fraudulent groups. Finally, through our visual interface, users can compare, investigate, and verify suspicious patient behavior with tailored visualizations that support different time scales. We conducted case studies in a real-world healthcare scenario, i.e., to help locate the actual fraud group and exclude the false positive group. The results and expert feedback proved the effectiveness and usability of the approach.

The past decade has witnessed rapid progress in AI research since the breakthrough in deep learning. AI technology has been applied in almost every field; therefore, technical and non-technical end-users must understand these technologies to exploit them. However existing materials are designed for experts, but non-technical users need appealing materials that deliver complex ideas in easy-to-follow steps. One notable tool that fits such a profile is scrollytelling, an approach to storytelling that provides readers with a natural and rich experience at the reader’s pace, along with in-depth interactive explanations of complex concepts. Hence, this work proposes a novel visualization design for creating a scrollytelling that can effectively explain an AI concept to non-technical users. As a demonstration of our design, we created a scrollytelling to explain the Siamese Neural Network for the visual similarity matching problem. Our approach helps create a visualization valuable for a short-timeline situation like a sales pitch. The results show that the visualization based on our novel design helps improve non-technical users’ perception and machine learning concept knowledge acquisition compared to traditional materials like online articles.

One main challenge for simplifying node-link diagrams of large-scale social networks lies in that simplified graphs generally contain dense subgroups or cohesive subgraphs. Graph triangles quantify the solid and stable relationships that maintain cohesive subgraphs. Understanding the mechanism of triangles within cohesive subgraphs contributes to illuminating patterns of connections within social networks. However, prior works can hardly handle and visualize triangles in cohesive subgraphs. In this paper, we propose a triangle-based graph simplification approach that can filter and visualize cohesive subgraphs by leveraging a triangle-connectivity called k-truss and a force-directed algorithm. We design and implement TriGraph, a web-based visual interface that provides detailed information for exploring and analyzing social networks. Quantitive comparions with existing methods, two case studies on real-world datasets, and the feedback from domain experts demonstrate the effectiveness of TriGraph.

Modeling multivariate time series (MTS) is critical in modern intelligent systems. The accurate forecast of MTS data is still challenging due to the complicated latent variable correlation. Recent works apply the Graph Neural Networks (GNNs) to the task, with the basic idea of representing the correlation as a static graph. However, predicting with a static graph causes significant bias because the correlation is time-varying in the real-world MTS data. Besides, there is no gap analysis between the actual correlation and the learned one in their works to validate the effectiveness. This paper proposes a temporal polynomial graph neural network (TPGNN) for accurate MTS forecasting, which represents the dynamic variable correlation as a temporal matrix polynomial in two steps. First, we capture the overall correlation with a static matrix basis. Then, we use a set of time-varying coefficients and the matrix basis to construct a matrix polynomial for each time step. The constructed result empirically captures the precise dynamic correlation of six synthetic MTS datasets generated by a non-repeating random walk model. Moreover, the theoretical analysis shows that TPGNN can achieve perfect approximation under a commutative condition. We conduct extensive experiments on two traffic datasets with prior structure and four benchmark datasets. The results indicate that TPGNN achieves the state-of-the-art on both short-term and long-term MTS forecastings.

Few-shot segmentation~(FSS) aims at performing semantic segmentation on novel classes given a few annotated support samples. With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised segmentation framework: Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head. Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework. This paper revives the straightforward framework of ``feature extractor $+$ linear classification head'' and proposes a novel Feature-Proxy Transformer (FPTrans) method, in which the ``proxy'' is the vector representing a semantic class in the linear classification head. FPTrans has two keypoints for learning discriminative features and representative proxies: 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from bottom to top layers using a novel prompting strategy. 2) FPTrans uses multiple local background proxies (instead of a single one) because the background is not homogeneous and may contain some novel foreground regions. These two keypoints are easily integrated into the vision transformer backbone with the prompting mechanism in the transformer. Given the learned features and proxies, FPTrans directly compares their cosine similarity for segmentation. Although the framework is straightforward, we show that FPTrans achieves competitive FSS accuracy on par with state-of-the-art decoder-based methods.

Analyzing the vulnerability of power systems in cascading failures is generally regarded as a challenging problem. Although existing studies can extract some critical rules, they fail to capture the complex subtleties under different operational conditions. In recent years, several deep learning methods have been applied to address this issue. However, most of the existing deep learning methods consider only the grid topology of a power system in terms of topological connections, but do not encompass a power system’s spatial information such as the electrical distance to increase the accuracy in the process of graph convolution. In this paper, we construct a novel power-weighted line graph that uses power system topology and spatial information to optimize the edge weight assignment of the line graph. Then we propose a multi-graph convolutional network (MGCN) based on a graph classification task, which preserves a power system’s spatial correlations and captures the relationships among physical components. Our model can better handle the problem with power systems that have parallel lines, where our method can maintain desirable accuracy in modeling systems with these extra topology features. To increase the interpretability of the model, we present the MGCN using layer-wise relevance propagation and quantify the contributing factors of model classification.

Analyzing network robustness under various circumstances is generally regarded as a challenging problem. Robustness against failure is one of the essential properties of large-scale dynamic network systems such as power grids, transportation systems, communication systems, and computer networks. Due to the network diversity and complexity, many topological features have been proposed to capture specific system properties. For power grids, a popular process for improving a network’s structural robustness is via the topology design. However, most of existing methods focus on localized network metrics, such as node connectivity and edge connectivity, which do not encompass a global perspective of cascading propagation in a power grid. In this paper, we use an informative global metric algebraic connectivity because it is sensitive to the connectedness in a broader spectrum of graphs. Our process involves decreasing the average propagation in a power grid by minimizing the increase in its algebraic connectivity. We propose a topology-based greedy strategy to optimize the robustness of the power grid. To evaluate the network robustness, we calculate the average propagation using MATCASC to simulate cascading line outages in power grids. Experimental results illustrate that our proposed method outperforms existing techniques.

Distributed processing and analysis of large-scale graph data remain challenging because of the high-level discrepancy among graphs. This study investigates a novel subproblem: the distributed multi-task learning on the graph, which jointly learns multiple analysis tasks from decentralized graphs. We propose a federated multi-task graph learning (FMTGL) framework to solve the problem within a privacy-preserving and scalable scheme. Its core is an innovative data-fusion mechanism and a low-latency distributed optimization method. The former captures multi-source data relatedness and generates universal task representation for local task analysis. The latter enables the quick update of our framework with gradients sparsification and tree-based aggregation. As a theoretical result, the proposed optimization method has a convergence rate interpolates between \( \mathcal {O}(1/T) \) and \( \mathcal {O}(1/\sqrt {T}) \) , up to logarithmic terms. Unlike previous studies, our work analyzes the convergence behavior with adaptive stepsize selection and non-convex assumption. Experimental results on three graph datasets verify the effectiveness and scalability of FMTGL.

Neurofibromatosis type 1 (NF1) is an autosomal dominant tumor predisposition syndrome that involves the central and peripheral nervous systems. Accurate detection and segmentation of neurofibromas are essential for assessing tumor burden and longitudinal tumor size changes. Automatic convolutional neural networks (CNNs) are sensitive and vulnerable as tumors' variable anatomical location and heterogeneous appearance on MRI. In this study, wepropose deep interactive networks (DINs) to address the above limitations. User interactions guide the model to recognize complicated tumors and quickly adapt to heterogeneous tumors. We introduce a simple but effective Exponential Distance Transform (ExpDT) that converts user interactions into guide maps regarded as the spatial and appearance prior. Comparing with popular Euclidean and geodesic distances, ExpDT is more robust to various image sizes, which reserves the distribution of interactive inputs. Furthermore, to enhance the tumor-related features, we design a deep interactive module to propagate the guides into deeper layers. We train and evaluate DINs on three MRI data sets from NF1 patients. The experiment results yield significant improvements of 44% and 14% in DSC comparing with automated and other interactive methods, respectively. We also experimentally demonstrate the efficiency of DINs in reducing user burden when comparing with conventional interactive methods.

The spatial time series generated by city sensors allow us to observe urban phenomena like environmental pollution and traffic congestion at an unprecedented scale. However, recovering causal relations from these observations to explain the sources of urban phenomena remains a challenging task because these causal relations tend to be time-varying and demand proper time series partitioning for effective analyses. The prior approaches extract one causal graph given long-time observations, which cannot be directly applied to capturing, interpreting, and validating dynamic urban causality. This paper presents Compass, a novel visual analytics approach for in-depth analyses of the dynamic causality in urban time series. To develop Compass, we identify and address three challenges: detecting urban causality, interpreting dynamic causal relations, and unveiling suspicious causal relations. First, multiple causal graphs over time among urban time series are obtained with a causal detection framework extended from the Granger causality test. Then, a dynamic causal graph visualization is designed to reveal the time-varying causal relations across these causal graphs and facilitate the exploration of the graphs along the time. Finally, a tailored multi-dimensional visualization is developed to support the identification of spurious causal relations, thereby improving the reliability of causal analyses. The effectiveness of Compass is evaluated with two case studies conducted on the real-world urban datasets, including the air pollution and traffic speed datasets, and positive feedback was received from domain experts.

Breaking news and first-hand reports often trend on social media platforms before traditional news outlets cover them. The real-time analysis of posts on such platforms can reveal valuable and timely insights for journalists, politicians, business analysts, and first responders, but the high number and diversity of new posts pose a challenge. In this work, we present an interactive system that enables the visual analysis of streaming social media data on a large scale in real-time. We propose an efficient and explainable dynamic clustering algorithm that powers a continuously updated visualization of the current thematic landscape as well as detailed visual summaries of specific topics of interest. Our parallel clustering strategy provides an adaptive stream with a digestible but diverse selection of recent posts related to relevant topics. We also integrate familiar visual metaphors that are highly interlinked for enabling both explorative and more focused monitoring tasks. Analysts can gradually increase the resolution to dive deeper into particular topics. In contrast to previous work, our system also works with non-geolocated posts and avoids extensive preprocessing such as detecting events. We evaluated our dynamic clustering algorithm and discuss several use cases that show the utility of our system.

This February 2022 issue of theIEEE Transactions on Visualization and Computer Graphics (TVCG)contains the proceedings of IEEE VIS 2021, held online on October 24-29, 2021, with General Chairs from Tulane University and Universidade de Sao Paulo. With IEEE VIS 2021, the conference series is in its 32nd year.

Collecting and analyzing anonymous personal information is required as a part of data analysis processes, such as medical diagnosis and restaurant recommendation. Such data should ostensibly be stored so that specific individual information cannot be disclosed. Unfortunately, inference attacks-integrating background knowledge and intelligent models-hinder classic sanitization techniques like syntactic anonymity and differential privacy from exhaustively protecting sensitive information. As a solution, we introduce a three-stage approach empowered within a visual interface, which depicts underlying inference behaviors via a Bayesian Network and supports a customized defense against inference attacks from unknown adversaries. In particular, our approach visually explains the process details of the underlying privacy preserving models, allowing users to verify if the results sufficiently satisfy the requirements of privacy preservation. We demonstrate the effectiveness of our approach through two case studies and expert reviews.

A graph is an abstract model that represents relations among entities, for example, the interactions between characters in a novel. A background story endows entities and relations with real-world meanings and describes the semantics and context of the abstract model, for example, the actual story that the novel presents. Considering practical experience and prior research, human viewers who are familiar with the background story of a graph and those who do not know the background story may perceive the same graph differently. However, no previous research has adequately addressed this problem. This research article thus presents an evaluation that investigated the effects of background stories on graph perception. Three hypotheses that focused on the role of visual focus areas, graph structure identification, and mental model formation on graph perception were formulated and guided three controlled experiments that evaluated the hypotheses using real-world graphs with background stories. An analysis of the resulting experimental data, which compared the performance of participants who read and did not read the background stories, obtained a set of instructive findings. First, having knowledge about a graph's background story influences participants' focus areas during interactive graph explorations. Second, such knowledge significantly affects one's ability to identify community structures but not high degree and bridge structures. Third, this knowledge influences graph recognition under blurred visual conditions. These findings can bring new considerations to the design of storytelling visualizations and interactive graph explorations.

Data-driven event analysis has gradually become the backbone of modern competitive sports analysis. Increasingly, competitive sports data analysis tasks use computer vision and machine learning models for intelligent data analysis. Existing sports visualization systems focus on the player–team data visualization, which is not intuitive enough for team season win–loss data and game time–series data visualization, and neglects the prediction of all-star players. This study used an interactive visualization system, the NPIPVis is designed with parallel aggregated ordered hypergraph dynamic hypergraphs, Calliope visualization data story technology, and iStoryline narrative visualization technology to visualize the regular statistics and game time data of players and teams. NPIPVis includes dynamic hypergraphs of team wins and losses and game plot narrative visualization components. In addition, an integrated learning-based all-star player prediction model SRR-voting, which starts from the existing minority and majority samples, was proposed using synthetic minority oversampling technique and RandomUnderSampler methods to generate and eliminate samples of a certain size to balance the number of all-star and average players in the datasets. A random forest algorithm was introduced next to extract and construct the features of players and combined with the voting integrated model to predict the all-star players, using GridSearchCV, to optimize the hyperparameters of each model in integrated learning and then combined with fivefold cross-validation to improve the generalization ability of the model, and, finally, the Shap model was introduced to enhance the interpretability of the model. The experimental results of comparing the SRR-voting model with six common models show that accuracy and F1-score and recall metrics are significantly improved, which verifies the effectiveness and practicality of the SRR-voting model. This paper combines data visualization and machine learning to design a National Basketball Association data visualization system to help the general audience visualize game data and predict all-star players which can be extended to other sports events or related fields.

Semi-supervised variational autoencoders (VAEs) have obtained strong results, but have also encountered the challenge that good ELBO values do not always imply accurate inference results.In this paper, we investigate and propose two causes of this problem: (1) The ELBO objective cannot utilize the label information directly. (2) A bottleneck value exists, and continuing to optimize ELBO after this value will not improve inference accuracy. On the basis of the experiment results, we propose SHOT-VAE to address these problems without introducing additional prior knowledge. The SHOT-VAE offers two contributions: (1) A new ELBO approximation named smooth-ELBO that integrates the label predictive loss into ELBO. (2) An approximation based on optimal interpolation that breaks the ELBO value bottleneck by reducing the margin between ELBO and the data likelihood. The SHOT-VAE achieves good performance with 25.30% error rate on CIFAR-100 with 10k labels and reduces the error rate to 6.11% on CIFAR-10 with 4k labels.

Data-driven scrollytelling has become a prevalent way of visual communication because of its comprehensive delivery of perspectives derived from the data. However, creating an expressive scrollytelling story requires both data and design literacy and is time-consuming. As a result, scrollytelling has been mainly used only by professional journalists to disseminate opinions. In this paper, we present an automatic method to generate expressive scrollytelling visualization, which can present easy-to-understand data facts through a carefully arranged sequence of views. The method first enumerates data facts of a given dataset, and scores and organizes them. The facts are further assembled, sequenced into a story, with reader input taken into consideration. Finally, visual graphs, transitions, and text descriptions are generated to synthesize the scrollytelling visualization. In this way, non-professionals can easily explore and share interesting perspectives from selected data attributes and fact types. We demonstrate the effectiveness and usability of our method through both use cases and an in-lab user study.

Conventional unsupervised multi-source domain adaptation (UMDA) methods assume all source domains can be accessed directly. However, this assumption neglects the privacy-preserving policy, where all the data and computations must be kept decentralized. There exist three challenges in this scenario: (1) Minimizing the domain distance requires the pairwise calculation of the data from the source and target domains, while the data on the source domain is not available. (2) The communication cost and privacy security limit the application of existing UMDA methods, such as the domain adversarial training. (3) Since users cannot govern the data quality, the irrelevant or malicious source domains are more likely to appear, which causes negative transfer. To address the above problems, we propose a privacy-preserving UMDA paradigm named Knowledge Distillation based Decentralized Domain Adaptation (KD3A), which performs domain adaptation through the knowledge distillation on models from different source domains. The extensive experiments show that KD3A significantly outperforms state-of-the-art UMDA approaches. Moreover, the KD3A is robust to the negative transfer and brings a 100x reduction of communication cost compared with other decentralized UMDA methods.

三元空间大数据一般定义为由其定义领域 (包括数据、 对象、 任务、 应用场景、 主体等) 所有元素组成的集合. 可视分析是一种新兴的人在回路大数据分析范式, 可利用人类感知提高人类认知效率. 本文探讨三元空间大数据跨域可视化分析, 强调三元空间大数据跨域性带来的新挑战——数据、 主题和任务域, 并提出一个新的可视分析模型和一套方法来应对这些挑战.

Quality control is of vital importance in compressing three-dimensional (3D) medical imaging data. Optimal compression parameters need to be determined based on the specific quality requirement. In high efficiency video coding (HEVC), regarded as the state-of-the-art compression tool, the quantization parameter (QP) plays a dominant role in controlling quality. The direct application of a video-based scheme in predicting the ideal parameters for 3D medical image compression cannot guarantee satisfactory results. In this paper we propose a learning-based parameter prediction scheme to achieve efficient quality control. Its kernel is a support vector regression (SVR) based learning model that is capable of predicting the optimal QP from both video-based and structural image features extracted directly from raw data, avoiding time-consuming processes such as pre-encoding and iteration, which are often needed in existing techniques. Experimental results on several datasets verify that our approach outperforms current video-based quality control methods.

Federated Learning (FL) provides a powerful solution to distributed machine learning on a large corpus of decentralized data. It ensures privacy and security by performing computation on devices (which we refer to as clients) based on local data to improve the shared global model. However, the inaccessibility of the data and the invisibility of the computation make it challenging to interpret and analyze the training process, especially to distinguish potential client anomalies. Identifying these anomalies can help experts diagnose and improve FL models. For this reason, we propose a visual analytics system, VADAF, to depict the training dynamics and facilitate analyzing potential client anomalies. Specifically, we design a visualization scheme that supports massive training dynamics in the FL environment. Moreover, we introduce an anomaly detection method to detect potential client anomalies, which are further analyzed based on both the client model’s visual and objective estimation. Three case studies have demonstrated the effectiveness of our system in understanding the FL training process and supporting abnormal client detection and analysis.

With the rapid development of sensing technologies, massive spatiotemporal data have been acquired from the urban space with respect to different domains, such as transportation and environment. Numerous co-occurrence patterns (e.g., traffic speed <; 10km/h, weather = foggy, and air quality = unhealthy) between the transportation data and other types of data can be obtained with given spatiotemporal constraints (e.g., within 3 kilometers and lasting for 2 hours) from these heterogeneous data sources. Such patterns present valuable implications for many urban applications, such as traffic management, pollution diagnosis, and transportation planning. However, extracting and understanding these patterns is beyond manual capability because of the scale, diversity, and heterogeneity of the data. To address this issue, a novel visual analytics system called CorVizor is proposed to identify and interpret these co-occurrence patterns. CorVizor comprises two major components. The first component is a co-occurrence mining framework involving three steps, namely, spatiotemporal indexing, co-occurring instance generation, and pattern mining. The second component is a visualization technique called CorView that implements a level-of-detail mechanism by integrating tailored visualizations to depict the extracted spatiotemporal co-occurrence patterns. The case studies and expert interviews are conducted to demonstrate the effectiveness of CorVizor.

Increasing attention is being given to the modeling and simulation of traffic flow and crowd movement, two phenomena that both deal with interactions between pedestrians and cars in many situations. In particular, crowd simulation is important for understanding mobility and transportation patterns. In this paper, we propose an emotion-based crowd simulation model integrating physical strength consumption. Inspired by the theory of “the devoted actor,” the movements of each individual in our model are determined by modeling the influence of physical strength consumption and the emotion of panic. In particular, human physical strength consumption is computed using a physics-based numerical method. Inspired by the James-Lange theory, panic levels are estimated by means of an enhanced emotional contagion model that leverages the inherent relationship between physical strength consumption and panic. To the best of our knowledge, our model is the first method integrating physical strength consumption into an emotion-based crowd simulation model by exploiting the relationship between physical strength consumption and emotion. We highlight the performance on different scenarios and compare the resulting behaviors with real-world video sequences. Our approach can reliably predict changes in physical strength consumption and panic levels of individuals in an emergency situation.

Research on intelligent vehicles has been popular in the past decade. To fill the gap between automatic approaches and man-machine control systems, it is indispensable to integrate visual human-computer interactions (VHCIs) into intelligent vehicles systems. In this article, we review existing studies on VHCI in intelligent vehicles from three aspects: 1) visual intelligence; 2) decision making; and 3) macro deployment. We discuss how VHCI evolves in intelligent vehicles and how it enhances the capability of intelligent vehicles. We present several simulated scenarios and cases for future intelligent transportation system.

Visualizing massive scale human movement in cities plays an important role in solving many of the problems that modern cities face (e.g., traffic optimization, business site configuration). In this article, we study a big mobile location dataset that covers millions of city residents, but is temporally sparse on the trajectory of individual user. Mapping sparse trajectories to illustrate population movement poses several challenges from both analysis and visualization perspectives. In the literature, there are a few techniques designed for sparse trajectory visualization; yet they do not consider trajectories collected from mobile apps that possess long-tailed sparsity with record intervals as long as hours. This article introduces UrbanMotion, a visual analytics system that extends the original wind map design by supporting map-matched local movements, multi-directional population flows, and population distributions. Effective methods are proposed to extract and aggregate population movements from dense parts of the trajectories leveraging their long-tailed sparsity. Both characteristic and anomalous patterns are discovered and visualized. We conducted three case studies, one comparative experiment, and collected expert feedback in the application domains of commuting analysis, event detection, and business site configuration. The study result demonstrates the significance and effectiveness of our system in helping to complete key analytics tasks for urban users.

Small multiples are miniature representations of visual information used generically across many domains. Handling large numbers of small multiples imposes challenges on many analytic tasks like inspection, comparison, navigation, or annotation. To address these challenges, we developed a framework and implemented a library called PILlNG.JS for designing interactive piling interfaces. Based on the piling metaphor, such interfaces afford flexible organization, exploration, and comparison of large numbers of small multiples by interactively aggregating visual objects into piles. Based on a systematic analysis of previous work, we present a structured design space to guide the design of visual piling interfaces. To enable designers to efficiently build their own visual piling interfaces, PILlNG.JS provides a declarative interface to avoid having to write low-level code and implements common aspects of the design space. An accompanying GUI additionally supports the dynamic configuration of the piling interface. We demonstrate the expressiveness of PILlNG.JS with examples from machine learning, immunofluorescence microscopy, genomics, and public health.

Dashboard visualizations are widely used in data-intensive applications such as business intelligence, operation monitoring, and urban planning. However, existing visualization authoring tools are inefficient in the rapid prototyping of dashboards because visualization expertise and user intention need to be integrated. We propose a novel approach to rapid conceptualization that can construct dashboard templates from exemplars to mitigate the burden of designing, implementing, and evaluating dashboard visualizations. The kernel of our approach is a novel deep learning-based model that can identify and locate charts of various categories and extract colors from an input image or sketch. We design and implement a web-based authoring tool for learning, composing, and customizing dashboard visualizations in a cloud computing environment. Examples, user studies, and user feedback from real scenarios in Alibaba Cloud verify the usability and efficiency of the proposed approach.

We design and evaluate a novel layout fine-tuning technique for node-link diagrams that facilitates exemplar-based adjustment of a group of substructures in batching mode. The key idea is to transfer user modifications on a local substructure to other substructures in the entire graph that are topologically similar to the exemplar. We first precompute a canonical representation for each substructure with node embedding techniques and then use it for on-the-fly substructure retrieval. We design and develop a light-weight interactive system to enable intuitive adjustment, modification transfer, and visual graph exploration. We also report some results of quantitative comparisons, three case studies, and a within-participant user study.

In soccer, passing is the most frequent interaction between players and plays a significant role in creating scoring chances. Experts are interested in analyzing players' passing behavior to learn passing tactics, i.e., how players build up an attack with passing. Various approaches have been proposed to facilitate the analysis of passing tactics. However, the dynamic changes of a team's employed tactics over a match have not been comprehensively investigated. To address the problem, we closely collaborate with domain experts and characterize requirements to analyze the dynamic changes of a team's passing tactics. To characterize the passing tactic employed for each attack, we propose a topic-based approach that provides a high-level abstraction of complex passing behaviors. Based on the model, we propose a glyph-based design to reveal the multi-variate information of passing tactics within different phases of attacks, including player identity, spatial context, and formation. We further design and develop PassVizor, a visual analytics system, to support the comprehensive analysis of passing dynamics. With the system, users can detect the changing patterns of passing tactics and examine the detailed passing process for evaluating passing tactics. We invite experts to conduct analysis with PassVizor and demonstrate the usability of the system through an expert interview.

Deep learning methods are being increasingly used for urban traffic prediction where spatiotemporal traffic data is aggregated into sequentially organized matrices that are then fed into convolution-based residual neural networks. However, the widely known modifiable areal unit problem within such aggregation processes can lead to perturbations in the network inputs. This issue can significantly destabilize the feature embeddings and the predictions - rendering deep networks much less useful for the experts. This paper approaches this challenge by leveraging unit visualization techniques that enable the investigation of many-to-many relationships between dynamically varied multi-scalar aggregations of urban traffic data and neural network predictions. Through regular exchanges with a domain expert, we design and develop a visual analytics solution that integrates 1) a Bivariate Map equipped with an advanced bivariate colormap to simultaneously depict input traffic and prediction errors across space, 2) a Moran's I Scatterplot that provides local indicators of spatial association analysis, and 3) a Multi-scale Attribution View that arranges non-linear dot plots in a tree layout to promote model analysis and comparison across scales. We evaluate our approach through a series of case studies involving a real-world dataset of Shenzhen taxi trips, and through interviews with domain experts. We observe that geographical scale variations have important impact on prediction performances, and interactive visual exploration of dynamically varying inputs and outputs benefit experts in the development of deep traffic prediction models.

Numerous sampling strategies have been proposed to simplify large-scale networks for highly readable visualizations. It is of great challenge to preserve contextual structures formed by nodes and edges with tight relationships in a sampled graph, because they are easily overlooked during the process of sampling due to their irregular distribution and immunity to scale. In this paper, a new graph sampling method is proposed oriented to the preservation of contextual structures. We first utilize a graph representation learning (GRL) model to transform nodes into vectors so that the contextual structures in a network can be effectively extracted and organized. Then, we propose a multi-objective blue noise sampling model to select a subset of nodes in the vectorized space to preserve contextual structures with the retention of relative data and cluster densities in addition to those features of significance, such as bridging nodes and graph connections. We also design a set of visual interfaces enabling users to interactively conduct context-aware sampling, visually compare results with various sampling strategies, and deeply explore large networks. Case studies and quantitative comparisons based on real-world datasets have demonstrated the effectiveness of our method in the abstraction and exploration of large networks.

Efficient layout of large-scale graphs remains a challenging problem: the force-directed and dimensionality reduction-based methods suffer from high overhead for graph distance and gradient computation. In this paper, we present a new graph layout algorithm, called DRGraph, that enhances the nonlinear dimensionality reduction process with three schemes: approximating graph distances by means of a sparse distance matrix, estimating the gradient by using the negative sampling technique, and accelerating the optimization process through a multi-level layout scheme. DRGraph achieves a linear complexity for the computation and memory consumption, and scales up to large-scale graphs with millions of nodes. Experimental results and comparisons with state-of-the-art graph layout methods demonstrate that DRGraph can generate visually comparable layouts with a faster running time and a lower memory requirement.

Graph visualization plays an important role in several fields, such as social media networks, protein–protein interaction networks, and traffic networks. A number of visualization design tools and programming toolkits have been widely used in graph-related applications. However, a key challenge remains in the high-efficiency visualization of large-scale graph data. In this study, we present NetV.js, an open-source and WebGL-based JavaScript library that supports the fast visualization of large-scale graph data (up to 50 thousand nodes and 1 million edges) at an interactive frame rate with a commodity computer. Experimental results demonstrate that our library outperforms existing toolkits (Sigma.js, D3.js, Cytoscape.js, and Stardust.js) in terms of performance.

Authoring graph visualization poses great challenges to developers due to its high requirements on both domain knowledge and development skills. Although existing libraries and tools reduce the difficulty of generating graph visualization, there are still many challenges. We work closely with developers and formulate several design goals, then design and implement G6, a web-based library for graph visualization. It combines template-based configuration for high usability and flexible customization for high expressiveness. To enhance development efficiency, G6 proposes a range of optimizations, including state management and interaction modes. We demonstrate its capabilities through an extensive gallery, a quantitative performance evaluation, and an expert interview. G6 was first released in 2017 and has been iterated for 317 versions. It has served as a web-based library for thousands of applications and received 8312 stars on GitHub.

Song Ci is treasured in traditional Chinese culture, which indicates social and cultural evolution in ancient times. Despite the efforts by historians and litterateurs in investigating the characteristics of Song Ci, it is still unclear how to effectively distribute and promote Song Ci in the public sphere. The complexity and abstraction of Song Ci hamper the general public from closely reading, analyzing, and appreciating these excellent works. By means of a set of visual analysis methods, e.g. the spatio-temporal visualization, we exploit visual storytelling to explicitly present the latent and abstractive features of Song Ci. We apply straightway visual charts and lighten the burden of understanding the stories, in order to achieve an effective public distribution. The effectiveness and aesthetics of our work are demonstrated by a user study of three participants with different backgrounds. The result reveals that our story is effective in the distribution, understanding, and promotion of Song Ci.

This work focuses on the extremely low-light image enhancement, which aims to improve image brightness and reveal hidden information in darken areas. Recently, image enhancement approaches have yielded impressive progress. However, existing methods still suffer from three main problems: (1) low-light images usually are high-contrast. Existing methods may fail to recover images details in extremely dark or bright areas; (2) current methods cannot precisely correct the color of low-light images; (3) when the object edges are unclear, the pixel-wise loss may treat pixels of different objects equally and produce blurry images. In this paper, we propose a two-stage method called Edge-Enhanced Multi-Exposure Fusion Network (EEMEFN) to enhance extremely low-light images. In the first stage, we employ a multi-exposure fusion module to address the high contrast and color bias issues. We synthesize a set of images with different exposure time from a single image and construct an accurate normal-light image by combining well-exposed areas under different illumination conditions. Thus, it can produce realistic initial images with correct color from extremely noisy and low-light images. Secondly, we introduce an edge enhancement module to refine the initial images with the help of the edge information. Therefore, our method can reconstruct high-quality images with sharp edges when minimizing the pixel-wise loss. Experiments on the See-in-the-Dark dataset indicate that our EEMEFN approach achieves state-of-the-art performance.

Chronic kidney disease (CKD) is a widespread renal disease throughout the world. Once it develops to the advanced stage, serious complications and high risk of death will follow. Hence, early screening is crucial for the treatment of CKD. Since ultrasonography has no side effects and enables radiologists to dynamically observe the morphology and pathological features of the kidney, it is commonly used for kidney examination. In this study, we propose a novel convolutional neural network (CNN) framework named the texture branch network to screen CKD based on ultrasound images. This introduces a texture branch into a typical CNN to extract and optimize texture features. The model can automatically generate texture features and deep features from input images, and use the fused information as the basis of classification. Furthermore, we train the base part of the network by means of transfer learning, and conduct experiments on a dataset with 226 ultrasound images. Experimental results demonstrate the effectiveness of the proposed approach, achieving an accuracy of 96.01% and a sensitivity of 99.44%.

A dynamic network refers to a graph structure whose nodes and/or links dynamically change over time. Existing visualization and analysis techniques focus mainly on summarizing and revealing the primary evolution patterns of the network structure. Little work focuses on detecting anomalous changing patterns in the dynamic network, the rare occurrence of which could damage the development of the entire structure. In this study, we introduce the first visual analysis system RCAnalyzer designed for detecting rare changes of sub-structures in a dynamic network. The proposed system employs a rare category detection algorithm to identify anomalous changing structures and visualize them in the context to help oracles examine the analysis results and label the data. In particular, a novel visualization is introduced, which represents the snapshots of a dynamic network in a series of connected triangular matrices. Hierarchical clustering and optimal tree cut are performed on each matrix to illustrate the detected rare change of nodes and links in the context of their surrounding structures. We evaluate our technique via a case study and a user study. The evaluation results verify the effectiveness of our system.

In modern cities, service providers want to identify the regions that are hard to reach from multiple fire stations, a citizen wants to meet with friends in a restaurant close to everyone, and administrators want to find whether an area far from two bus stations needs a new one. Such tasks involve studying the dynamic accessibility of the urban structures over multiple geospatial and temporal constraints, which is an important topic in geographical sciences and urban transportation. In this paper, we present a new computational model and a visualization system that help domain users to interactively study the jointly constrained accessible regions, street segments, and Points of Interest (POIs). In particular, Urban Structure Accessibility Visualization system is built upon a new Min-Max Joint Set model, where specifically designed set operations not only represent the accessible regions but also compute the minimum and maximum access times to urban structures from the joint constraints. The computation and visualization are supported by a new graph model that accommodates the real-world dynamic traffic situation and the geographical settings of urban street segments and POIs. The visualization system allows the users to conveniently construct and manage accessible regions and visually explore the urban structures inside them.

The choropleth map is an essential tool for spatial data analysis. However, the underlying attribute values of a spatial unit greatly influence the statistical analyses and map classification procedures when generating a choropleth map. If the attribute values incorporate a range of uncertainty, a critical task is determining how much the uncertainty impacts both the map visualization and the statistical analysis. In this paper, we present a visual analytics system that enhances our understanding of the impact of attribute uncertainty on data visualization and statistical analyses of these data. Our system consists of a parallel coordinates-based uncertainty specification view, an impact river and impact matrix visualization for region-based and simulation-based analysis, and a dual-choropleth map and t-SNE plot for visualizing the changes in classification and spatial autocorrelation over the range of uncertainty in the attribute values. We demonstrate our system through three use cases illustrating the impact of attribute uncertainty in geographic analysis.

Visual querying is essential for interactively exploring massive trajectory data. However, the data uncertainty imposes profound challenges to fulfill advanced analytics requirements. On the one hand, many underlying data does not contain accurate geographic coordinates, e.g., positions of a mobile phone only refer to the regions (i.e., mobile cell stations) in which it resides, instead of accurate GPS coordinates. On the other hand, domain experts and general users prefer a natural way, such as using a natural language sentence, to access and analyze massive movement data. In this paper, we propose a visual analytics approach that can extract spatial-temporal constraints from a textual sentence and support an effective query method over uncertain mobile trajectory data. It is built up on encoding massive, spatially uncertain trajectories by the semantic information of the POls and regions covered by them, and then storing the trajectory documents in text database with an effective indexing scheme. The visual interface facilitates query condition specification, situation-aware visualization, and semantic exploration of large trajectory data. Usage scenarios on real-world human mobility datasets demonstrate the effectiveness of our approach.

We propose an approach to analyzing data in which texts are associated with spatial and temporal references with the aim to understand how the text semantics vary over space and time. To represent the semantics, we apply probabilistic topic modeling. After extracting a set of topics and representing the texts by vectors of topic weights, we aggregate the data into a data cube with the dimensions corresponding to the set of topics, the set of spatial locations (e.g., regions), and the time divided into suitable intervals according to the scale of the planned analysis. Each cube cell corresponds to a combination (topic, location, time interval) and contains aggregate measures characterizing the subset of the texts concerning this topic and having the spatial and temporal references within these location and interval. Based on this structure, we systematically describe the space of analysis tasks on exploring the interrelationships among the three heterogeneous information facets, semantics, space, and time. We introduce the operations of projecting and slicing the cube, which are used to decompose complex tasks into simpler subtasks. We then present a design of a visual analytics system intended to support these subtasks. To reduce the complexity of the user interface, we apply the principles of structural, visual, and operational uniformity while respecting the specific properties of each facet. The aggregated data are represented in three parallel views corresponding to the three facets and providing different complementary perspectives on the data. The views have similar look-and-feel to the extent allowed by the facet specifics. Uniform interactive operations applicable to any view support establishing links between the facets. The uniformity principle is also applied in supporting the projecting and slicing operations on the data cube. We evaluate the feasibility and utility of the approach by applying it in two analysis scenarios using geolocated social media data for studying people's reactions to social and natural events of different spatial and temporal scales.

Similarity measuring methods are widely adopted in a broad range of visualization applications. In this work, we address the challenge of representing human perception in the visual analysis of scatterplots by introducing a novel deep-learning-based approach, ScatterNet, captures perception-driven similarities of such plots. The approach exploits deep neural networks to extract semantic features of scatterplot images for similarity calculation. We create a large labeled dataset consisting of similar and dissimilar images of scatterplots to train the deep neural network. We conduct a set of evaluations including performance experiments and a user study to demonstrate the effectiveness and efficiency of our approach. The evaluations confirm that the learned features capture the human perception of scatterplot similarity effectively. We describe two scenarios to show how ScatterNet can be applied in visual analysis applications.

Analysts commonly investigate the data distributions derived from statistical aggregations of data that are represented by charts, such as histograms and binned scatterplots, to visualize and analyze a large-scale dataset. Aggregate queries are implicitly executed through such a process. Datasets are constantly extremely large; thus, the response time should be accelerated by calculating predefined data cubes. However, the queries are limited to the predefined binning schema of preprocessed data cubes. Such limitation hinders analysts' flexible adjustment of visual specifications to investigate the implicit patterns in the data effectively. Particularly, RSATree enables arbitrary queries and flexible binning strategies by leveraging three schemes, namely, an R-tree-based space partitioning scheme to catch the data distribution, a locality-sensitive hashing technique to achieve locality-preserving random access to data items, and a summed area table scheme to support interactive query of aggregated values with a linear computational complexity. This study presents and implements a web-based visual query system that supports visual specification, query, and exploration of large-scale tabular data with user-adjustable granularities. We demonstrate the efficiency and utility of our approach by performing various experiments on real-world datasets and analyzing time and space complexity.

Scatterplots are frequently scaled to fit display areas in multi-view and multi-device data analysis environments. A common method used for scaling is to enlarge or shrink the entire scatterplot together with the inside points synchronously and proportionally. This process is called geometric scaling. However, geometric scaling of scatterplots may cause a perceptual bias, that is, the perceived and physical values of visual features may be dissociated with respect to geometric scaling. For example, if a scatterplot is projected from a laptop to a large projector screen, then observers may feel that the scatterplot shown on the projector has fewer points than that viewed on the laptop. This paper presents an evaluation study on the perceptual bias of visual features in scatterplots caused by geometric scaling. The study focuses on three fundamental visual features (i.e., numerosity, correlation, and cluster separation) and three hypotheses that are formulated on the basis of our experience. We carefully design three controlled experiments by using well-prepared synthetic data and recruit participants to complete the experiments on the basis of their subjective experience. With a detailed analysis of the experimental results, we obtain a set of instructive findings. First, geometric scaling causes a bias that has a linear relationship with the scale ratio. Second, no significant difference exists between the biases measured from normally and uniformly distributed scatterplots. Third, changing the point radius can correct the bias to a certain extent. These findings can be used to inspire the design decisions of scatterplots in various scenarios.

Traditional radio monitoring and management largely depend on radio spectrum data analysis, which requires considerable domain experience and heavy cognition effort and frequently results in incorrect signal judgment and incomprehensive situation awareness. Faced with increasingly complicated electromagnetic environments, radio supervisors urgently need additional data sources and advanced analytical technologies to enhance their situation awareness ability. This paper introduces a visual analytics approach for electromagnetic situation awareness. Guided by a detailed scenario and requirement analysis, we first propose a signal clustering method to process radio signal data and a situation assessment model to obtain qualitative and quantitative descriptions of the electromagnetic situations. We then design a two-module interface with a set of visualization views and interactions to help radio supervisors perceive and understand the electromagnetic situations by a joint analysis of radio signal data and radio spectrum data. Evaluations on real-world data sets and an interview with actual users demonstrate the effectiveness of our prototype system. Finally, we discuss the limitations of the proposed approach and provide future work directions.

In recent years, the technology and applications of visualizations on large high-resolution displays (LHDs) have received widespread attention because of its perceptual benefits and improved productivity. However, existing work on LHD visualization lacks both comprehensive guidance for design requirements and tools developed for its specific usage scenarios. In this paper, we present the scenarios, design, and implementation of DataV, a Software-as-a-Service (SaaS) visual deployment tool that enables rapid construction and cross-platform publishing of interactive visualization on LHDs. Our framework can support rich components for the high-performance rendering of multi-source heterogeneous data. DataV provides a full-fledged toolchain to help the user efficiently specify layout and interactions. We present its accessibility and impressive visual effects with examples and comparison with Tableau, Power BI, VisComposer, and iVisDesigner. We also report the performance of using DataV for 3D map rendering by comparing it with deck.gl.

The coronavirus disease 2019 (COVID-19) pandemic started in early 2020. At the beginning of February, a public welfare activity in epidemic data visualization, jointly launched by China Computer Federation (CCF) (CCF) CAD & CG Technical Committee, Alibaba Cloud Tianchi (Alibaba Cloud Tianch), JiqiZhixin (JiqiZhixin), Alibaba Cloud DataV (Alibaba Cloud DataV), and DataWhale (DataWhale), was launched with the theme "Fighting the Epidemic with One Mind and Talents like Tianchi." Developers in general are expected to focus on several demand scenarios, such as epidemic situation display, epidemic popular science, trend prediction, material-supply situation, and rework and return situation of employees from all sectors and areas, to discover the relationship between complex heterogeneous multi-source data, develop various upbeat works and present useful information to the public in a coherent manner. The entry works take the form of data visualization and are divided into two categories: popular science publicity and application scenarios. The popular science publicity category includes works for the public, focused on epidemic situation display, epidemic popular science publicity, epidemic prevention and control, and others. The application scenario category consists of the works of frontline officers, which can provide anti-epidemic workers with effective data tools for efficient and intuitive epidemic analysis; offer reliable, understandable, and easily transmitted information for disease prevention; and assist governments, enterprises, and institutions in the fight against COVID-19.

The rapid development of massively multiplayer online role-playing games (MMORPGs) has led operators to record huge amounts of fine-grained data from the in-game activities of players. These data provide considerable opportunities with which to study the dynamic interplay among player behaviors and investigate the roles of various social structures that underlie such interplay. However, modeling and visualizing these behavioral data remain a challenge. In this study, we propose a novel influence-susceptible model to measure the dynamic interplay among multiple behaviors. Based on this model, we introduce a new visual analytics system called BeXplorer. BeXplorer enables analysts to interactively explore the dynamic interplay between player purchase and communication behaviors and to examine the manner in which this interplay is bound by social structures where players are embedded.

In this paper, we focus on generating realistic images from text descriptions. Current methods first generate an initial image with rough shape and color, and then refine the initial image to a high-resolution one. Most existing text-to-image synthesis methods have two main problems. (1) These methods depend heavily on the quality of the initial images. If the initial image is not well initialized, the following processes can hardly refine the image to a satisfactory quality. (2) Each word contributes a different level of importance when depicting different image contents, however, unchanged text representation is used in existing image refinement processes. In this paper, we propose the Dynamic Memory Generative Adversarial Network (DM-GAN) to generate high-quality images. The proposed method introduces a dynamic memory module to refine fuzzy image contents, when the initial images are not well generated. A memory writing gate is designed to select the important text information based on the initial image content, which enables our method to accurately generate images from the text description. We also utilize a response gate to adaptively fuse the information read from the memories and the image features. We evaluate the DM-GAN model on the Caltech-UCSD Birds 200 dataset and the Microsoft Common Objects in Context dataset. Experimental results demonstrate that our DM-GAN model performs favorably against the state-of-the-art approaches.

<?tight?>Discovering the correlations among variables of air quality data is challenging, because the correlation time series are long-lasting, multi-faceted, and information-sparse. In this article, we propose a novel visual representation, called Time-correlation-partitioning (TCP) tree, that compactly characterizes correlations of multiple air quality variables and their evolutions. A TCP tree is generated by partitioning the information-theoretic correlation time series into pieces with respect to the variable hierarchy and temporal variations, and reorganizing these pieces into a hierarchically nested structure. The visual exploration of a TCP tree provides a sparse data traversal of the correlation variations and a situation-aware analysis of correlations among variables. This can help meteorologists understand the correlations among air quality variables better. We demonstrate the efficiency of our approach in a real-world air quality investigation scenario.

The increased accessibility of urban sensor data and the popularity of social network applications is enabling the discovery of crowd mobility and personal communication patterns. However, studying the egocentric relationships of an individual can be very challenging because available data may refer to direct contacts, such as phone calls between individuals, or indirect contacts, such as paired location presence. In this article, we develop methods to integrate three facets extracted from heterogeneous urban data (timelines, calls, and locations) through a progressive visual reasoning and inspection scheme. Our approach uses a detect-and-filter scheme such that, prior to visual refinement and analysis, a coarse detection is performed to extract the target individual and construct the timeline of the target. It then detects spatio-temporal co-occurrences or call-based contacts to develop the egocentric network of the individual. The filtering stage is enhanced with a line-based visual reasoning interface that facilitates a flexible and comprehensive investigation of egocentric relationships and connections in terms of time, space, and social networks. The integrated system, RelationLines, is demonstrated using a dataset that contains taxi GPS data, cell-base mobility data, mobile calling data, microblog data, and point-of-interest (POI) data from a city with millions of citizens. We examine the effectiveness and efficiency of our system with three case studies and user review.

Visual analysis is widely applied to study human mobility due to the ability of integrating contextual information from multiple data sources. Analyzing trajectory data through visualization improves the efficiency and accuracy of the analysis, yet it may induce exposure of the location privacy. To balance the location privacy and analysis effectiveness, this work focuses on the behaviors of different geo-based contexts in the process of trajectory interpretation. Three types of geo-based contexts are identified after surveying 94 related literatures. We further conduct experiments to investigate their capability by evaluating how they benefit the analysis, and whether they lead to location privacy exposure. Finally, we report and discuss interesting findings, and provide guidelines to the design of privacy-preserving analysis approaches for human periodic trajectories.

Understanding the relationship between urban locations is an essential task in urban planning and transportation management. Although prior works have focused on studying urban locations by aggregating location-based properties, our scheme preserves the mutual influence between urban locations and mobility behavior, and thereby enables situation-aware exploration of urban regions. By leveraging word embedding techniques, we encode urban locations with a vectorized representation while retaining situational awareness. Specifically, we design a spatial embedding algorithm that is precomputed by incorporating the interactions between urban locations and moving objects. To explore our proposed technique, we have designed and implemented a web-based visual exploration system that supports the comprehensive analysis of human mobility, location functionality, and traffic assessment by leveraging the proposed visual representation. The case studies demonstrate the effectiveness of our approach.

When analyzing a visualized network, users need to explore different sections of the network to gain insight. However, effective exploration of large networks is often a challenge. While various tools are available for users to explore the global and local features of a network, these tools usually require significant interaction activities, such as repetitive navigation actions to follow network nodes and edges. In this paper, we propose a structure-based suggestive exploration approach to support effective exploration of large networks by suggesting appropriate structures upon user request. Encoding nodes with vectorized representations by transforming information of surrounding structures of nodes into a high dimensional space, our approach can identify similar structures within a large network, enable user interaction with multiple similar structures simultaneously, and guide the exploration of unexplored structures. We develop a web-based visual exploration system to incorporate this suggestive exploration approach and compare performances of our approach under different vectorizing methods and networks. We also present the usability and effectiveness of our approach through a controlled user study with two datasets.

Analyzing social networks reveals the relationships between individuals and groups in the data. However, such analysis can also lead to privacy exposure (whether intentionally or inadvertently): leaking the real-world identity of ostensibly anonymous individuals. Most sanitization strategies modify the graph's structure based on hypothesized tactics that an adversary would employ. While combining multiple anonymization schemes provides a more comprehensive privacy protection, deciding the appropriate set of techniques-along with evaluating how applying the strategies will affect the utility of the anonymized results-remains a significant challenge. To address this problem, we introduce GraphProtector, a visual interface that guides a user through a privacy preservation pipeline. GraphProtector enables multiple privacy protection schemes which can be simultaneously combined together as a hybrid approach. To demonstrate the effectiveness of GraphProtector, we report several case studies and feedback collected from interviews with expert users in various scenarios.

Regarded as a high-level tactic in soccer, a team formation assigns players different tasks and indicates their active regions on the pitch, thereby influencing the team performance significantly. Analysis of formations in soccer has become particularly indispensable for soccer analysts. However, formations of a team are intrinsically time-varying and contain inherent spatial information. The spatio-temporal nature of formations and other characteristics of soccer data, such as multivariate features, make analysis of formations in soccer a challenging problem. In this study, we closely worked with domain experts to characterize domain problems of formation analysis in soccer and formulated several design goals. We design a novel spatio-temporal visual representation of changes in team formation, allowing analysts to visually analyze the evolution of formations and track the spatial flow of players within formations over time. Based on the new design, we further design and develop ForVizor, a visual analytics system, which empowers users to track the spatio-temporal changes in formation and understand how and why such changes occur. With ForVizor, domain experts conduct formation analysis of two games. Analysis results with insights and useful feedback are summarized in two case studies.

Fuzzy clustering assigns a probability of membership for a datum to a cluster, which veritably reflects real-world clustering scenarios but significantly increases the complexity of understanding fuzzy clusters. Many studies have demonstrated that visualization techniques for multi-dimensional data are beneficial to understand fuzzy clusters. However, no empirical evidence exists on the effectiveness and efficiency of these visualization techniques in solving analytical tasks featured by fuzzy clusters. In this paper, we conduct a controlled experiment to evaluate the ability of fuzzy clusters analysis to use four multi-dimensional visualization techniques, namely, parallel coordinate plot, scatterplot matrix, principal component analysis, and Radviz. First, we define the analytical tasks and their representative questions specific to fuzzy clusters analysis. Then, we design objective questionnaires to compare the accuracy, time, and satisfaction in using the four techniques to solve the questions. We also design subjective questionnaires to collect the experience of the volunteers with the four techniques in terms of ease of use, informativeness, and helpfulness. With a complete experiment process and a detailed result analysis, we test against four hypotheses that are formulated on the basis of our experience, and provide instructive guidance for analysts in selecting appropriate and efficient visualization techniques to analyze fuzzy clusters.

A variety of human movement datasets are represented in an Origin-Destination(OD) form, such as taxi trips, mobile phone locations, etc. As a commonly-used method to visualize OD data, flow map always fails to discover patterns of human mobility, due to massive intersections and occlusions of lines on a 2D geographical map. A large number of techniques have been proposed to reduce visual clutter of flow maps, such as filtering, clustering and edge bundling, but the correlations of OD flows are often neglected, which makes the simplified OD flow map present little semantic information. In this paper, a characterization of OD flows is established based on an analogy between OD flows and natural language processing (NPL) terms. Then, an iterative multi-objective sampling scheme is designed to select OD flows in a vectorized representation space. To enhance the readability of sampled OD flows, a set of meaningful visual encodings are designed to present the interactions of OD flows. We design and implement a visual exploration system that supports visual inspection and quantitative evaluation from a variety of perspectives. Case studies based on real-world datasets and interviews with domain experts have demonstrated the effectiveness of our system in reducing the visual clutter and enhancing correlations of OD flows.

With the rapid development of massively multiplayer online role-playing games (MMORPGs), a huge amount of fine-grained data on the in-game activities of players have been recorded by MMORPGs operators. These data provide considerable opportunities with which to study the dynamic interplay between player behaviors and investigate the roles of various social structures that underlie such interplay. However, it is challenging to model and visualize these behavioral data. This study proposes a novel influence-susceptible model to measure the dynamic interplay between behaviors. Based on this model, we introduce a new visual analytics system called BeXplorer. This system enables analysts to interactively explore the dynamic interplay between player purchase and communication behaviors and to examine the manner in which this interplay is bound by social structures where players are embedded. Three case studies and a task-based evaluation are conducted to demonstrate the effectiveness and applicability of our method.

Modeling human mobility is a critical task in fields such as urban planning, ecology, and epidemiology. Given the current use of mobile phones, there is an abundance of data that can be used to create models of high reliability. Existing techniques can reveal the macropatterns of crowd movement or analyze the trajectory of a person; however, they typically focus on geographical characteristics. This paper presents a graph-based approach for structuring crowd mobility transition over multiple granularities in the context of social behavior. The key to our approach is an adaptive data representation, the adaptive mobility transition graph (AMTG), which is globally generated from citywide human mobility data by defining the temporal trends of human mobility and the interleaved transitions between different mobility patterns. We describe the design, creation, and manipulation of the AMTG and introduce a visual analysis system that supports the multifaceted exploration of citywide human mobility patterns.

Urban data is massive, heterogeneous, and spatio-temporal, posing a substantial challenge for visualization and analysis. In this paper, we design and implement a novel visual analytics approach, Visual Analyzer for Urban Data (VAUD), that supports the visualization, querying, and exploration of urban data. Our approach allows for cross-domain correlation from multiple data sources by leveraging spatial-temporal and social inter-connectedness features. Through our approach, the analyst is able to select, filter, aggregate across multiple data sources and extract information that would be hidden to a single data subset. To illustrate the effectiveness of our approach, we provide case studies on a real urban dataset that contains the cyber-, physical-, and social- information of 14 million citizens over 22 days.

The use of scatterplots is an important method for multivariate data visualization. The point distribution on the scatterplot, along with variable values represented by each point, can help analyze underlying patterns in data. However, determining the multivariate data variation on a scatterplot generated using projection methods, such as multidimensional scaling, is difficult. Furthermore, the point distribution becomes unclear when the data scale is large and clutter problems occur. These conditions can significantly decrease the usability of scatterplots on multivariate data analysis. In this study, we present a cluster-based visual abstraction method to enhance the visualization of multivariate scatterplots. Our method leverages an adapted multilabel clustering method to provide abstractions of high quality for scatterplots. An image-based method is used to deal with large scale data problem. Furthermore, a suite of glyphs is designed to visualize the data at different levels of detail and support data exploration. The view coordination between the glyph-based visualization and the table lens can effectively enhance the multivariate data analysis. Through numerical evaluations for data abstraction quality, case studies and a user study, we demonstrate the effectiveness and usability of the proposed techniques for multivariate data analysis on scatterplots.

Sharing data for public usage requires sanitization to prevent sensitive information from leaking. Previous studies have presented methods for creating privacy preserving visualizations. However, few of them provide sufficient feedback to users on how much utility is reduced (or preserved) during such a process. To address this, we design a visual interface along with a data manipulation pipeline that allows users to gauge utility loss while interactively and iteratively handling privacy issues in their data. Widely known and discussed types of privacy models, i.e., syntactic anonymity and differential privacy, are integrated and compared under different use case scenarios. Case study results on a variety of examples demonstrate the effectiveness of our approach.

Many approaches for analyzing a high-dimensional dataset assume that the dataset contains specific structures, e.g., clusters in linear subspaces or non-linear manifolds. This yields a trial-and-error process to verify the appropriate model and parameters. This paper contributes an exploratory interface that supports visual identification of low-dimensional structures in a high-dimensional dataset, and facilitates the optimized selection of data models and configurations. Our key idea is to abstract a set of global and local feature descriptors from the neighborhood graph-based representation of the latent low-dimensional structure, such as pairwise geodesic distance (GD) among points and pairwise local tangent space divergence (LTSD) among pointwise local tangent spaces (LTS). We propose a new LTSD-GD view, which is constructed by mapping LTSD and GD to the axis and axis using 1D multidimensional scaling, respectively. Unlike traditional dimensionality reduction methods that preserve various kinds of distances among points, the LTSD-GD view presents the distribution of pointwise LTS ( axis) and the variation of LTS in structures (the combination of axis and axis). We design and implement a suite of visual tools for navigating and reasoning about intrinsic structures of a high-dimensional dataset. Three case studies verify the effectiveness of our approach.

While there have been a dozen of authoring systems and programming toolkits for visual design and development, users who do not have programming skills, such as data analysts or interface designers, still may feel cumbersome to efficiently implement a web-based visualization. In this paper, we present ECharts, an open-sourced, web-based, cross-platform framework that supports the rapid construction of interactive visualization. The motivation is driven by three goals: easy-to-use, rich built-in interactions, and high performance. The kernel of ECharts is a suite of declarative visual design language that customizes built-in chart types. The underlying streaming architecture, together with a high-performance graphics renderer based on HTML5 canvas, enables the high expandability and performance of ECharts. We report the design, implementation, and applications of ECharts with a diverse variety of examples. We compare the utility and performance of ECharts with C3.js, HighCharts, and Chart.js. Results of the experiments demonstrate the efficiency and scalability of our framework. Since the first release in June 2013, ECharts has iterated 63 versions, and attracted over 22,000 star counts and over 1700 related projects in the GitHub. ECharts is regarded as a leading visualization development tool in the world, and ranks the third in the GitHub visualization tab.

As the amount of data being collected has increased, the need for tools that can enable the visual exploration of data has also grown. This has led to the development of a variety of widely used programming frameworks for information visualization. Unfortunately, such frameworks demand comprehensive visualization and coding skills and require users to develop visualization from scratch. An alternative is to create interactive visualization design environments that require little to no programming. However, these tools only supports a small portion of visual forms. We present a programmable integrated development environment (IDE), VisComposer, that supports the development of expressive visualization using a drag-and-drop visual interface. VisComposer exposes the programmability by customizing desired components within a modularized visualization composition pipeline, effectively balancing the capability gap between expert coders and visualization artists. The implemented system empowers users to compose comprehensive visualizations with real-time preview and optimization features, and supports prototyping, sharing and reuse of the effects by means of an intuitive visual composer. Visual programming and textual programming integrated in our system allow users to compose more complex visual effects while retaining the simplicity of use. We demonstrate the performance of VisComposer with a variety of examples and an informal user evaluation.

Support vector machines (SVMs) are supervised learning models traditionally employed for classification and regression analysis. In classification analysis, a set of training data is chosen, and each instance in the training data is assigned a categorical class. An SVM then constructs a model based on a separating plane that maximizes the margin between different classes. Despite being one of the most popular classification models because of its strong performance empirically, understanding the knowledge captured in an SVM remains difficult. SVMs are typically applied in a black-box manner where the details of parameter tuning, training, and even the final constructed model are hidden from the users. This is natural since these details are often complex and difficult to understand without proper visualization tools. However, such an approach often brings about various problems including trial-and-error tuning and suspicious users who are forced to trust these models blindly. The contribution of this paper is a visual analysis approach for building SVMs in an open-box manner. Our goal is to improve an analyst’s understanding of the SVM modeling process through a suite of visualization techniques that allow users to have full interactive visual control over the entire SVM training process. Our visual exploration tools have been developed to enable intuitive parameter tuning, training data manipulation, and rule extraction as part of the SVM training process. To demonstrate the efficacy of our approach, we conduct a case study using a real-world robot control dataset.

Sensing technologies, social media, and large-scale computing infrastructures have produced a variety of traffic and transportation data, e.g., human mobility, mobile trajectories, mobile phone calls, traffic, and geographical data. Despite the wealth of research on intelligent transportation systems, contemporary analytical tools are often inadequate for handling the data with the character of large volume, sparseness, and heterogeneity, let alone for supporting interactive visual analysis for data-intensive applications. Visual analytics can build bridges between the capability of data processing and human intelligence to promote addressing various transportation problems. On one hand, by employing visual channels to represent datasets and transforming various types of data into appropriate visual components, visualization can enhance understanding and analysis. On the other hand, an interactive interface allows users to investigate and directly access selected data points or features, discover interesting patterns or events, and engage in visual reasoning that allows users to gain insights, e.g., it is desirable to only show the most relevant portions of a dataset while giving directions for potential exploration.

Many cities and countries are now striving to create intelligent transportation systems that utilize the current abundance of multisource and multiform data related to the functionality and the use of transportation infrastructure to better support human mobility, interests, and lifestyles. Such intelligent transportation systems aim to provide novel services that can enable transportation consumers and managers to be better informed and make safer and more efficient use of the infrastructure. However, the transportation domain is characterized by both complex data and complex problems, which calls for visual analytics approaches. The science of visual analytics is continuing to develop principles, methods, and tools to enable synergistic work between humans and computers through interactive visual interfaces. Such interfaces support the unique capabilities of humans (such as the flexible application of prior knowledge and experiences, creative thinking, and insight) and couple these abilities with machines' computational strengths, enabling the generation of new knowledge from large and complex data. In this paper, we describe recent developments in visual analytics that are related to the study of movement and transportation systems and discuss how visual analytics can enable and improve the intelligent transportation systems of the future. We provide a survey of literature from the visual analytics domain and organize the survey with respect to the different types of transportation data, movement and its relationship to infrastructure and behavior, and modeling and planning. We conclude with lessons learned and future directions, including social transportation, recommender systems, and policy implications.

Thanks to the ubiquitous cell phone use, we have never been so close to uncover population mobility patterns in urban area. While some researches utilize cellphone call records to mine population patterns, few works aim to depict population movement in adaptively spatial and temporal representations, i.e., from a community, a district in the city over an hour, a day to a week. In this paper, we construct a system which deciphers, transforms, queries, and visualizes the records from the millions of users in a city. In particular, we design a data structure, namely MobiHash, which collects phone call records over base stations and indexes them by utilizing a Voronoi division of the urban space. MobiHash supports responsive data queries so that users can interactively retrieve trajectories reflecting population flows in areas of interest. Moreover, population movement is represented as vector fields to reduce visual clutter and occlusions. Because of sparse moving points, a novel radiation model is proposed to interpolate population passing zones. Case studies and experts' feedback validate the utility and efficiency by comparing population moving patterns in different times by using our system.

Massive taxi trajectory data is exploited for knowledge discovery in transportation and urban planning. Existing tools typically require users to select and brush geospatial regions on a map when retrieving and exploring taxi trajectories and passenger trips. To answer seemingly simple questions such as "What were the taxi trips starting from Main Street and ending at Wall Street in the morning?" or "Where are the taxis arriving at the Art Museum at noon typically coming from?", tedious and time consuming interactions are usually needed since the numeric GPS points of trajectories are not directly linked to the keywords such as "Main Street", "Wall Street", and "Art Museum". In this paper, we present SemanticTraj, a new method for managing and visualizing taxi trajectory data in an intuitive, semantic rich, and efficient means. With SemanticTraj, domain and public users can find answers to the aforementioned questions easily through direct queries based on the terms. They can also interactively explore the retrieved data in visualizations enhanced by semantic information of the trajectories and trips. In particular, taxi trajectories are converted into taxi documents through a textualization transformation process. This process maps GPS points into a series of street/POI names and pick-up/drop-off locations. It also converts vehicle speeds into user-defined descriptive terms. Then, a corpus of taxi documents is formed and indexed to enable flexible semantic queries over a text search engine. Semantic labels and meta-summaries of the results are integrated with a set of visualizations in a SemanticTraj prototype, which helps users study taxi trajectories quickly and easily. A set of usage scenarios are presented to show the usability of the system. We also collected feedback from domain experts and conducted a preliminary user study to evaluate the visual system.

Visual analytics plays a key role in the era of connected industry (or industry 4.0, industrial internet) as modern machines and assembly lines generate large amounts of data and effective visual exploration techniques are needed for troubleshooting, process optimization, and decision making. However, developing effective visual analytics solutions for this application domain is a challenging task due to the sheer volume and the complexity of the data collected in the manufacturing processes. We report the design and implementation of a comprehensive visual analytics system, ViDX. It supports both real-time tracking of assembly line performance and historical data exploration to identify inefficiencies, locate anomalies, and form hypotheses about their causes and effects. The system is designed based on a set of requirements gathered through discussions with the managers and operators from manufacturing sites. It features interlinked views displaying data at different levels of detail. In particular, we apply and extend the Marey's graph by introducing a time-aware outlier-preserving visual aggregation technique to support effective troubleshooting in manufacturing processes. We also introduce two novel interaction techniques, namely the quantiles brush and samples brush, for the users to interactively steer the outlier detection algorithms. We evaluate the system with example use cases and an in-depth user interview, both conducted together with the managers and operators from manufacturing plants. The result demonstrates its effectiveness and reports a successful pilot application of visual analytics for manufacturing in smart factories.

Recent years have witnessed the rapid development and wide adoption of immersive head-mounted devices, such as HTC VIVE, Oculus Rift, and Microsoft HoloLens. These immersive devices have the potential to significantly extend the methodology of urban visual analytics by providing critical 3D context information and creating a sense of presence. In this paper, we propose a theoretical model to characterize the visualizations in immersive urban analytics. Furthermore, based on our comprehensive and concise model, we contribute a typology of combination methods of 2D and 3D visualizations that distinguishes between linked views, embedded views , and mixed views . We also propose a supporting guideline to assist users in selecting a proper view under certain circumstances by considering visual geometry and spatial distribution of the 2D and 3D visualizations. Finally, based on existing work, possible future research opportunities are explored and discussed.

Association rules have been widely used for detecting relations between attribute-value pairs of categorical datasets. Existing solutions of mining interesting association rules are based on the support-confidence theory. However, it is non-trivial for the user to understand and modify the rules or the results of intermediate steps in the mining process, because the interestingness of rules might differ largely for various tasks and users. In this paper we propose to reinforce conventional association rule mining process by mapping the entire process into a visualization assisted loop, with which the user workload for modulating parameters and mining rules is reduced, and the mining efficiency is greatly improved. A matrix-based visualization technique is employed to encode the measure computation value, the data distribution and the intermediate results. We also design a set of visual exploration tools to support interactively inspection and manipulation of association measures, constraints of different types, and the results of intermediate steps. The effectiveness of our approach is demonstrated with various scenarios.

Abstract Previous viewpoint selection methods in volume visualization are generally based on some deterministic measures of viewpoint quality. However, they may not express the familiarity and aesthetic sense of users for features of interest. In this paper, we propose an image‐based viewpoint selection model to learn how visualization experts choose representative viewpoints for volumes with similar features. For a given volume, we first collect images with similar features, and these images reflect the viewpoint preferences of the experts when visualizing these features. Each collected image tallies votes to the viewpoints with the best matching based on an image similarity measure, which evaluates the spatial shape and appearance similarity between the collected image and the rendered image from the viewpoint. The optimal viewpoint is the one with the most votes from the collected images, that is, the viewpoint chosen by most visualization experts for similar features. We performed experiments on various volumes available in volume visualization, and made comparisons with traditional viewpoint selection methods. The results demonstrate that our model can select more canonical viewpoints, which are consistent with human perception.

Even with simple everyday tasks like online shopping or choosing a restaurant, users are easily overwhelmed with the large number of choices available today, each with a large number of inter-related attributes. We present ExRank, an interactive interface for exploring data that helps users understand the relationship between attribute values and find interesting items in the dataset. Based on a kNN graph and a PageRank algorithm, ExRank suggests which attributes the user should look at, and how expressed choices in particular attributes affect the distribution of values in other attributes for candidate objects. It solves the problem of empty result by showing similar items and when there are too many results, it ranks the data for the user. This demo consists of 1) the description of the software architecture and the user interface 2) the logic and reason behind our solution and 3) a list of demonstration scenarios for showing to the audience.

Studying human movement citywide is important for understanding mobility and transportation patterns. Rather than investigating the trajectories of individuals, we employ an Eulerian approach to analyze the crowd flows among a geographical network and a social network, which are extracted from mobile phone data. We design a suite of visualization techniques to illustrate the dynamic evolutions of the flow over the networks. We contribute the design and implementation of a visual analytics system, which is called Mobility Viewer, that supports situation-aware understanding and visual reasoning of human mobility. We exemplify our approach with a real citywide data set of seven million users in two months.

Big data for social transportation brings us unprecedented opportunities for resolving transportation problems for which traditional approaches are not competent and for building the next-generation intelligent transportation systems. Although social data have been applied for transportation analysis, there are still many challenges. First, social data evolve with time and contain abundant information, posing a crucial need for data collection and cleaning. Meanwhile, each type of data has specific advantages and limitations for social transportation, and one data type alone is not capable of describing the overall state of a transportation system. Systematic data fusing approaches or frameworks for combining social signal data with different features, structures, resolutions, and precision are needed. Second, data processing and mining techniques, such as natural language processing and analysis of streaming data, require further revolutions in effective utilization of real-time traffic information. Third, social data are connected to cyber and physical spaces. To address practical problems in social transportation, a suite of schemes are demanded for realizing big data in social transportation systems, such as crowdsourcing, visual analysis, and task-based services. In this paper, we overview data sources, analytical approaches, and application systems for social transportation, and we also suggest a few future research directions for this new social transportation field.

Although basketball games have received broad attention, the forms of game reports and webcast are purely content-based cross-media: texts, videos, snapshots, and performance figures. Analytical narrations of games that seek to compose a complete game from heterogeneous datasets are challenging for general media producers because such a composition is time-consuming and heavily depends on domain experts. In particular, an appropriate analytical commentary of basketball games requires two factors, namely, rich context and domain knowledge, which includes game events, player locations, player profiles, and team profiles, among others. This type of analytical commentary elicits a timely and effective basketball game data visualization made up of different sources of media. Existing visualizations of basketball games mainly profile a particular aspect of the game. Therefore, this paper presents an expressive visualization scheme that comprehensively illustrates NBA games with three levels of details: a season level, a game level, and a session level. We reorganize a basketball game as a sequence of sessions to depict the game states and heated confrontations. We design and implement a live system that integrates multimedia NBA datasets: play-by-play text data, box score data, game video data, and action area data. We demonstrate the effectiveness of this scheme with case studies and user feedbacks.

Cloud computing is an essential technology to Big Data analytics and services. A cloud computing system is often comprised of a large number of parallel computing and storage devices. Monitoring the usage and performance of such a system is important for efficient operations, maintenance, and security. Tracing every application on a large cloud system is untenable due to scale and privacy issues. But profile data can be collected relatively efficiently by regularly sampling the state of the system, including properties such as CPU load, memory usage, network usage, and others, creating a set of multivariate time series for each system. Adequate tools for studying such large-scale, multidimensional data are lacking. In this paper, we present a visual based analysis approach to understanding and analyzing the performance and behavior of cloud computing systems. Our design is based on similarity measures and a layout method to portray the behavior of each compute node over time. When visualizing a large number of behavioral lines together, distinct patterns often appear suggesting particular types of performance bottleneck. The resulting system provides multiple linked views, which allow the user to interactively explore the data by examining the data or a selected subset at different levels of detail. Our case studies, which use datasets collected from two different cloud systems, show that this visual based approach is effective in identifying trends and anomalies of the systems.

This paper presents an accurate parallel implementation of unsteady flow line integral convolution (UFLIC) for high-performance visualization of large time-varying flows. Our approach differs from previous implementations by using a novel value scattering+gathering mechanism to parallelize UFLIC and designing a pathline reuse strategy to reduce the computational cost of pathline integration. By exploiting the massive parallelism of modern graphical processing units (GPU), the proposed method allows for real-time dense visualization of unsteady flows with high spatial-temporal coherence.

Joint channel-aware and buffer-aware scheduling is a promising way to improve Quality of Service (QoS) given energy efficiency constraint. In this paper, optimal probabilistic scheduling strategy with random data arrival and fading channel is analysed. More specifically, we focus on the fast fading scenario where a data packet has to be transmitted with several timeslots. A linear programming problem is formulated, where we minimize the average delay given an average power constraint. By deriving the analytical solution to this linear programming, optimal delay-power tradeoff along with the corresponding threshold-based scheduling policy is presented.

Narrow, inextensible, and naturally flat ribbons have some special and interesting phenomena under isometric deformations. Although a ribbon has a shape between rod and shell, directly applying the geometric representation designed for them imposes a challenge to faithfully reproduce interesting behaviors. We thus parameterize the ribbon surface as a developable ruled surface along its centerline and represent it using a framed centerline curve. Then the elastic and kinetic energy of the ribbon surface can be equivalently yet compactly described by the framed centerline curve only. To avoid numerical singularity when develop ability is violated, a finite Taylor series approximation to the potential energy is adopted. Under the observation that the off-centerline part of ribbon contributes little dynamic effect, the kinetic energy is simplified with respect to the centerline velocity only. For efficiency, each time step is separated into two stages: dynamically evolving the centerline, and then quasi-statically updating the ruling. We validate the method with qualitative analysis and ribbon specific phenomena comparisons with real-world scenarios. A set of comparisons to rod and shell model is also provided to demonstrate the advantages of our method.

Exploring multivariate spatial data attracts much attention in the visualization community. The main challenge lies in that automatic analysis techniques is insufficient in discovering complicated patterns with the perspective of human beings, while visualization techniques are incapable of accurately identifying the features of interest. This paper addresses this contradiction by enhancing automatic analysis techniques with human intelligence in an iterative visual exploration process. The integrated system, called EasyXplorer, provides a suite of intuitive clustering, dimension reduction, visual encoding and filtering widgets within 2D and 3D views, allowing an inexperienced user to visually explore and reason undiscovered features with several simple interactions. Case studies show the quality and scalability of our approach in quite challenging examples.

Graph cut has proven to be an effective scheme to solve a wide variety of segmentation problems in vision and graphics community. The main limitation of conventional graph-cut implementations is that they can hardly handle large images or videos because of high computational complexity. Even though there are some parallelization solutions, they commonly suffer from the problems of low parallelism (on CPU) or low convergence speed (on GPU). In this paper, we present a novel graph-cut algorithm that leverages a parallelized jump flooding technique and an heuristic push-relabel scheme to enhance the graph-cut process, namely, back-and-forth relabel, convergence detection, and block-wise push-relabel. The entire process is parallelizable on GPU, and outperforms the existing GPU-based implementations in terms of global convergence, information propagation, and performance. We design an intuitive user interface for specifying interested regions in cases of occlusions when handling video sequences. Experiments on a variety of data sets, including images (up to 15 K × 10 K), videos (up to 2.5 K × 1.5 K × 50), and volumetric data, achieve high-quality results and a maximum 40-fold (139-fold) speedup over conventional GPU (CPU-)-based approaches.

Data-driven intelligent transportation systems utilize data resources generated within intelligent systems to improve the performance of transportation systems and provide convenient and reliable services. Traffic data refer to datasets generated and collected on moving vehicles and objects. Data visualization is an efficient means to represent distributions and structures of datasets and reveal hidden patterns in the data. This paper introduces the basic concept and pipeline of traffic data visualization, provides an overview of related data processing techniques, and summarizes existing methods for depicting the temporal, spatial, numerical, and categorical properties of traffic data.

While visualization has been widely used as a data presentation tool in both desktop and mobile devices, the rapid visualization of information from images is still underexplored. In this work, we present a smartphone image acquisition and visualization approach for text-based data. Our prototype, ShotVis, takes images of text captured from mobile devices and extracts information for visualization. First, scattered characters in the text are processed and interactively reformulated to be stored as structured data (i.e., tables of numbers, lists of words, sentences). From there, ShotVis allows users to interactively bind visual forms to the underlying data and produce visualizations of the selected forms through touch-based interactions. In this manner, ShotVis can quickly summarize text from images into word clouds, scatterplots, and various other visualizations all through a simple click of the camera. In this way, ShotVis facilitates the interactive exploration of text data captured via cameras in smartphone devices. To demonstrate our prototype, several case studies are presented along with one user study to demonstrate the effectiveness of our approach.

This paper presents an efficient visualization and exploration approach for modeling and characterizing the relationships and uncertainties in the context of a multidimensional ensemble dataset. Its core is a novel dissimilarity-preserving projection technique that characterizes not only the relationships among the mean values of the ensemble data objects but also the relationships among the distributions of ensemble members. This uncertainty-aware projection scheme leads to an improved understanding of the intrinsic structure in an ensemble dataset. The analysis of the ensemble dataset is further augmented by a suite of visual encoding and exploration tools. Experimental results on both artificial and real-world datasets demonstrate the effectiveness of our approach.

Illustrating the differences between DTI fiber models is important for the purposes of group comparison, atlas construction, and uncertainty analysis. Standard approaches show difference between DTI models in 3D space, with either voxel-based or fiber based comparison. We proposes a new method that embeds a high-dimensional 3D fiber model as a continuous map in a 2D normalized space, built on fibers from all targeted fiber models. One advantage over previous 2D embedding approaches is that our method represents all fiber models in a unified coordinate system by taking a cluster-and-project routine, ensuring that closely located fibers from different models will be projected to close points on the 2D map. Using this projected 2D map, subtle differences that are difficult to distinguish in 3D space can be clearly displayed and individual fiber model can be effectively summarized. We applied our method to group comparison and analysis of various fiber models.

The popularity of social media services has been innovating the way of information acquisition in modern society. Meanwhile, mass information is generated in every single day. To extract useful knowledge, much effort has been invested in analyzing social media contents, e.g., (emerging) topic discovery. With these findings, however, users may still find it hard to obtain knowledge of great interest in conformity with their preference. In this paper, we present a novel system which brings proper context to continuously incoming social media contents, such that mass information can be indexed, organized and analyzed around Wikipedia entities. Four data analytics tools are employed in the system. Three of them aim to enrich each Wikipedia entity by analyzing the relevant contents while the other one builds an information network among the most relevant Wikipedia entities. With our system, users can easily pinpoint valuable information and knowledge they are interested in, as well as navigate to other closely related entities through the information network for further exploration.

In this paper, we present a new automatic shader simplification method using surface signal approximation. We regard the entire multi-stage rendering pipeline as a process that generates signals on surfaces, and we formulate the simplification of the fragment shader as a global simplification problem across multi-shader stages. Three new shader simplification rules are proposed to solve the problem. First, the code transformation rule transforms fragment shader code to other shader stages in order to redistribute computations on pixels up to the level of geometry primitives. Second, the surface-wise approximation rule uses high-order polynomial basis functions on surfaces to approximate pixel-wise computations in the fragment shader. These approximations are pre-cached and simplify computations at runtime. Third, the surface subdivision rule tessellates surfaces into smaller patches. It combines with the previous two rules to approximate pixel-wise signals at different levels of tessellations with different computation times and visual errors. To evaluate simplified shaders using these simplification rules, we introduce a new cost model that includes the visual quality, rendering time and memory consumption. With these simplification rules and the cost model, we present an integrated shader simplification algorithm that is capable of automatically generating variants of simplified shaders and selecting a sequence of preferable shaders. Results show that the sequence of selected simplified shaders balance performance, accuracy and memory consumption well.

The articles in this special issue present extended versions of the five best papers from the 2013 IEEE Pacific Visualization Symposium (PacificVis'13) which was held in Sydney, Australia from February 26 to March 1, 2013. The objective of this symposium series is to foster greater exchange between visualization researchers and practitioners, with a focus on the Asia-Pacific region.

Scatterplots are widely used to visualize scatter dataset for exploring outliers, clusters, local trends, and correlations. Depicting multi-class scattered points within a single scatterplot view, however, may suffer from heavy overdraw, making it inefficient for data analysis. This paper presents a new visual abstraction scheme that employs a hierarchical multi-class sampling technique to show a feature-preserving simplification. To enhance the density contrast, the colors of multiple classes are optimized by taking the multi-class point distributions into account. We design a visual exploration system that supports visual inspection and quantitative analysis from different perspectives. We have applied our system to several challenging datasets, and the results demonstrate the efficiency of our approach.

Previous studies on E-transaction time-series have mainly focused on finding temporal trends of transaction behavior. Interesting transactions that are time-stamped and situation-relevant may easily be obscured in a large amount of information. This paper proposes a visual analytics system, Visual Analysis of E-transaction Time-Series (VAET), that allows the analysts to interactively explore large transaction datasets for insights about time-varying transactions. With a set of analyst-determined training samples, VAET automatically estimates the saliency of each transaction in a large time-series using a probabilistic decision tree learner. It provides an effective time-of-saliency (TOS) map where the analysts can explore a large number of transactions at different time granularities. Interesting transactions are further encoded with KnotLines, a compact visual representation that captures both the temporal variations and the contextual connection of transactions. The analysts can thus explore, select, and investigate knotlines of interest. A case study and user study with a real E-transactions dataset (26 million records) demonstrate the effectiveness of VAET.

Social data refers to data individuals create that is knowingly and voluntarily shared by them and is an exciting avenue into gaining insight into interpersonal behaviors and interaction. However, such data is large, heterogeneous and often incomplete, properties that make the analysis of such data extremely challenging. One common method of exploring such data is through cluster analysis, which can enable analysts to find groups of related users, behaviors and interactions. This paper presents a novel visual analysis approach for detecting clusters within large-scale social networks by utilizing a divide-analyze-recombine scheme that sequentially performs data partitioning, subset clustering and result recombination within an integrated visual interface. A case study on a microblog messaging data (with 4.8 millions users) is used to demonstrate the feasibility of this approach and comparisons are also provided to illustrate the performance benefits of this approach with respect to existing solutions.

We present an interactive rendering solution that aims at visualizing large-scale mesh with hundred millions of triangles up to extremely high resolution on cloud computing platform. Firstly, we propose a novel adaptive parallel rasterization method based on MapReduce, whose results are stored in a data format called enhanced layered depth image (ELDI). In order to fully apply the powerful capabilities of cloud computing, our proposed method integrates pixel and triangle based strategies in one processing pipeline so as to significantly reduce data transfer between Map and Reduce steps. Consequently, a light-weight Web service for interactive rendering is also proposed to couple with the rasterization step, which enables user to interactively visualize large-scale mesh via browser on both mobile devices and work stations. According to demonstrated examples in the paper, once the rasterization results are obtained, users can freely adjust visualizing effects of complex mesh. Our rendering approach is promising for big data exploration, analysis or authoring.

Digital image halftoning is a widely used technique. However, achieving high fidelity tone reproduction and structural preservation with low computational time cost remains a challenging problem. This paper presents a highly parallel algorithm to boost real-time application of serial structure-preserving error diffusion. The contrast-aware halftoning approach is one such technique with superior structure preservation, but it offers only a limited opportunity for graphics processing unit (GPU) acceleration. Our method integrates contrast-aware halftoning into a new parallelizable error-diffusion halftoning framework. To eliminate visually disturbing artifacts resulting from parallelization, we propose a novel multiple quantization model and space-filling curve to maintain tone consistency, blue-noise property, and structure consistency. Our GPU implementation on a commodity personal computer achieves a real-time performance for a moderately sized image. We demonstrate the high quality and performance of the proposed approach with a variety of examples, and provide comparisons with state-of-the-art methods.

ABSTRACT The establishment of trust relations is recognized as an important approach to initiate cooperation between unfamiliar entities in distributed computing environments. Particularly, trust relations can play a key role to help nodes make decisions on service discoveries in wireless environments. Visualizing trust relations can help us understand and analyze the real‐time threats that the distributed system is facing and consequently identify the attacks in order to deploy corresponding countermeasures. The main challenges to visualize trust in such distributed environments are as follows: (i) visually reorganizing and presenting trust relations to show underlying cooperation between attackers; (ii) mapping attacker behaviors to visual patterns in an ambiguity‐free manner; (iii) properly organizing and encoding other contextual information to analyze how these factors play along with trust. In this paper, we introduce TrustVis, a tool that helps users visually analyze trust relations to identify attacks in a semi‐automatic fashion. In parallel with a running trust evaluation engine that actively monitors trust relations, TrustVis reorganizes and presents trust relations with a matrix to map the cooperative attack schemes to visual patterns. By coordinating the trust matrix and other contextual information into a multi‐faceted view, TrustVis incorporates the intelligence of domain experts to interactively monitor the networked system toward identifying both the attackers' identities and the adopted attack schemes. Copyright © 2012 John Wiley & Sons, Ltd.

Image-space line integral convolution (LIC) is a popular scheme for visualizing surface vector fields due to its simplicity and high efficiency. To avoid inconsistencies or color blur during the user interactions, existing approaches employ surface parameterization or 3D volume texture schemes. However, they often require expensive computation or memory cost, and cannot achieve consistent results in terms of both the granularity and color distribution on different scales. This paper introduces a novel image-space surface flow visualization approach that preserves the coherence during user interactions. To make the noise texture under different viewpoints coherent, we propose to precompute a sequence of mipmap noise textures in a coarse-to-fine manner for consistent transition, and map the textures onto each triangle with randomly assigned and constant texture coordinates. Further, a standard image-space LIC is performed to generate the flow texture. The proposed approach is simple and GPU-friendly, and can be easily combined with various texture-based flow visualization techniques. By leveraging viewpoint-dependent backward tracing and mipmap noise phase, our method can be incorporated with the image-based flow visualization (IBFV) technique for coherent visualization of unsteady flows. We demonstrate consistent and highly efficient flow visualization on a variety of data sets.

Image-space line integral convolution (LIC) is a popular approach for visualizing surface vector fields due to its simplicity and high efficiency. To avoid inconsistencies or color blur during the user interactions in the image-space approach, some methods use surface parameterization or 3D volume texture for the effect of smooth transition, which often require expensive computational or memory cost. Furthermore, those methods cannot achieve consistent LIC results in both granularity and color distribution on different scales. This paper introduces a novel image-space LIC for surface flows that preserves the texture coherence during user interactions. To make the noise textures under different viewpoints coherent, we propose a simple texture mapping technique that is local, robust and effective. Meanwhile, our approach pre-computes a sequence of mipmap noise textures in a coarse-to-fine manner, leading to consistent transition when the model is zoomed. Prior to perform LIC in the image space, the mipmap noise textures are mapped onto each triangle with randomly assigned texture coordinates. Then, a standard image-space LIC based on the projected vector fields is performed to generate the flow texture. The proposed approach is simple and very suitable for GPU acceleration. Our implementation demonstrates consistent and highly efficient LIC visualization on a variety of datasets.

Graph is an intuitive and powerful tool to present relationships between entities. Efficiently visualizing graphs with node-link diagrams is a great challenge due to visual clutter induced by edge crossing and node-edge overlapping. Many edge bundling methods are proposed to disclose high-level edge patterns. Though previous methods can successfully reveal the coarse graph structure, the relation patterns at individual node level can be overlooked. In addition, many edge bundling algorithms are computationally complex to prevent them from scaling up for extremely large graphs. In this paper, we propose SideKnot, an efficient node-based bundling method to cluster and knot edges. SideKnot is light, runs faster than most existing algorithms, and can disclose the relation patterns of an individual node such as its standing in the graph and pattern of relations with its neighbors.

This paper presents a time line visualization approach, which allows users to study temporal relationships through encoding their interested data properties to time lines with different shapes and locations. Specifically, our approach extracts key data features as virtual words and uses them to encode various data properties. The distributions of virtual words across time are further applied to study various temporal relationships by generating time lines, which renders sampled time steps as points and temporal sequence as a line. Our approach consists of the three following components. First, we select feature points and collect feature descriptors to build a space of data properties, where virtual words are extracted as representative vectors. Second, the virtual words are applied to characterize feature points and their distribution statistics are used to measure temporal relationships. Third, we present several case studies to visualize time lines for different data visualization and analysis purposes. Our time line visualization can be used for both summarization and exploration of overall temporal relationships. We demonstrate with examples that time lines can serve as effective exploration, comparison, and visualization tools to study time-varying datasets.

When analyzing metabolomics data, cancer care researchers are searching for differences between known healthy samples and unhealthy samples. By analyzing and understanding these differences, researchers hope to identify cancer biomarkers. Due to the size and complexity of the data produced, however, analysis can still be very slow and time consuming. This is further complicated by the fact that datasets obtained will exhibit incidental differences in intensity and retention time, not related to actual chemical differences in the samples being evaluated. Additionally, automated tools to correct these errors do not always produce reliable results. This work presents a new analytics system that enables interactive comparative visualization and analytics of metabolomics data obtained by two-dimensional gas chromatography-mass spectrometry (GC × GC-MS). The key features of this system are the ability to produce visualizations of multiple GC × GC-MS data sets, and to explore those data sets interactively, allowing a user to discover differences and features in real time. The system provides statistical support in the form of difference, standard deviation, and kernel density estimation calculations to aid users in identifying meaningful differences between samples. These are combined with novel transfer functions and multiform, linked visualizations in order to provide researchers with a powerful new tool for GC × GC-MS exploration and bio-marker discovery.

For many applications involving time series data, people are often interested in the changes of item values over time as well as their ranking changes. For example, people search many words via search engines like Google and Bing every day. Analysts are interested in both the absolute searching number for each word as well as their relative rankings. Both sets of statistics may change over time. For very large time series data with thousands of items, how to visually present ranking changes is an interesting challenge. In this paper, we propose RankExplorer, a novel visualization method based on ThemeRiver to reveal the ranking changes. Our method consists of four major components: 1) a segmentation method which partitions a large set of time series curves into a manageable number of ranking categories; 2) an extended ThemeRiver view with embedded color bars and changing glyphs to show the evolution of aggregation values related to each ranking category over time as well as the content changes in each ranking category; 3) a trend curve to show the degree of ranking changes over time; 4) rich user interactions to support interactive exploration of ranking changes. We have applied our method to some real time series data and the case studies demonstrate that our method can reveal the underlying patterns related to ranking changes which might otherwise be obscured in traditional visualizations.

While many approaches have been developed to visualize 3D city scenes, most of them exhibit the visualization results in a uniform rendering style. This paper presents an expressive rendering approach for visualizing large-scale 3D city scenes with various rendering styles integrated in a seamless way. Each view is actually a combination of the photorealistic rendering, the nonphotorealistic rendering, and the line drawing, so as to highlight the information that is interesting for the users and de-emphasize the other that is less important. At run-time, the users are allowed to specify their interested locations with pre-determined 3D landmarks. Our system automatically computes the salience of each location and visualize the entire scene with emphasis in the area of interests. The GPU-based implementation enables real-time performance, and demonstrates outstanding practicality.

This paper presents a real-time foveation rendering algorithm to simulate the gaze contingent vision of a 3Dscene. The novelty of our approach lies in two aspects, namely, an image-space adaptive sampling mask adhering to the Contrast Sensitivity Function (CSF) of the HVS, an image based multi-layer representation of the 3D scene constructed by depth peeling. We demonstrate high quality and performance of the proposed approach with a variety of examples, and provide comparisons with recent methods.

Abstract Due to 3D occlusion, the specification of proper opacities in direct volume rendering is a time‐consuming and unintuitive process. The visibility histograms introduced by Correa and Ma reflect the effect of occlusion by measuring the influence of each sample in the histogram to the rendered image. However, the visibility is defined on individual samples, while volume exploration focuses on conveying the spatial relationships between features. Moreover, the high computational cost and large memory requirement limits its application in multi‐dimensional transfer function design. In this paper, we extend visibility histograms to feature visibility, which measures the contribution of each feature in the rendered image. Compared to visibility histograms, it has two distinctive advantages for opacity specification. First, the user can directly specify the visibilities for features and the opacities are automatically generated using an optimization algorithm. Second, its calculation requires only one rendering pass with no additional memory requirement. This feature visibility based opacity specification is fast and compatible with all types of transfer function design. Furthermore, we introduce a two‐step volume exploration scheme, in which an automatic optimization is first performed to provide a clear illustration of the spatial relationship and then the user adjusts the visibilities directly to achieve the desired feature enhancement. The effectiveness of this scheme is demonstrated by experimental results on several volumetric datasets.

To date, comparing and visualizing genome sequences remain challenging due to the large genome size. Existing approaches take advantage of the stable property of oligonucleotides and exhibit the main characteristics of the whole genome, yet they commonly fail to show progression patterns of the genome adjustably. This paper presents a novel visual encoding technique, which not only supports the binning process (phylogenetic analysis), but also allows the sequential analysis of the genome. The key idea is to regard the combination of each k -nucleotide and its reverse complement as a visual word, and to represent a long genome sequence with a list of local statistical feature vectors derived from the local frequency of the visual words. Experimental results on a variety of examples demonstrate that the presented approach has the ability to quickly and intuitively visualize DNA sequences, and to help the user identify regions of differences among multiple datasets.

Color vision deficiency (CVD) affects a high percentage of the population worldwide. When seeing a volume visualization result, persons with CVD may be incapable of discriminating the classification information expressed in the image if the color transfer function or the color blending used in the direct volume rendering is not appropriate. Conventional methods used to address this problem adopt advanced image recoloring techniques to enhance the rendering results frame-by-frame; unfortunately, problematic perceptual results may still be generated. This paper proposes an alternative solution that complements the image recoloring scheme by reconfiguring the components of the direct volume rendering (DVR) pipeline. Our approach optimizes the mapped colors of a transfer function to simulate CVD-friendly effect that is generated by applying the image recoloring to the results with the initial transfer function. The optimization process has a low computational complexity, and only needs to be performed once for a given transfer function. To achieve detail-preserving and perceptually natural semi-transparent effects, we introduce a new color composition mode that works in the color space of dichromats. Experimental results and a pilot study demonstrates that our approach can yield dichromats-friendly and consistent volume visualization in real-time.

The multidimensional transfer function is a flexible and effective tool for exploring volume data. However, designing an appropriate transfer function is a trial-and-error process and remains a challenge. In this paper, we propose a novel volume exploration scheme that explores volumetric structures in the feature space by modeling the space using the Gaussian mixture model (GMM). Our new approach has three distinctive advantages. First, an initial feature separation can be automatically achieved through GMM estimation. Second, the calculated Gaussians can be directly mapped to a set of elliptical transfer functions (ETFs), facilitating a fast pre-integrated volume rendering process. Third, an inexperienced user can flexibly manipulate the ETFs with the assistance of a suite of simple widgets, and discover potential features with several interactions. We further extend the GMM-based exploration scheme to time-varying data sets using an incremental GMM estimation algorithm. The algorithm estimates the GMM for one time step by using itself and the GMM generated from its previous steps. Sequentially applying the incremental algorithm to all time steps in a selected time interval yields a preliminary classification for each time step. In addition, the computed ETFs can be freely adjusted. The adjustments are then automatically propagated to other time steps. In this way, coherent user-guided exploration of a given time interval is achieved. Our GPU implementation demonstrates interactive performance and good scalability. The effectiveness of our approach is verified on several data sets.

This paper presents a novel object-space line drawing algorithm that can depict shapes with view-dependent feature lines in real time. Strongly inspired by the Laplacian-of-Gaussian (LoG) edge detector in image processing, we define Laplacian lines as the zero-crossing points of the Laplacian of the surface illumination. Compared to other view-dependent feature lines, Laplacian lines are computationally efficient because most expensive computations can be preprocessed. We further extend Laplacian lines to volumetric data and develop the algorithm to compute volumetric Laplacian lines without isosurface extraction. We apply the proposed Laplacian lines to a wide range of real-world models and demonstrate that Laplacian lines are more efficient than the existing computer generated feature lines, and can be used in interactive graphics applications.

This paper proposes a novel visualization approach, which can depict the variations between different human motion data. This is achieved by representing the time dimension of each animation sequence with a sequential curve in a locality-preserving reference 2D space, called the motion track representation. The principal advantage of this representation over standard representations of motion capture data - generally either a keyframed timeline or a 2D motion map in its entirety - is that it maps the motion differences along the time dimension into parallel perceptible spatial dimensions but at the same time captures the primary content of the source data. Latent semantic differences that are difficult to be visually distinguished can be clearly displayed, favoring effective summary, clustering, comparison and analysis of motion database.

This paper presents an interactive transfer function design tool based on ellipsoidal Gaussian transfer functions (ETFs). Our approach explores volumetric features in the statistical space by modeling the space using the Gaussian mixture model (GMM) with a small number of Gaussians to maximize the likelihood of feature separation. Instant visual feedback is possible by mapping these Gaussians to ETFs and analytically integrating these ETFs in the context of the pre-integrated volume rendering process. A suite of intuitive control widgets is designed to offer automatic transfer function generation and flexible manipulations, allowing an inexperienced user to easily explore undiscovered features with several simple interactions. Our GPU implementation demonstrates interactive performance and plausible scalability which compare favorably with existing solutions. The effectiveness of our approach has been verified on several datasets.

Abstract This paper presents a digital storytelling approach that generates automatic animations for time‐varying data visualization. Our approach simulates the composition and transition of storytelling techniques and synthesizes animations to describe various event features. Specifically, we analyze information related to a given event and abstract it as an event graph, which represents data features as nodes and event relationships as links. This graph embeds a tree‐like hierarchical structure which encodes data features at different scales. Next, narrative structures are built by exploring starting nodes and suitable search strategies in this graph. Different stages of narrative structures are considered in our automatic rendering parameter decision process to generate animations as digital stories. We integrate this animation generation approach into an interactive exploration process of time‐varying data, so that more comprehensive information can be provided in a timely fashion. We demonstrate with a storm surge application that our approach allows semantic visualization of time‐varying data and easy animation generation for users without special knowledge about the underlying visualization techniques.

In this paper, we propose an image driven shape deformation approach for stylizing a 3D mesh using styles learned from existing 2D illustrations. Our approach models a 2D illustration as a planar mesh and represents the shape styles with four components: the object contour, the context curves, user-specified features and local shape details. After the correspondence between the input model and the 2D illustration is established, shape stylization is formulated as a style-constrained differential mesh editing problem. A distinguishing feature of our approach is that it allows users to directly transfer styles from hand-drawn 2D illustrations with individual perception and cognition, which are difficult to identify and create with 3D modeling and editing approaches. We present a sequence of challenging examples including unrealistic and exaggerated paintings to illustrate the effectiveness of our approach.

Diffusion tensor fields reveal the underlying anatomical structures in biological tissues such as neural fibers in the brain. Most current methods for visualizing the diffusion tensor field can be categorized into two classes: integral curves and glyphs. Integral curves are continuous and represent the underlying fiber structures, but are prone to integration error and loss of local information. Glyphs are useful for representing local tensor information, but do not convey the connectivity in the anatomical structures well. We introduce a simple yet effective visualization technique that extends the streamball method in flow visualization to tensor ellipsoids. Each tensor ellipsoid represents a local tensor, and either blends with neighboring tensors or breaks away from them depending on their orientations and anisotropies. The resulting visualization shows the connectivity information in the underlying anatomy while characterizing the local tenors in detail. By interactively changing an iso-value parameter, users can examine the diffusion tensor field in the entire spectrum between the continuous integral curves and the discrete glyphs. Expert evaluation indicates that this method conveys very useful visual information about local anisotropy in white matter fibers. Such information was previously unavailable in tractography models. Our method provides a visual tool for assessing variability in DTI fiber tract integrity and its relation to function.

This paper presents a novel object-space line drawing algorithm that can depict shape with view dependent feature lines in real-time. Strongly inspired by the Laplacian-of-Gaussian (LoG) edge detector in image processing, we define Laplacian Lines as the zero-crossing points of the Laplacian of the surface illumination. Compared to other view dependent features, Laplacian lines are computationally efficient because most expensive computations can be pre-processed. Thus, Laplacian lines are very promising for interactively illustrating large-scale models.

Abstract In this paper, we present a novel two‐dimensional (2D) shape context preserving image manipulation approach which constructs and manipulates a 2D mesh with a new differential mesh editing algorithm. We introduce a novel shape context descriptor and integrate it into the deformation framework, facilitating shape‐preserving deformation for 2D anatomical illustrations. Our new scheme utilizes an analogy based shape transfer technique in order to learn shape styles from reference images. Experimental results show that visually plausible deformation can be quickly generated from an existing example at interactive frame rates. An experienced artist has evaluated our approach and his feedback is quite encouraging.

Abstract Multi‐dimensional transfer functions are commonly used in rectilinear volume renderings to effectively portray materials, material boundaries and even subtle variations along boundaries. However, most unstructured grid rendering algorithms only employ one‐dimensional transfer functions. This paper proposes a novel pre‐integrated Projected Tetrahedra (PT) rendering technique that applies bivariate transfer functions on unstructured grids. For each type of bivariate transfer function, an analytical form that pre‐integrates the contribution of a ray segment in one tetrahedron is derived, and can be precomputed as a lookup table to compute the color and opacity in a projected tetrahedron on‐the‐fly. Further, we show how to approximate the integral using the pre‐integration method for faster unstructured grid rendering. We demonstrate the advantages of our approach with a variety of examples and comparisons with one‐dimensional transfer functions.

Abstract This paper presents an interactive volume modeling method that constructs skeletal muscles from an existing volumetric dataset. Our approach provides users with an intuitive modeling interface and produces compelling results that conform to the characteristic anatomy in the input volume. The algorithmic core of our method is an intuitive anatomy classification approach, suited to accommodate spatial constraints on the muscle volume. The presented work is useful in illustrative visualization, volumetric information fusion and volume illustration that involve muscle modeling, where the spatial context should be faithfully preserved.

The semi-transparent nature of direct volume rendered images is useful to depict layered structures in a volume. However, obtaining a semi-transparent result with the layers clearly revealed is difficult and may involve tedious adjustment on opacity and other rendering parameters. Furthermore, the visual quality of layers also depends on various perceptual factors. In this paper, we propose an auto-correction method for enhancing the perceived quality of the semi-transparent layers in direct volume rendered images. We introduce a suite of new measures based on psychological principles to evaluate the perceptual quality of transparent structures in the rendered images. By optimizing rendering parameters within an adaptive and intuitive user interaction process, the quality of the images is enhanced such that specific user requirements can be met. Experimental results on various datasets demonstrate the effectiveness and robustness of our method.

Visual exploration is essential to the visualization and analysis of densely sampled 3D DTI fibers in biological specimens, due to the high geometric, spatial, and anatomical complexity of fiber tracts. Previous methods for DTI fiber visualization use zooming, color-mapping, selection, and abstraction to deliver the characteristics of the fibers. However, these schemes mainly focus on the optimization of visualization in the 3D space where cluttering and occlusion make grasping even a few thousand fibers difficult. This paper introduces a novel interaction method that augments the 3D visualization with a 2D representation containing a low-dimensional embedding of the DTI fibers. This embedding preserves the relationship between the fibers and removes the visual clutter that is inherent in 3D renderings of the fibers. This new interface allows the user to manipulate the DTI fibers as both 3D curves and 2D embedded points and easily compare or validate his or her results in both domains. The implementation of the framework is GPU based to achieve real-time interaction. The framework was applied to several tasks, and the results show that our method reduces the user's workload in recognizing 3D DTI fibers and permits quick and accurate DTI fiber selection.

Medical illustration has demonstrated its effectiveness to depict salient anatomical features while hiding the irrelevant details. Current solutions are ineffective for visualizing fibrous structures such as muscle, because typical datasets (CT or MRI) do not contain directional details. In this paper, we introduce a new muscle illustration approach that leverages diffusion tensor imaging (DTI) data and example-based texture synthesis techniques. Beginning with a volumetric diffusion tensor image, we reformulate it into a scalar field and an auxiliary guidance vector field to represent the structure and orientation of a muscle bundle. A muscle mask derived from the input diffusion tensor image is used to classify the muscle structure. The guidance vector field is further refined to remove noise and clarify structure. To simulate the internal appearance of the muscle, we propose a new two-dimensional example based solid texture synthesis algorithm that builds a solid texture constrained by the guidance vector field. Illustrating the constructed scalar field and solid texture efficiently highlights the global appearance of the muscle as well as the local shape and structure of the muscle fibers in an illustrative fashion. We have applied the proposed approach to five example datasets (four pig hearts and a pig leg), demonstrating plausible illustration and expressiveness.

The use of multi-dimensional transfer functions for direct volume rendering has been shown to be an effective means of extracting materials and their boundaries for both scalar and multivariate data. The most common multi-dimensional transfer function consists of a two-dimensional (2D) histogram with axes representing a subset of the feature space (e.g., value vs. value gradient magnitude), with each entry in the 2D histogram being the number of voxels at a given feature space pair. Users then assign color and opacity to the voxel distributions within the given feature space through the use of interactive widgets (e.g., box, circular, triangular selection). Unfortunately, such tools lead users through a trial-and-error approach as they assess which data values within the feature space map to a given area of interest within the volumetric space. In this work, we propose the addition of non-parametric clustering within the transfer function feature space in order to extract patterns and guide transfer function generation. We apply a non-parametric kernel density estimation to group voxels of similar features within the 2D histogram. These groups are then binned and colored based on their estimated density, and the user may interactively grow and shrink the binned regions to explore feature boundaries and extract regions of interest. We also extend this scheme to temporal volumetric data in which time steps of 2D histograms are composited into a histogram volume. A three-dimensional (3D) density estimation is then applied, and users can explore regions within the feature space across time without adjusting the transfer function at each time step. Our work enables users to effectively explore the structures found within a feature space of the volume and provide a context in which the user can understand how these structures relate to their volumetric data. We provide tools for enhanced exploration and manipulation of the transfer function, and we show that the initial transfer function generation serves as a reasonable base for volumetric rendering, reducing the trial-and-error overhead typically found in transfer function design.

Abstract Diffusion tensor imaging (DTI) has been used to generate fibrous structures in both brain white matter and muscles. Fiber clustering groups the DTI fibers into spatially and anatomically related tracts. As an increasing number of fiber clustering methods have been recently developed, it is important to display, compare, and explore the clustering results efficiently and effectively. In this paper, we present an anatomical visualization technique that reduces the geometric complexity of the fiber tracts and emphasizes the high‐level structures. Beginning with a volumetric diffusion tensor image, we first construct a hierarchical clustering representation of the fiber bundles. These bundles are then reformulated into a 3D multi‐valued volume data. We then build a set of geometric hulls and principal fibers to approximate the shape and orientation of each fiber bundle. By simultaneously visualizing the geometric hulls, individual fibers, and other data sets such as fractional anisotropy, the overall shape of the fiber tracts are highlighted, while preserving the fibrous details. A rater with expert knowledge of white matter structure has evaluated the resulting interactive illustration and confirmed the improvement over straightforward DTI fiber tract visualization.

Abstract We present a new image completion method based on an additional large displacement view (LDV) of the same scene for faithfully repairing large missing regions on the target image in an automatic way. A coarse‐to‐fine distortion correction algorithm is proposed to minimize the perspective distortion in the corresponding parts for the common scene regions on the LDV image. First, under the assumption of a planar scene, the LDV image is warped according to a homography to generate the initial correction result. Second, the residual distortions in the common known scene regions are revealed by means of a mismatch detection mechanism and relaxed by energy optimization of overlap correspondences, with the expectations of color constancy and displacement field smoothness. The fundamental matrix for the two views is then computed based on the reliable correspondence set. Third, under the constraints of epipolar geometry, displacement field smoothness and color consistency of the neighboring pixels, the missing pixels are orderly restored according to a specially defined repairing priority function. We finally eliminate the ghost effect between the repaired region and its surroundings by Poisson image blending. Experimental results demonstrate that our method outperforms recent state‐of‐the‐art image completion methods for repairing large missing area with complex structure information.

Summary form only given. Traditionally, synthesizing the animation of floral blossom is accomplished mostly manually and hence a time-consuming and laborious task. In this paper we propose an interactive biology-based flower modeling and animating approach. We model the initial shape of the floral components including petal, pistil, stamen and pedicel, and two terminal status of floral blossom with a parameterized sketching interface. The subtle geometry of the whole flower is created according to the phyllotactic rules. We then extract a set of growth parameters for describing the floral blossom, and build a dynamic growth model to regulate the continuous morphing of floral components in the process of blossom. Finally, a sequence of naturally deforming flower models is generated with these parameters. Our initial experimental results demonstrate that our approach can efficiently generate visually pleasing simulation of floral blossom, which is consistent with biological rules.

Abstract We introduce a novel volume illustration technique for regularly sampled volume datasets. The fundamental difference between previous volume illustration algorithms and ours is that our results are shape‐aware, as they depend not only on the rendering styles, but also the shape styles. We propose a new data structure that is derived from the input volume and consists of a distance volume and a segmentation volume. The distance volume is used to reconstruct a continuous field around the object boundary, facilitating smooth illustrations of boundaries and silhouettes. The segmentation volume allows us to abstract or remove distracting details and noise, and apply different rendering styles to different objects and components. We also demonstrate how to modify the shape of illustrated objects using a new 2D curve analogy technique. This provides an interactive method for learning shape variations from 2D hand‐painted illustrations by drawing several lines. Our experiments on several volume datasets demonstrate that the proposed approach can achieve visually appealing and shape‐aware illustrations. The feedback from medical illustrators is quite encouraging.

Abstract We present a data‐driven approach that synthesizes tree animations from a set of pre‐computed motion data. Our approach improves previous motion synthesis algorithms for character animation in several aspects. We first introduce a simple yet effective sampling scheme to generate a rich and reusable motion database for each tree model. We also propose a novel technique to generate a fine set of transitions that are uniformly distributed in the motion database. The transition lengths are adaptively determined according to the similarity of the transiting frame pairs. In the runtime, we employ a greedy searching algorithm to synthesize smooth tree animations under an adjustable wind condition. Experimental results show that our approach achieves comparable quality to physically based methods, while in orders of magnitude faster performance. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract We propose an efficient approach for authoring dynamic and realistic waterfall scenes based on an acquired video sequence. Traditional video based techniques generate new images by synthesizing 2D samples, i.e., texture sprites chosen from a video sequence. However, they are limited to one fixed viewpoint and cannot provide arbitrary walkthrough into 3D scenes. Our approach extends this scheme by synthesizing dynamic 2D texture sprites and projecting them into 3D space. We first generate a set of basis texture sprites, which capture the representative appearance and motions of waterfall scenes contained in the video sequence. To model the shape and motion of a new waterfall scene, we interactively construct a set of flow lines taking account of physical principles. Along each flow line, the basis texture sprites are manipulated and animated dynamically, yielding a sequence of dynamic texture sprites in 3D space. These texture sprites are displayed using the point splatting technique, which can be accelerated efficiently by graphics hardware. By choosing varied basis texture sprites, waterfall scenes with different appearance and shapes can be conveniently simulated. The experimental results demonstrate that our approach achieves realistic effects and real‐time frame rates on consumer PC platforms. Copyright © 2006 John Wiley & Sons, Ltd.

Abstract We present an efficient approach that merges the virtual objects into video sequences taken by a freely moving camera in a realistic manner. The composition is visually and geometrically consistent through three main steps. First, a robust camera tracking algorithm based on key frames is proposed, which precisely recovers the focal length with a novel multi‐frame strategy. Next, the concerned 3D models of the real scenes are reconstructed by means of an extended multi‐baseline algorithm. Finally, the virtual objects in the form of 3D models are integrated into the real scenes, with special cares on the interaction consistency including shadow casting, occlusions, and object animation. A variety of experiments have been implemented, which demonstrate the robustness and efficiency of our approach. Copyright © 2006 John Wiley & Sons, Ltd.

Most of the existing algorithms for protein similarity comparison focus on the sequence comparison and structure comparison. However, the 3D structure of a protein is determined by the force field generated by all of its atoms in essence. Thus the similarity of force field implies the similarity of 3D structure. In this paper, we propose a novel approach to compare the similarity of protein's force fields. First, we use blurred map to sample the force field into volumetric data sets. Second, the volume data set is resampled into a unified resolution. Third, the data set is band-pass filtered and quantized to reveal its physical attributes. The resulting voxels are then normalized into a canonical coordinate system concerning the center of mass and scale. Subsequently, a series of uniformly spaced concentric shells around the center of mass are constructed, based on which spherical harmonics analysis (SHA) is applied. The coefficient of SHA constitutes rotation invariant spectrum descriptors which are used to measure the similarity between two data sets. The algorithm has been performed on a set of proteins (taken from PDB) and the preliminary results are fairly inspiring.

In this paper, we present an efficient voxelization algorithm for complex polygonal models by exploiting newest programmable graphics hardware. We first convert the model into three discrete voxel spaces according to its surface orientation. The resultant voxels are encoded as 2D textures and stored in three intermediate sheet buffers called directional sheet buffers. These buffers are finally synthesized into one worksheet, which records the volumetric representation of the target. The whole algorithm traverses the geometric model only once and is accomplished entirely in GPU (graphics processing unit), achieving real-time frame rate for models with up to 2 million triangles.

We present a hardware-accelerated adaptive EWA (elliptical weighted average) volume splatting algorithm. EWA splatting combines a Gaussian reconstruction kernel with a low-pass image filter for high image quality without aliasing artifacts or excessive blurring. We introduce a novel adaptive filtering scheme to reduce the computational cost of EWA splatting. We show how this algorithm can be efficiently implemented on modern graphics processing units (GPUs). Our implementation includes interactive classification and fast lighting. To accelerate the rendering we store splat geometry and 3D volume data locally in GPU memory. We present results for several rectilinear volume datasets that demonstrate the high image quality and interactive rendering speed of our method.

In this paper we present an interactive collision detection algorithm for complex and deformable objects. For two target models, our approach rapidly calculates their region of interests (ROI), which is the overlapping of their axis aligned bounding boxes (AABBs), in CPU. The surfaces of both models inside the ROI are then voxelized using a novel GPU-based real-time voxelization method. The resultant volumes are represented by two 2D textures in video memory. The collision query is efficiently accomplished by comparing these 2D textures in GPU. The algorithm is robust to handle arbitrary shapes, no matter geometric models are convex or concave, closed or open, rigid or deformable. Our preliminary implementation achieves interactive frame rate for complex models with up to one million triangles on commodity desktop PCs.