Conversational interfaces are increasingly used for data analysis, enabling data workers to express complex analytical intents in natural language. Yet, these interactions unfold as long, linear transcripts that are misaligned with the iterative, nonlinear nature of real-world analyses. Revisiting and summarizing conversations for different contexts is therefore challenging. This paper investigates how data workers navigate, make sense of, and communicate prior analytical conversations. To study behaviors beyond those supported by standard interfaces (i.e., scrolling and keyword search), we develop a design probe that supplements analytical conversations with structured elements and affordances (e.g., filtering, multi-level navigation and detail-on-demand). In a user study (n = 10), participants used the probe to navigate and communicate past analyses, fulfilling information needs (recall, reorient, prioritize) through navigation strategies (visual recall, sequential and abstractive) and summarization practices (adding process details and context). Based on these findings, we discuss design implications to support re-visitation and communication of analytical conversations.

Large Language Models (LLMs) are transforming Conversational Visual Analytics (CVA) by enabling data analysis through natural language. However, evaluating LLMs for CVA remains a challenge: requiring programming expertise, overlooking real-world complexity, and lacking interpretable metrics for multi-format (visualizations and text) outputs. Through interviews with 22 CVA developers and 16 end-users, we identified use cases, evaluation criteria and workflows. We present Lexara, a user-centered evaluation toolkit for CVA that operationalizes these insights into: (i) test cases spanning real-world scenarios; (ii) interpretable metrics covering visualization quality (data fidelity, semantic alignment, functional correctness, design clarity) and language quality (factual grounding, analytical reasoning, conversational coherence) using rule-based and LLM-as-a-Judge methods; and (iii) an interactive toolkit enabling experimental setup and multi-format and multi-level exploration of results without programming expertise. We conducted a two-week diary study with six CVA developers, drawn from our initial cohort of 22. Their feedback demonstrated Lexara’s effectiveness for guiding appropriate model and prompt selection.

Text in dashboards plays multiple critical roles, including providing context, offering insights, guiding interactions, and summarizing key information. Despite its importance, most dashboarding tools focus on visualizations and offer limited support for text authoring. To address this gap, we developed Plume, a system to help authors craft effective dashboard text. Through a formative review of exemplar dashboards, we created a typology of text parameters and articulated the relationship between visual placement and semantic connections, which informed Plume's design. Plume employs large language models (LLMs) to generate contextually appropriate content and provides guidelines for writing clear, readable text. A preliminary evaluation with 12 dashboard authors explored how assisted text authoring integrates into workflows, revealing strengths and limitations of LLM-generated text and the value of our human-in-the-loop approach. Our findings suggest opportunities to improve dashboard authoring tools by better supporting the diverse roles that text plays in conveying insights.

Journaling plays a crucial role in managing chronic conditions by allowing patients to document symptoms and medication intake, providing essential data for long-term care. While valuable, traditional journaling methods often rely on static, self-directed entries, lacking interactive feedback and real-time guidance. This gap can result in incomplete or imprecise information, limiting its usefulness for effective treatment. To address this gap, we introduce PATRIKA, an AI-enabled prototype designed specifically for people with Parkinson's disease (PwPD). The system incorporates cooperative conversation principles, clinical interview simulations, and personalization to create a more effective and user-friendly journaling experience. Through two user studies with PwPD and iterative refinement of PATRIKA, we demonstrate conversational journaling's significant potential in patient engagement and collecting clinically valuable information. Our results showed that generating probing questions PATRIKA turned journaling into a bi-directional interaction. Additionally, we offer insights for designing journaling systems for healthcare and future directions for promoting sustained journaling.

Mining and conveying actionable insights from complex data is a key challenge of exploratory data analysis (EDA) and storytelling.To address this challenge, we present a design space for actionable EDA and storytelling.Synthesizing theory and expert interviews, we highlight how semantic precision, rhetorical persuasion, and pragmatic relevance underpin effective EDA and storytelling.We also show how this design space subsumes common challenges in actionable EDA and storytelling, such as identifying appropriate analytical strategies and leveraging relevant domain knowledge.Building on the potential of LLMs to generate coherent narratives with commonsense reasoning, we contribute Jupybara, an AI-enabled assistant for actionable EDA and storytelling implemented as a Jupyter Notebook extension.Jupybara employs two strategiesdesign-space-aware prompting and multi-agent architectures-to operationalize our design space.An expert evaluation confirms Jupybara's usability, steerability, explainability, and reparability, as well as the effectiveness of our strategies in operationalizing the design space framework with LLMs.

Abstract Data‐driven storytelling has gained prominence in journalism and other data reporting fields. However, the process of creating these stories remains challenging, often requiring the integration of effective visualizations with compelling narratives to form a cohesive, interactive presentation. To help streamline this process, we present an integrated authoring framework and system, D ata W eaver , that supports both visualization‐to‐text and text‐to‐visualization composition. D ata W eaver enables users to create data narratives anchored to data facts derived from “call‐out” interactions, i.e., user‐initiated highlights of visualization elements that prompt relevant narrative content. In addition to this “vis‐to‐text” composition, D ata W eaver also supports a “text‐initiated” approach, generating relevant interactive visualizations from existing narratives. Key findings from an evaluation with 13 participants highlighted the utility and usability of D ata W eaver and the effectiveness of its integrated authoring framework. The evaluation also revealed opportunities to enhance the framework by refining filtering mechanisms and visualization recommendations and better support authoring creativity by introducing advanced customization options.

Dashboards remain ubiquitous tools for analyzing data and disseminating the findings. Understanding the range of dashboard designs, from simple to complex, can support development of authoring tools that enable end-users to meet their analysis and communication goals. Yet, there has been little work that provides a quantifiable, systematic, and descriptive overview of dashboard design patterns. Instead, existing approaches only consider a handful of designs, which limits the breadth of patterns that can be surfaced. More quantifiable approaches, inspired by machine learning (ML), are presently limited to single visualizations or capture narrow features of dashboard designs. To address this gap, we present an approach for modeling the content and composition of dashboards using a graph representation. The graph decomposes dashboard designs into nodes featuring content "blocks'; and uses edges to model "relationships", such as layout proximity and interaction, between nodes. To demonstrate the utility of this approach, and its extension over prior work, we apply this representation to derive a census of 25,620 dashboards from Tableau Public, providing a descriptive overview of the core building blocks of dashboards in the wild and summarizing prevalent dashboard design patterns. We discuss concrete applications of both a graph representation for dashboard designs and the resulting census to guide the development of dashboard authoring tools, making dashboards accessible, and for leveraging AI/ML techniques. Our findings underscore the importance of meeting users where they are by broadly cataloging dashboard designs, both common and exotic.

There is increased interest in understanding the interplay between text and visuals in the field of data visualization. However, this attention has predominantly been on the use of text in standalone visualizations (such as text annotation overlays) or augmenting text stories supported by a series of independent views. In this paper, we shift from the traditional focus on single-chart annotations to characterize the nuanced but crucial communication role of text in the complex environment of interactive dashboards. Through a survey and analysis of 190 dashboards in the wild, plus 13 expert interview sessions with experienced dashboard authors, we highlight the distinctive nature of text as an integral component of the dashboard experience, while delving into the categories, semantic levels, and functional roles of text, and exploring how these text elements are coalesced by dashboard authors to guide and inform dashboard users. Our contributions are threefold. First, we distill qualitative and quantitative findings from our studies to characterize current practices of text use in dashboards, including a categorization of text-based components and design patterns. Second, we leverage current practices and existing literature to propose, discuss, and validate recommended practices for text in dashboards, embodied as a set of 12 heuristics that underscore the semantic and functional role of text in offering navigational cues, contextualizing data insights, supporting reading order, among other concerns. Third, we reflect on our findings to identify gaps and propose opportunities for data visualization researchers to push the boundaries on text usage for dashboards, from authoring support and interactivity to text generation and content personalization. Our research underscores the significance of elevating text as a first-class citizen in data visualization, and the need to support the inclusion of textual components and their interactive affordances in dashboard design.

Natural language and search interfaces intuitively facilitate data exploration and provide visualization responses to diverse analytical queries based on the underlying datasets. However, these interfaces often fail to interpret more complex analytical intents, such as discerning subtleties and quantifiable differences between terms like “bump’’ and “spike’’ in the context of COVID cases, for example. We address this gap by extending the capabilities of a data exploration search interface for interpreting semantic concepts in time series trends. We first create a comprehensive dataset of semantic concepts by mapping quantifiable univariate data trends such as slope and angle to crowdsourced, semantically meaningful trend labels. The dataset contains quantifiable properties that capture the slope-scalar effect of semantic modifiers like “sharply” and “gradually,” as well as multi-line trends (e.g., “peak,” “valley”). We demonstrate the utility of this dataset in SlopeSeeker, a tool that supports natural language querying of quantifiable trends, such as “show me stocks that tanked in 2010.” The tool incorporates novel scoring and ranking techniques based on semantic relevance and visual prominence to present relevant trend chart responses containing these semantic trend concepts. In addition, SlopeSeeker provides a faceted search interface for users to navigate a semantic hierarchy of concepts from general trends (e.g., “increase’’) to more specific ones (e.g., “sharp increase’’). A preliminary user evaluation of the tool demonstrates that the search interface supports greater expressivity of queries containing concepts that describe data trends. We identify potential future directions for leveraging our publicly available quantitative semantics dataset in other data domains and for novel visual analytics interfaces.

Integrating textual content, such as titles, annotations, and captions, with visualizations facilitates comprehension and takeaways during data exploration. Yet current tools often lack mechanisms for integrating meaningful long-form prose with visual data. This paper introduces DASH, a bimodal data exploration tool that supports integrating semantic levels into the interactive process of visualization and text-based analysis. DASH operationalizes a modified version of Lundgard et al.’s semantic hierarchy model that catego-rizes data descriptions into four levels ranging from basic encodings to high-level insights. By leveraging this structured semantic level framework and a large language model’s text generation capabilities, DASH enables the creation of data-driven narratives via drag-and-drop user interaction. Through a preliminary user evaluation, we discuss the utility of DASH’s text and chart integration capabilities when participants perform data exploration with the tool.

Visualization tools now commonly present automated insights highlighting salient data patterns, including correlations, distributions, outliers, and differences, among others. While these insights are valuable for data exploration and chart interpretation, users currently only have a binary choice of accepting or rejecting them, lacking the flexibility to refine the system logic or customize the insight generation process. To address this limitation, we present Groot, a prototype system that allows users to proactively specify and refine automated data insights. The system allows users to directly manipulate chart elements to receive insight recommendations based on their selections. Additionally, Groot provides users with a manual editing interface to customize, reconfigure, or add new insights to individual charts and propagate them to future explorations. We describe a usage scenario to illustrate how these features collectively support insight editing and configuration and discuss opportunities for future work, including incorporating Large Language Models (LLMs), improving semantic data and visualization search, and supporting insight management.

Abstract Understanding and communicating data uncertainty is crucial for making informed decisions in sectors like finance and healthcare. Previous work has explored how to express uncertainty in various modes. For example, uncertainty can be expressed visually with quantile dot plots or linguistically with hedge words and prosody. Our research aims to systematically explore how variations within each mode contribute to communicating uncertainty to the user; this allows us to better understand each mode's affordances and limitations. We completed an exploration of the uncertainty design space based on pilot studies and ran two crowdsourced experiments examining how speech, text, and visualization modes and variants within them impact decision‐making with uncertain data. Visualization and text were most effective for rational decision‐making, though text resulted in lower confidence. Speech garnered the highest trust despite sometimes leading to risky decisions. Results from these studies indicate meaningful trade‐offs among modes of information and encourage exploration of multimodal data representations.

Recent work has shown that when both the chart and caption emphasize the same aspects of the data, readers tend to remember the doubly-emphasized features as takeaways; when there is a mismatch, readers rely on the chart to form takeaways and can miss information in the caption text. Through a survey of 280 chart-caption pairs in real-world sources (e.g., news media, poll reports, government reports, academic articles, and Tableau Public), we find that captions often do not emphasize the same information in practice, which could limit how effectively readers take away the authors' intended messages. Motivated by the survey findings, we present EMPHASISCHECKER, an interactive tool that highlights visually prominent chart features as well as the features emphasized by the caption text along with any mismatches in the emphasis. The tool implements a time-series prominent feature detector based on the Ramer-Douglas-Peucker algorithm and a text reference extractor that identifies time references and data descriptions in the caption and matches them with chart data. This information enables authors to compare features emphasized by these two modalities, quickly see mismatches, and make necessary revisions. A user study confirms that our tool is both useful and easy to use when authoring charts and captions.

Dashboards are no longer mere static displays of metrics; through functionality such as interaction and storytelling, they have evolved to support analytic and communicative goals like monitoring and reporting. Existing dashboard design guidelines, however, are often unable to account for this expanded scope as they largely focus on best practices for visual design. In contrast, we frame dashboard design as facilitating an analytical conversation: a cooperative, interactive experience where a user may interact with, reason about, or freely query the underlying data. By drawing on established principles of conversational flow and communication, we define the concept of a cooperative dashboard as one that enables a fruitful and productive analytical conversation, and derive a set of 39 dashboard design heuristics to support effective analytical conversations. To assess the utility of this framing, we asked 52 computer science and engineering graduate students to apply our heuristics to critique and design dashboards as part of an ungraded, opt-in homework assignment. Feedback from participants demonstrates that our heuristics surface new reasons dashboards may fail, and encourage a more fluid, supportive, and responsive style of dashboard design. Our approach suggests several compelling directions for future work, including dashboard authoring tools that better anticipate conversational turn-taking, repair, and refinement and extending cooperative principles to other analytical workflows.

Search and information retrieval systems are becoming more expressive in interpreting user queries beyond the traditional weighted bag-of-words model of document retrieval. For example, searching for a flight status or a game score returns a dynamically generated response along with supporting, pre-authored documents contextually relevant to the query. In this paper, we extend this hybrid search paradigm to data repositories that contain curated data sources and visualization content. We introduce a semantic search interface, Olio, that provides a hybrid set of results comprising both auto-generated visualization responses and pre-authored charts to blend analytical question-answering with content discovery search goals. We specifically explore three search scenarios - question-and-answering, exploratory search, and design search over data repositories. The interface also provides faceted search support for users to refine and filter the conventional best-first search results based on parameters such as author name, time, and chart type. A preliminary user evaluation of the system demonstrates that Olio’s interface and the hybrid search paradigm collectively afford greater expressivity in how users discover insights and visualization content in data repositories.

The language for expressing comparisons is often complex and nuanced, making supporting natural language-based visual comparison a non-trivial task. To better understand how people reason about comparisons in natural language, we explore a design space of utterances for comparing data entities. We identified different parameters of comparison utterances that indicate what is being compared (i.e., data variables and attributes) as well as how these parameters are specified (i.e., explicitly or implicitly). We conducted a user study with sixteen data visualization experts and non-experts to investigate how they designed visualizations for comparisons in our design space. Based on the rich set of visualization techniques observed, we extracted key design features from the visualizations and synthesized them into a subset of sixteen representative visualization designs. We then conducted a follow-up study to validate user preferences for the sixteen representative visualizations corresponding to utterances in our design space. Findings from these studies suggest guidelines and future directions for designing natural language interfaces and recommendation tools to better support natural language comparisons in visual analytics.

Despite the ever-growing popularity of dashboards across a wide range of domains, their authoring still remains a tedious and complex process. Current tools offer considerable support for creating individual visualizations but provide limited support for discovering groups of visualizations that can be collectively useful for composing analytic dashboards. To address this problem, we present MEDLEY, a mixed-initiative interface that assists in dashboard composition by recommending dashboard collections (i.e., a logically grouped set of views and filtering widgets) that map to specific analytical intents. Users can specify dashboard intents (namely, measure analysis, change analysis, category analysis, or distribution analysis) explicitly through an input panel in the interface or implicitly by selecting data attributes and views of interest. The system recommends collections based on these analytic intents, and views and widgets can be selected to compose a variety of dashboards. MEDLEY also provides a lightweight direct manipulation interface to configure interactions between views in a dashboard. Based on a study with 13 participants performing both targeted and open-ended tasks, we discuss how MEDLEY's recommendations guide dashboard composition and facilitate different user workflows. Observations from the study identify potential directions for future work, including combining manual view specification with dashboard recommendations and designing natural language interfaces for dashboard authoring.

While visualizations are an effective way to represent insights about information, they rarely stand alone. When designing a visualization, text is often added to provide additional context and guidance for the reader. However, there is little experimental evidence to guide designers as to what is the right amount of text to show within a chart, what its qualitative properties should be, and where it should be placed. Prior work also shows variation in personal preferences for charts versus textual representations. In this paper, we explore several research questions about the relative value of textual components of visualizations. 302 participants ranked univariate line charts containing varying amounts of text, ranging from no text (except for the axes) to a written paragraph with no visuals. Participants also described what information they could take away from line charts containing text with varying semantic content. We find that heavily annotated charts were not penalized. In fact, participants preferred the charts with the largest number of textual annotations over charts with fewer annotations or text alone. We also find effects of semantic content. For instance, the text that describes statistical or relational components of a chart leads to more takeaways referring to statistics or relational comparisons than text describing elemental or encoded components. Finally, we find different effects for the semantic levels based on the placement of the text on the chart; some kinds of information are best placed in the title, while others should be placed closer to the data. We compile these results into four chart design guidelines and discuss future implications for the combination of text and charts.

Chatbots have garnered interest as conversational interfaces for a variety of tasks. While general design guidelines exist for chatbot interfaces, little work explores analytical chatbots that support conversing with data. We explore Gricean Maxims to help inform the basic design of effective conversational interaction. We also draw inspiration from natural language interfaces for data exploration to support ambiguity and intent handling. We ran Wizard of Oz studies with 30 participants to evaluate user expectations for text and voice chatbot design variants. Results identified preferences for intent interpretation and revealed variations in user expectations based on the interface affordances. We subsequently conducted an exploratory analysis of three analytical chatbot systems (text + chart, voice + chart, voice-only) that implement these preferred design variants. Empirical evidence from a second 30-participant study informs implications specific to data-driven conversation such as interpreting intent, data orientation, and establishing trust through appropriate system responses.

Visualization recommendation (VisRec) systems provide users with suggestions for potentially interesting and useful next steps during exploratory data analysis. These recommendations are typically organized into categories based on their analytical actions, i.e., operations employed to transition from the current exploration state to a recommended visualization. However, despite the emergence of a plethora of VisRec systems in recent work, the utility of the categories employed by these systems in analytical workflows has not been systematically investigated. Our article explores the efficacy of recommendation categories by formalizing a taxonomy of common categories and developing a system, Frontier, that implements these categories. Using Frontier, we evaluate workflow strategies adopted by users and how categories influence those strategies. Participants found recommendations that add attributes to enhance the current visualization and recommendations that filter to sub-populations to be comparatively most useful during data exploration. Our findings pave the way for next-generation VisRec systems that are adaptive and personalized via carefully chosen, effective recommendation categories.

Well-designed data visualizations can lead to more powerful and intuitive processing by a viewer. To help a viewer intuitively compare values to quickly generate key takeaways, visualization designers can manipulate how data values are arranged in a chart to afford particular comparisons. Using simple bar charts as a case study, we empirically tested the comparison affordances of four common arrangements: vertically juxtaposed, horizontally juxtaposed, overlaid, and stacked. We asked participants to type out what patterns they perceived in a chart and we coded their takeaways into types of comparisons. In a second study, we asked data visualization design experts to predict which arrangement they would use to afford each type of comparison and found both alignments and mismatches with our findings. These results provide concrete guidelines for how both human designers and automatic chart recommendation systems can make visualizations that help viewers extract the "right" takeaway.

Charts often contain visually prominent features that draw attention to aspects of the data and include text captions that emphasize aspects of the data. Through a crowdsourced study, we explore how readers gather takeaways when considering charts and captions together. We first ask participants to mark visually prominent regions in a set of line charts. We then generate text captions based on the prominent features and ask participants to report their takeaways after observing chart-caption pairs. We find that when both the chart and caption describe a high-prominence feature, readers treat the doubly emphasized high-prominence feature as the takeaway; when the caption describes a low-prominence chart feature, readers rely on the chart and report a higher-prominence feature as the takeaway. We also find that external information that provides context, helps further convey the caption’s message to the reader. We use these findings to provide guidelines for authoring effective chart-caption pairs.

Natural language interfaces (NLIs) have become a prevalent medium for conducting visual data analysis, enabling people with varying levels of analytic experience to ask questions of and interact with their data. While there have been notable improvements with respect to language understanding capabilities in these systems, fundamental user experience and interaction challenges including the lack of analytic guidance (i.e., knowing what aspects of the data to consider) and discoverability of natural language input (i.e., knowing how to phrase input utterances) persist. To address these challenges, we investigate utterance recommendations that contextually provide analytic guidance by suggesting data features (e.g., attributes, values, trends) while implicitly making users aware of the types of phrasings that an NLI supports. We present Snowy, a prototype system that generates and recommends utterances for visual analysis based on a combination of data interestingness metrics and language pragmatics. Through a preliminary user study, we found that utterance recommendations in Snowy support conversational visual analysis by guiding the participants’ analytic workflows and making them aware of the system’s language interpretation capabilities. Based on the feedback and observations from the study, we discuss potential implications and considerations for incorporating recommendations in future NLIs for visual analysis.

Natural language interaction has evolved as a useful modality to help users explore and interact with their data during visual analysis. Little work has been done to explore how autocompletion can help with data discovery while helping users formulate analytical questions. We developed a system called \system as a design probe to better understand the usefulness of autocompletion for visual analysis. We ran three Mechanical Turk studies to evaluate user preferences for various text- and visualization widget-based autocompletion design variants for helping with partial search queries. Our findings indicate that users found data previews to be useful in the suggestions. Widgets were preferred for previewing temporal, geospatial, and numerical data while text autocompletion was preferred for categorical and hierarchical data. We conducted an exploratory analysis of our system implementing this specific subset of preferred autocompletion variants. Our insights regarding the efficacy of these autocompletion suggestions can inform the future design of natural language interfaces supporting visual analysis.

Users often do not use keyboard shortcuts in applications as recalling and choosing the correct shortcut is a higher-order cognitive task. Mouse driven menus, toolbars, and icons are easier for a user to learn because they present hints and make visible what operations are possible, drawing on the power of recognition rather than recall. How can we better support the usage of shortcuts with such menus? Two existing methods are text in the icons, and popups with mouse hover. While the first is space inefficient; the second limits exposure and imposes an interaction cost. We propose a third method, ShoCons, that is spatially more efficient and neither limits user exposure nor imposes an interaction cost. To achieve this, ShoCons use a succinct iconic display of meta keys, limiting textual display to one character. We examine these alternatives in a controlled study, and find that when used with a high-level task, ShoCons enable faster task performance and an immediate increase in the accuracy of shortcut use.

Handling ambiguity and underspecification of users' utterances is challenging, particularly for natural language interfaces that help with visual analytical tasks. Constraints in the underlying analytical platform and the users' expectations of high precision and recall require thoughtful inferencing to help generate useful responses. In this paper, we introduce a system to resolve partial utterances based on syntactic and semantic constraints of the underlying analytical expressions. We extend inferencing based on best practices in information visualization to generate useful visualization responses. We employ heuristics to help constrain the solution space of possible inferences, and apply ranking logic to the interpretations based on relevancy. We evaluate the quality of inferred interpretations based on relevancy and analytical usefulness.

Multimodal interaction offers many potential benefits for data visualization. It can help people stay in the flow of their visual analysis and presentation, with the strengths of one interaction modality offsetting the weaknesses of others. Furthermore, multimodal interaction offers strong promise for leveraging data visualization on diverse display hardware including mobile, AR/VR, and large displays. However, prior research on visualization and interaction techniques has mostly explored a single input modality such as mouse, touch, pen, or more recently, natural language. The unique challenges and opportunities of synergistic multimodal interaction for data visualization have yet to be investigated. This workshop will bring together researchers with expertise in visualization, interaction design, and natural user interfaces. We aim to build a community of researchers focusing on multimodal interaction for data visualization, explore opportunities and challenges in our research, and establish an agenda for multimodal interaction research specifically for data visualization.

Interactive visual data analysis is most productive when users can focus on answering the questions they have about their data, rather than focusing on how to operate the interface to the analysis tool. One viable approach to engaging users in interactive conversations with their data is a natural language interface to visualizations. These interfaces have the potential to be both more expressive and more accessible than other interaction paradigms. We explore how principles from language pragmatics can be applied to the flow of visual analytical conversations, using natural language as an input modality. We evaluate the effectiveness of pragmatics support in our system Evizeon, and present design considerations for conversation interfaces to visual analytics tools.

Natural language interfaces for visualizations have emerged as a promising new way of interacting with data and performing analytics. Many of these systems have fundamental limitations. Most return minimally interactive visualizations in response to queries and often require experts to perform modeling for a set of predicted user queries before the systems are effective. Eviza provides a natural language interface for an interactive query dialog with an existing visualization rather than starting from a blank sheet and asking closed-ended questions that return a single text answer or static visualization. The system employs a probabilistic grammar based approach with predefined rules that are dynamically updated based on the data from the visualization, as opposed to computationally intensive deep learning or knowledge based approaches.

When data categories have strong color associations, it is useful to use these semantically meaningful concept-color associations in data visualizations. In this paper, we explore how linguistic information about the terms defining the data can be used to generate semantically meaningful colors. To do this effectively, we need first to establish that a term has a strong semantic color association, then discover which color or colors express it. Using co-occurrence measures of color name frequencies from Google n-grams, we define a measure for colorability that describes how strongly associated a given term is to any of a set of basic color terms. We then show how this colorability score can be used with additional semantic analysis to rank and retrieve a representative color from Google Images. Alternatively, we use symbolic relationships defined by WordNet to select identity colors for categories such as countries or brands. To create visually distinct color palettes, we use k-means clustering to create visually distinct sets, iteratively reassigning terms with multiple basic color associations as needed. This can be additionally constrained to use colors only in a predefined palette.

Although mobile devices are generating a rapidly increasing proportion of search queries, search interfaces have not changed significantly to accommodate mobile constraints. In particular, imprecise search exists in the no-man's land between specific fact-finding and general browsing, and can be especially challenging on mobile devices, when user input is difficult and environmental distractions make remembering related information difficult. We examined the prevalence of these mobile search use cases in a two-week diary study, finding that imprecise and general search accounted for the large majority of difficulty with search. Hypothesizing that the ability to view a link neighborhood around the search result could be quite helpful in these cases, we designed GraphTiles, a visual interface for mobile search that exploits the structured entity relationships present in a significant portion of online datasets (e.g. IMDb [5] and LinkedIn [6]). In an experimental evaluation, users performed imprecise searches more quickly with GraphTiles than with a standard mobile site.

Authors use icon encodings to indicate the semantics of categorical information in visualizations. The default icon libraries found in visualization tools often do not match the semantics of the data. Users often manually search for or create icons that are more semantically meaningful. This process can hinder the flow of visual analysis, especially when the amount of data is large, leading to a suboptimal user experience. We propose a technique for automatically generating semantically relevant icon encodings for categorical dimensions of data points. The algorithm employs natural language processing in order to find relevant imagery from the Internet. We evaluate our approach on Mechanical Turk by generating large libraries of icons using Tableau Public workbooks that represent real analytical effort by people out in the world. Our results show that the automatic algorithm does nearly as well as the manually created icons, and particularly has higher user satisfaction for larger cardinalities of data.

Bar charts are one of the most common visualization types. In a classic graphical perception paper, Cleveland & McGill studied how different bar chart designs impact the accuracy with which viewers can complete simple perceptual tasks. They found that people perform substantially worse on stacked bar charts than on aligned bar charts, and that comparisons between adjacent bars are more accurate than between widely separated bars. However, the study did not explore why these differences occur. In this paper, we describe a series of follow-up experiments to further explore and explain their results. While our results generally confirm Cleveland & McGill's ranking of various bar chart configurations, we provide additional insight into the bar chart reading task and the sources of participants' errors. We use our results to propose new hypotheses on the perception of bar charts.

Automakers are increasingly providing connectivity enhancements for vehicles to download navigational data, as well as to upload sensor information to the cloud. Generally, while more data may be better, for the driver on-the-go, information needs to be displayed in a manner that can be comprehended rather quickly. One of the major problems with visualizing route maps is that the amount of information visualized is always the same regardless of the fact that an individual may be more familiar with the region or whether an individual is driving at varying speeds. Research has shown that complex visualizations with visual clutter can cause cognitive overload that adversely affects the performance of a user. Additionally, the attention and interaction abilities of a driver are significantly compromised in a vehicular environment. We propose DriveSense, a context-sensitive visualization system that automatically varies the GPS updates and the corresponding visualization being displayed to the user based on the speed of the vehicle as well as the familiarity of the region that the user is driving in. Based on a user evaluation, we found that subjects preferred using the automatic visualizations of route maps generated by DriveSense than the visual representations shown by a standard GPS. We also computed visual clutter for our visualizations at varying speeds and found that the clutter was significantly less for the routes displayed by DriveSense for faster speeds as compared to slower speeds.

People use mobile devices to search, locate and discover local information around them. Mobile local search is frequently a social activity. This paper presents the results of a survey and an exploratory user study of collaborative mobile local search. The survey results show that people frequently search with others and that these searches often involve the use of more than one mobile device. We prototyped a collaborative mobile search app, which we used as a tool to investigate users' collaborative mobile search behavior. Our study results provide insights into how users collaborate while performing search. We also provide design considerations to inform future mobile local search technologies.

Users often engage in tasks that span multiple personal devices. Although many current solutions exist to provide ubiquitous access to one's data, users continue to struggle with cross-device tasks. These solutions often require them to plan ahead for their information needs. In this paper, we present Myngle, a device-agnostic system that lets users quickly find the information they are looking for from previously visited web pages without having to plan ahead. Myngle provides a unified web history from multiple personal devices, and allows users to filter their history based on high-level categories influenced by common mobile information need categories (e.g., address, phone number). We evaluated Myngle with 32 users and found that our category-based method of filtering eases the burden of continuing cross-device tasks.

Typical web navigation techniques tend to support undirected web browsing, a depth-first search of information pages. This search strategy often results in the unintentional behavior of 'web surfing', where a user starts in search of information, but is sidetracked by tangential links. A mobile user in particular, would prefer to extract the desired information quickly and with minimal mental effort. In this paper, we introduce 'SemantiLynx' to visually augment hyperlinks on web pages for better supporting the task of directed searches on small-screen ubiquitous platforms. Our algorithm comprises four parts: establishing the context of information related to a hyperlink, retrieving relevant imagery based on this context, applying image simplification, and finally compositing a visual icon for the given hyperlink. We evaluated our system by conducting user studies for directed web search tasks and comparing the results to using textual snippets and webpage thumbnails.

The popularity of mobile interfaces and application development is increasing along with the rapid expansion of the mobile electronics market and its migration from text-based applications to various multimedia applications. Real-time graphics and web applications are becoming one of the most attractive applications in mobile terminals due to their benefits for enterprise, gaming, and social media. This hands-on course will cover a comprehensive set of topics for developing mobile visual interfaces, including an overview of the mobile market, a comparison of mobile and desktop applications, and a survey of mobile development environments. We will also undertake a detailed discussion of UI development for mobiles, and graphics development for mobiles. During the course, various smartphones will be loaned to attendees for trying out several in-class exercises.

People interact with numerous personal devices on a daily basis. Sharing content among these devices is often done depending on the device capabilities and context of use; following turn-by-turn directions is more appropriate when mobile. Although several solutions exist to share content among one's devices, these solutions rely on the user planning ahead for the data he may need on another device. In this paper, we describe a system that addresses the unplanned activities, by automatically extracting addresses and points of interest that users view in their web browser and making those readily available through an in-car interface.

In many developing countries such as India and China, low educational levels often hinder economic empowerment. In this paper, we argue that mobile learning games can play an important role in the Chinese literacy acquisition process. We report on the unique challenges in the learning Chinese language, especially its logographic writing system. Based on an analysis of 25 traditional Chinese games currently played by children in China, we present the design and implementation of two culturally inspired mobile group learning games, Multimedia Word and Drumming Strokes. These two mobile games are designed to match Chinese children's understanding of everyday games. An informal evaluation reveals that these two games have the potential to enhance the intuitiveness and engagement of traditional games, and children may improve their knowledge of Chinese characters through group learning activities such as controversy, judgments and self-correction during the game play.

All in-text\treferences\tunderlined\tin\tblue\tare\tlinked\tto\tpublications\ton\tResearchGate, letting you\taccess\tand\tread\tthem\timmediately.

Typical web navigation techniques tend to support undirected web browsing, a depth-first search of information pages. This search strategy often results in the unintentional behavior of 'web surfing', where a user starts in search of information, but is sidetracked by tangential links. A mobile user in particular, would prefer to extract the desired information quickly and with minimal mental effort. In this paper, we introduce 'SemantiLynx' to visually augment hyperlinks on web pages for better supporting the task of directed searches on small-screen ubiquitous platforms. Our algorithm comprises four parts: establishing the context of information related to a hyperlink, retrieving relevant imagery based on this context, applying image simplification, and finally compositing a visual icon for the given hyperlink.

Increasingly, smartphones are being used to access all manner of information: email messages, Facebook status updates, tweets, RSS feeds, photographs and more. Approaches to dealing with this multi-faceted information stream developed on the desktop, such as switching between multiple applications or multiple browser windows, are unwieldy and scale poorly for mobile devices. In this paper, we propose the combination of the universal inbox and a system called 'Lenses' for extracting information of interest as part of a solution to this problem. These mechanisms allow the user to easily specify ways to sort, filter and manage their universal inbox in an intuitive way. We culminate with a discussion of implications for mobile phone interface design.

SMS messages provide an easy and simple method to com-municate with others. These short messages are useful, but can sometimes feel restricted due to the limitations of textual communication. The ability to express subtle nuances and contexts around the message could help add enjoyment and amplify the emotions being expressed by the mobile user. We present SensorComix, a new way of creating comics us-ing SMS messages combined with gestures on mobile phones. Comics are automatically generated from users ’ SMS mes-sages, and augmented with visual icons based on the per-formed gestures. We demonstrate that gestures mapped to comics can help influence the expressiveness of messages sent by mobile users.

The convergence of the Internet and ubiquitous technologies offers an unprecedented level of convenience for information collection, accessing, and sharing through mobile devices and Web services. Mobile devices have become increasingly context aware and connected to the Internet, which lowers the barriers for capturing and sharing contextual information (e.g. location and time). Web applications are increasingly taking advantage of the so-called 'long tail', offering specialized services to a small number of users and allowing them to annotate, produce and consume relevant information. This paper presents travel scrapbooks, an online service that automatically stitches information together from various online sources based on geotagged photos from social networks, and visually presents the information to the user in the form of a scrapbook metaphor.

We present a novel data-driven algorithm that leverages online image repositories such as Flickr for automatically generating tourist maps. Our hypothesis is that, given a large enough dataset of images with geo-based metadata, clusters of matching images from that dataset tend to provide reliable cues as to what the popular tourist spots may be. Our algorithm takes the geographical area of interest as input and retrieves geotagged photos from online photo collections. By clustering the photos based on their locations and identifying the popular tags for each cluster, our algorithm generates a set of points of interest (POIs) for the area. After retrieving additional photos based on these discovered POI tags, we use image matching to find the most representative landmark view for each POI. Finally, we remove clutter from the representative image and apply tooning to generate a map icon for each landmark.

Recently, many exciting media search applications have been introduced to take advantage of smart phones' audiovisual capture capabilities and their being always on and connected. These applications address a real pain point for most mobile users and allow them to search with minimal text entry, if any. Is the mobile platform an ideal fit for media search? Are audio and visual signal processing technologies sufficiently accurate to support most mobile search applications? What are the killer applications of mobile media search? Earlier in 2009 at ICASSP, a panel on this topic stirred up great interest and enthusiasm while leaving many questions untouched due to the limited time.

We present a new technique called 'Tilt Menu' for better extending selection capabilities of pen-based interfaces. The Tilt Menu is implemented by using 3D orientation information of pen devices while performing selection tasks. The Tilt Menu has the potential to aid traditional one-handed techniques as it simultaneously generates the secondary input (e.g., a command or parameter selection) while drawing/interacting with a pen tip without having to use the second hand or another device. We conduct two experiments to explore the performance of the Tilt Menu. In the first experiment, we analyze the effect of parameters of the Tilt Menu, such as the menu size and orientation of the item, on its usability. Results of the first experiment suggest some design guidelines for the Tilt Menu. In the second experiment, the Tilt Menu is compared to two types of techniques while performing connect-the-dot tasks using freeform drawing mechanism. Results of the second experiment show that the Tilt Menu perform better in comparison to the Tool Palette, and is as good as the Toolglass.

In order to improve stimulus-response compatibility of touchpad in pen-based user interface, we present the tilt cursor, i.e. a cursor dynamically reshapes itself to providing the 3D orientation cue of pen. We also present two experiments that evaluate the tilt cursor's performance in circular menu selection and specific marking menu selection tasks. Results show that in a specific marking menu selection task, the tilt cursor significantly outperforms the shape-fixed arrow cursor and the live cursor [4]. In addition, results show that by using the tilt cursor, the response latencies for adjusting drawing directions are smaller than that by using the other two kinds of cursors.

'Rich media' is a term that implies the integration of all of the advances we have made in the mobile space delivering music, speech, text, graphics and video. This is true, but it is more than the sum of its parts. Rich media is the ability to deliver these modalities, to interact with these modalities, and to do it in a way that allows for the construction, delivery and use of compelling mobile services in an effective and economic manner. In this paper, we introduce a system called mobile open rich-media environment ('MORE') that helps realize such mobile rich media services, combining various technologies of W3C, OMA, 3GPP and IETF standards. The different components of the system include formatting, packaging, transporting, rendering and interacting with rich media files and streams

In this paper, we present an approach for facilitating user interaction on mobile devices, focusing on camera-enabled mobile phones. A user interacts with an application by moving their device. An on-board camera is used to capture incoming video and the scrolling direction and magnitude are estimated using a computer vision-based algorithm. The direction is used as the scroll direction in the application, and the magnitude is used to set the zoom level. The camera is treated as a pointing device and zoom level control in applications. Our approach generates mouse events, so any application that is mouse-driven can make use of this technique. The user is free to browse through large data sets on a limited size display with one hand, ideal for the mobile domain.

We present a non-photorealistic algorithm for retargeting large images to small size displays, particularly on mobile devices. This method adapts large images so that important objects in the image are still recognizable when displayed at a lower target resolution. Existing image manipulation techniques such as cropping works well for images containing a single important object, and down-sampling works well for images containing low frequency information. However, when these techniques are automatically applied to images with multiple objects, the image quality degrades and important information may be lost. Our algorithm addresses the case of multiple important objects in an image. The retargeting algorithm segments an image into regions, identifies important regions, removes them, fills the resulting gaps, resizes the remaining image, and re-inserts the important regions. Our approach lies in constructing a topologically constrained epitome of an image based on a visual attention model that is both comprehensible and size varying, making the method suitable for display-critical applications.

We present a method that preserves the recognizability of key object interactions in a vector animation. The method allows an artist to author an animation once, and then output it to any display device. We specifically target mobile devices with small screen sizes. In order to adapt an animation, the author specifies an importance value for objects in the animation. The algorithm then identifies and categorizes the vector graphics objects that comprise the animation, leveraging the implicit relationship between extensible Markup Language (XML) and scalable vector graphics (SVG). Based on importance, the animation can then be automatically retargeted for any display using artistically motivated resizing and grouping algorithms that budget size and spatial detail for each object.

Semanticons can enhance the representation of files by offering symbols that are both meaningful and easily distinguishable. The semantics of a file is estimated by parsing its name, location, and content to generate a ‘context’, which is used to query an image database. The resulting images are simplified by segmenting them, computing an importance value for each segmented region, and removing unimportant regions. The abstract look-and-feel of icons is achieved using non-photorealistic techniques for image stylization. We increase the effectiveness of the semanticons by compositing them with traditional and familiar interface icons. Two psychophysical studies using semanticons as stimuli demonstrate that semanticons decrease the time necessary to locate a file in a visual search task and enhance performance in a memory task.

This article describes the implementation of a system that 'imagines' while a movie is being played by finding associations in the movie's content and presenting them to the viewer. This related information, in the form of images and movie clips, helps enhance the viewer's experience in a new immerse environment.