Existing frameworks in visualization and HCI emphasize iteration, data grounding, and stakeholder needs; however, they have not fully explored how evaluation might persist across phases, adapt to compressed timelines, and aid stakeholder engagement and elicitation. Building on prior frameworks, we introduce Evaluation-first design EvalOps that centers evaluation as a material component in the design process, emphasizing tighter feedback loops, co-evaluation with stakeholders, malleable forms of evaluation, and goals-to-metrics grounding. We illustrate how EvalOps shapes design outcomes through two case studies of data-visualization and LLM-enabled reasoning tools, demonstrating how evaluation-driven design facilitates alignment and trust, uncovers opportunities earlier, and supports cohesiveness under rapidly changing constraints. We contrast EvalOps with current visualization design methodologies and discuss opportunities for expanding evaluation-centered framings to other active areas of design research.

Online user studies of visualizations, visual encodings, and interaction techniques are ubiquitous in visualization research. Yet, designing, conducting, and analyzing studies effectively is still a major burden. Although various packages support such user studies, most solutions address only facets of the experiment life cycle, make reproducibility difficult, or do not cater to nuanced study designs or interactions. We introduce reVISit 2, a software framework that supports visualization researchers at all stages of designing and conducting browser-based user studies. ReVISit supports researchers in the design, debug & pilot, data collection, analysis, and dissemination experiment phases by providing both technical affordances (such as replay of participant interactions) and sociotechnical aids (such as a mindfully maintained community of support). It is a proven system that can be (and has been) used in publication-quality studies-which we demonstrate through a series of experimental replications. We reflect on the design of the system via interviews and an analysis of its technical dimensions. Through this work, we seek to elevate the ease with which studies are conducted, improve the reproducibility of studies within our community, and support the construction of advanced interactive studies.

Abstract People craft choropleth maps to monitor, analyze, and understand spatially distributed data. Recent visualization work has addressed several known biases in choropleth maps by developing new model‐ and metrics‐ based approaches (e.g. Bayesian surprise). However, effective use of these techniques requires extensive parameter setting and tuning, making them difficult or impossible for users without substantial technical skills. In this paper we describe SurpriseExplora, which addresses this gap through direct manipulation techniques for re‐targeting a Bayesian surprise model's scope and parameters. We present three use cases to illustrate the capabilities of SurpriseExplora, showing for example how models calculated at a national level can obscure key findings that can be revealed through interaction sequences common to map visualizations (e.g. zooming), and how augmenting funnel‐plot visualizations with interactions that adjust underlying models can account for outliers or skews in spatial datasets. We evaluate SurpriseExplora through an expert review with visualization researchers and practitioners. We conclude by discussing how SurpriseExplora uncovers new opportunities for sense‐making within the broader ecosystem of map visualizations, as well as potential empirical studies with non‐expert populations. Code and demo video available at https://osf.io/7m89w/

Abstract Decisions involving algorithmic rankings affect our lives in many ways, from product recommendations, receiving scholarships, to securing jobs. While tools have been developed for interactively constructing fair consensus rankings from a handful of rankings, addressing the more complex real‐world scenario— where diverse opinions are represented by a larger collection of rankings— remains a challenge. In this paper, we address these challenges by reformulating the exploration of rankings as a dimension reduction problem in a system called FairSpace. FairSpace provides new views, including Fair Divergence View and Cluster Views, by juxtaposing fairness metrics of different local and alternative global consensus rankings to aid ranking analysis tasks. We illustrate the effectiveness of FairSpace through a series of use cases, demonstrating via interactive workflows that users are empowered to create local consensuses by grouping rankings similar in their fairness or utility properties, followed by hierarchically aggregating local consensuses into a global consensus through direct manipulation. We discuss how FairSpace opens the possibility for advances in dimension reduction visualization to benefit the research area of supporting fair decision‐making in ranking based decision‐making contexts. Code, datasets and demo video available at: osf.io/d7cwk