Abstract Decades of advocacy for reproducibility and replication have advanced open, transparent practices in the sciences. However, traditional notions of reproducibility fit poorly with design‐oriented visualization research, where insights emerge through subjective, situated, and iterative work. So how can we ensure rigor and transparency in processes that are inherently unreproducible? To introduce transparency in design‐oriented research, we propose to focus on traceability: surfacing the origin and development of research contributions based on rich sets of artifacts documenting the design process. We investigated traceability through a collaborative autoethnographic reflection that builds on several years of work exploring ways to make design‐oriented research transparent. This exploration includes an experiment to build a tool to support traceability, which we called tRRRacer. The tRRRacer tool provided a testbed for us to operationalize the three tenets of a traceable process: (1) Record abundant, annotated artifacts representative of research activities; (2) Report curated research threads that articulate rationale and evolution of the process, allowing others to (3) Read via interfaces that help retrace claims and assess plausibility. Reflecting on our experiences, we contribute a theorization of traceability and reflections on how we might support it.

Recent studies have reported a shortcoming of visual exploratory data analysis (EDA) that can lead analysts to report spurious findings. These findings have prompted the advocacy of incorporating belief elicitation within the data analysis process. However, the results from these studies primarily drew from laboratory experiments, which can differ from real-world analysis contexts. In this article, we present outcomes from a longitudinal study with students enrolled in a university-level visual analytics course tackling the VAST Challenge. The students formed teams that were randomly assigned to the belief elicitation and non-belief elicitation conditions. Our study results indicate teams that underwent belief elicitation adopted an intentional approach, while teams in the non-belief elicitation condition reported greater diversity in findings, aligning with prior research. Surprisingly, teams from both conditions achieved equal success in solving the VAST Challenge, suggesting that analysts can incorporate belief elicitation strategically for different goals. We provide guidelines for incorporating belief elicitation into data analysis and teaching material for educators to include belief elicitation in visual analytics courses.

Dimensionality reduction techniques are widely used for visualizing high-dimensional data. However, support for interpreting patterns of dimension reduction results in the context of the original data space is often insufficient. Consequently, users may struggle to extract insights from the projections. In this paper, we introduce DimBridge, a visual analytics tool that allows users to interact with visual patterns in a projection and retrieve corresponding data patterns. DimBridge supports several interactions, allowing users to perform various analyses, from contrasting multiple clusters to explaining complex latent structures. Leveraging first-order predicate logic, DimBridge identifies subspaces in the original dimensions relevant to a queried pattern and provides an interface for users to visualize and interact with them. We demonstrate how DimBridge can help users overcome the challenges associated with interpreting visual patterns in projections.

Automated Machine Learning (AutoML) technology can lower barriers in data work yet still requires human intervention to be functional. However, the complex and collaborative process resulting from humans and machines trading off work makes it difficult to trace what was done, by whom (or what), and when. In this research, we construct a taxonomy of data work artifacts that captures AutoML and human processes. We present a rigorous methodology for its creation and discuss its transferability to the visual design process. We operationalize the taxonomy through the development of AutoML Trace a visual interactive sketch showing both the context and temporality of human-ML/AI collaboration in data work. Finally, we demonstrate the utility of our approach via a usage scenario with an enterprise software development team. Collectively, our research process and findings explore challenges and fruitful avenues for developing data visualization tools that interrogate the sociotechnical relationships in automated data work.

This discussion paper presents learning engineering as an emerging new domain of engineering. It asks what role IEEE should play in establishing this branch of engineering in support of its mission to foster technological innovation and excellence for the benefit of humanity.

Recent reports indicate increased organizational appetite and spend in the energy industry in both the areas of operational risk management training and enablement and in extended reality hardware and software, as part of larger automation and digital transformation initiatives. Furthermore, recent advances in immersive technology, along with more dispersed, asynchronous working conditions due to COVID, have resulted in scalable, immersive simulations that more and more closely resemble real world environments. While recent standards have defined JSON syntax appropriate for tracking and measuring human behavior data in generic learning environments (IEEE P9274.1) and in a manner that more closely approximates human behavior in the workplace, as typically tracked in operational risk management systems, no risk-based ontology has yet been defined that more closely crosswalks and correlates data from simulated environment systems to those in operational environments. Thus, the true efficacy of extended reality-based risk mitigation training cannot be fully measured. In this effort, a risk-based ontology and matrix was constructed in accordance with the xAPI standard syntax and allowable extensions and was utilized to transform a subset of historical data from simulated operational risk-based scenarios from the energy industry. Transformed data from this initial subset closely approximated operational risk reporting data and provided insights into human behavior data in simulated environments that can be easily compared and correlated to existing operational excellence and risk mitigation KPIs. Implications for mapping of additional advanced data from simulated environments in larger, more complex datasets, such as eye tracking and biometrics, were also considered and explored.

Design study is an established approach of conducting problem-driven visualization research. The academic visualization community has produced a large body of work for reporting on design studies, informed by a handful of theoretical frameworks, and applied to a broad range of application areas. The result is an abundance of reported insights into visualization design, with an emphasis on novel visualization techniques and systems as the primary contribution of these studies. In recent work we proposed a new, interpretivist perspective on design study and six companion criteria for rigor that highlight the opportunities for researchers to contribute knowledge that extends beyond visualization idioms and software. In this work we conducted a year-long collaboration with evolutionary biologists to develop an interactive tool for visual exploration of multivariate datasets and phylogenetic trees. During this design study we experimented with methods to support three of the rigor criteria: ABUNDANT, REFLEXIVE, and TRANSPARENT. As a result we contribute two novel visualization techniques for the analysis of multivariate phylogenetic datasets, three methodological recommendations for conducting design studies drawn from reflections over our process of experimentation, and two writing devices for reporting interpretivist design study. We offer this work as an example for implementing the rigor criteria to produce a diverse range of knowledge contributions.

BACKGROUND: In phylogenetics, we often seek to reconcile gene trees with species trees within the framework of an evolutionary model. While the most popular models for eukaryotic species allow for only gene duplication and gene loss or only multispecies coalescence, recent work has combined these phenomena through a reconciliation structure, the labeled coalescent tree (LCT), that simultaneously describes the duplication-loss and coalescent history of a gene family. However, the LCT makes the simplifying assumption that only one individual is sampled per species whereas, with advances in gene sequencing, we now have access to multiple samples per species. RESULTS: We demonstrate that with these additional samples, there exist gene tree topologies that are impossible to reconcile with any species tree. In particular, the multiple samples enforce new constraints on the placement of duplications within a valid reconciliation. To model these constraints, we extend the LCT to a new structure, the partially labeled coalescent tree (PLCT) and demonstrate how to use the PLCT to evaluate the feasibility of a gene tree topology. We apply our algorithm to two clades of apes and flies to characterize possible sources of infeasibility. CONCLUSION: Going forward, we believe that this model represents a first step towards understanding reconciliations in duplication-loss-coalescence models with multiple samples per species.