We present EvalAssist, a framework that simplifies the LLM- as-a-judge workflow. The system provides an online criteria development environment, where users can interactively build, test, and share custom evaluation criteria in a structured and portable format. A library of LLM based evaluators is made available that incorporates various algorithmic innovations such as token-probability based judgement, positional bias checking, and certainty estimation that help to engender trust in the evaluation process. We have computed extensive benchmarks and also deployed the system internally in our organization with several hundreds of users.

Agentic systems interleave large language model (LLM) reasoning, tool usage, and tool observations over multiple iterations to tackle complex tasks. The raw data from an agent's problem-solving process (the agents' trajectory) is not an ideal format for human analysis and oversight. There is a need for tooling that converts this primary data into an easily navigable and understandable visual format for better human feedback. To address this opportunity, we developed the Agent Trajectory Explorer, a tool designed to help AI developers and researchers visualize, annotate, and demonstrate agent behavior.

Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Kush R. Varshney, Eitan Farchi, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, Miao Liu, Prasanna Sattigeri. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

User flow diagram for EvalAssist in the direct assessment evaluation, illustrating criteria definition, test data input, annotation, AI evaluator selection, result review, iterative adjustments, and criteria export for dataset-wide evaluation via SDK.

Subject matter experts play an important role in customer support communities by responding to user queries. Some communities have adopted chatbots in addition to SMEs to address commonly asked questions. Yet, SME-bot interactions, particularly teaching paradigms between SMEs and bots remain understudied. We investigate human-AI machine teaching interactions in a scenario-based study (n=48). Participants selected their preferred teaching method in simulated community interactions with a consumer, an SME, and an AI Bot. We investigated preferences across three interactions: demonstration (Showing), preference elicitation (Sorting), and labeling (Categorization). Participants preferred the Showing interaction, followed by Sorting and Categorizing. Participants changed their preferences from lower-effort interactions when considering downstream outcomes. Users considered the community’s perception of interactions between the bot and the SME, specifically transparency of learning outcome, orientation of the feedback, querying the bot and disruptiveness of the interaction. We discuss implications for our findings for teaching interactions in human-AI communities.

We demonstrate an AI assisted data labeling system which applies unsupervised and semi-supervised machine learning to facilitate accurate and efficient labeling of large data sets. Our system (1) applies representative data sampling and active learning in order to seed and maintain a semi-supervised learner that assists the human labeler (2) provides visual labeling assistance and optimizes labeling mechanics using predicted labels (3) seamlessly updates and learns from ongoing human labeling activity (4) captures and presents metrics that indicate the quality of labeling assistance, and (5) provides an interactive auto labeling interface to group, review and apply predicted labels in a scalable manner.

Human data labeling with multiple labelers and the resulting conflict resolution remains the norm for many enterprise machine learning pipelines. Conflict resolution can be a time-intensive and costly process. Our goal was to study how human-AI collaboration can improve conflict resolution, by enabling users to automate groups of conflict resolution tasks. However, little is known about whether and how people will rely on automation during conflict resolution. Currently, automation commonly uses labelers' majority vote labels for conflict resolution, as the top chosen label by most labelers is often correct. We envisioned a system where an AI would assist in finding cases where the labeler majority vote was wrong and where automation is supported for batches or groups of conflicts. In order to understand whether humans could use labeler and AI information effectively, we investigated how and when users rely on labeler and AI information and on automated group conflict resolution. We ran a study with 144 Mechanical Turk workers. We found that automation increased users' accuracy/time, use of automated conflict resolution was relatively similar regardless of whether the automation was based on labeler or AI selected labels, and providing labeler and AI selected labels may reduce inappropriate reliance on automation.

Ground-truth labeling is an important activity in machine learning. Many studies have examined how crowdworkers apply labels to records in machine learning datasets. However, there have been few studies that have examined the work of domain experts when their knowledge and expertise are needed to apply labels.

Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides which of several potential labels to apply to each example. Prior work has shown that providing AI assistance can improve the accuracy of binary decision tasks. However, the role of AI assistance in more complex data-labeling scenarios with a larger set of labels has not yet been explored. We designed an AI labeling assistant that uses a semi-supervised learning algorithm to predict the most probable labels for each example. We leverage these predictions to provide assistance in two ways: (i) providing a label recommendation and (ii) reducing the labeler’s decision space by focusing their attention on only the most probable labels. We conducted a user study (n=54) to evaluate an AI-assisted interface for data labeling in this context. Our results highlight that the AI assistance improves both labeler accuracy and speed, especially when the labeler finds the correct label in the reduced label space. We discuss findings related to the presentation of AI assistance and design implications for intelligent labeling interfaces.

Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.

Sketching facilitates design in the exploration of ideas about concrete objects and abstractions. In fact, throughout the software engineering process when grappling with new ideas, people reach for a pen and start sketching. While pen and paper work well, digital media can provide additional features to benefit the sketcher. Digital support will only be successful, however, if it does not detract from the core sketching experience. Based on research that defines characteristics of sketches and sketching, this paper offers three preliminary tool examples. Each example is intended to enable sketching while maintaining its characteristic experience.

Tables are an important tool used by business analysts engaged in early requirements activities (in fact it is safe to say that tables appeal to many other types of user, in a variety of activities and domains). Business analysts typically use the tables provided by office tools. These tables offer great flexibility, but no underlying model, and hence no consistency management, multiple views or other advantages familiar to the users of modeling tools. Modeling tools, however, are usually too rigid for business analysts. In this paper we present a flexible modeling approach to tables, which combines the advantages of both office and modeling tools. Freeform information can co-exist with information managed by an underlying model, and an incremental formalization approach allows each item of information to transition fluidly between freeform and managed. As the model evolves, it is used to guide the user in the process of formalizing any remaining freeform information. The model therefore helps users without restricting them. Early feedback is described, and the approach is analyzed briefly in terms of cognitive dimensions.

A serious tool gap exists at the start of the software lifecy-cle, before requirements formulation. Pre-requirements analysts gather information, organize it to gain insight, en-vision possible futures, and present insights and recom-mendations to stakeholders. They typically use office tools, which give great freedom, but no help with consistency management, change propagation, or information migration to downstream tools. Despite these downsides, office tools are still favored over modeling tools, which are constrain-ing and difficult to use. We introduce the notion of flexible modeling tools, which blend the advantages of office and modeling tools. We propose a conceptual architecture for such tools, and outline research challenges to be met in realizing them. We briefly describe the Business Insight Toolkit, a prototype tool embodying this architecture.

Diagrams play a key role in the information systems domain. However to be meaningful, the diagrams are understood by interpreting visual cues in specific, conventionalized ways, termed conceptual models. One of the major pain points of conceptual models, specified as visual languages, is the inability to capture these visual languages effectively in conventional modeling tools. Instead, conceptual models are drawn using drawing tools and sometimes even by hand. We propose an automatic procedure to derive the syntactic building blocks of graph-based conceptual models. This high-level specification of the visual language can then serve as input for the automatic construction of syntax-aware diagram editors. Our aim is to achieve minimum effort on the part of the users when they eventually work with the graphical editor to produce a new diagram using the proposed syntax.

A common form of live technical presentation is that given by programmers for a programming audience during conferences, demonstrations, code reviews, and tutorials. Such presentations require manual switching between general presentation software and the integrated development environment (IDE), as well as reconfiguration of the IDE's UI to be readable by an audience. In this paper, we present a novel system that allows programmers to easily combine traditional slideware with seamless transitions to user-specified regions of the IDE along with special effects for live demonstration.

Fluid documents enable the interactive incorporation of related material into a primary document. Visual cues within primary material indicate that related content is available and user interaction reveals related content in context. Fluid documents reduce the need for explicit navigation, maintain context when considering related material and support synthesization of disjoint yet related information. Source code documents are an excellent medium for the application of fluid document technology. The definition of software in the form of text based documents manifests a highly linked and fragmented information space requiring considerable navigation between elements during code examination and comprehension. In this paper we present fluid source code views, the application of fluid document technology to source code documents