Accessibility support in prototyping tools remains fragmented, despite increasing regulatory demands and well-established guidelines.Current accessibility support in prototyping tools primarily addresses visual issues such as color contrast or touch target size, leaving large gaps for issues requiring semantic understanding.

Figure 1: GUI prototype (1) and three views (2-4) of our assistant for GUI prototype designers integrated as a plug-in into a prototyping tool.Our assistant displays user stories (2) imported from collaboration tools (e.g., JIRA) for prototype designers to reference while working.It detects whether a user story is implemented (3, 4), identifies relevant GUI components (3), and generates GUI components for user stories (4). Figure uses Google Material 3 Design Kit [24] components under CC BY 4.0.

Graphical User Interface (GUI) prototyping is a fundamental component in the development of modern interactive systems, which are now ubiquitous across diverse application domains. GUI prototypes play a critical role in requirements elicitation by enabling stakeholders to visualize, assess, and refine system concepts collaboratively. Moreover, prototypes serve as effective tools for early testing, iterative evaluation, and validation of design ideas with both end users and development teams. Despite these advantages, the process of constructing GUI prototypes remains resource-intensive and time-consuming, frequently demanding substantial effort and expertise. Recent research has sought to alleviate this burden through natural language (NL)-based GUI retrieval approaches, which typically rely on embedding-based retrieval or tailored ranking models for specific GUI repositories. However, these methods often suffer from limited retrieval performance and struggle to generalize across arbitrary GUI datasets. In this work, we present GUI-ReRank, a novel framework that integrates rapid embedding-based constrained retrieval models with highly effective multi-modal (M)LLM-based reranking techniques. GUI-ReRank further introduces a fully customizable GUI repository annotation and embedding pipeline, enabling users to effortlessly make their own GUI repositories searchable, which allows for rapid discovery of relevant GUIs for inspiration or seamless integration into customized LLM-based retrieval-augmented generation (RAG) workflows. We evaluated our approach on an established NL-based GUI retrieval benchmark, demonstrating that GUI-ReRank significantly outperforms state-of-the-art (SOTA) tailored Learning-to-Rank (LTR) models in both retrieval accuracy and generalizability. Additionally, we conducted a comprehensive cost and efficiency analysis of employing MLLMs for reranking, providing valuable insights regarding the trade-offs between retrieval effectiveness and computational resources. Video presentation of GUI-ReRank available at: https://youtu.be/7x9UCh82ug

In recent years, the availability of easy-to-use prototyping tools has empowered designers to create GUI designs. In parallel, a broad spectrum of usability testing tools have been proposed to support the collection of quantitative test data from crowd-testers. However, test specification and results are typically disconnected from created GUI designs and, therefore, difficult to translate into improvements. In this paper, we present TESY, a prototyping assistant integrating usability test specification and resulting test data. We implement TESY as a plug-in in the prototyping tool Figma. In a controlled lab study, we compare how 34 untrained designers create prototypes, specify usability tests, and improve the prototypes using the collected test data with and without TESY. We contribute by demonstrating how TESY empowers untrained designers to create enhanced GUI designs following a usability test-driven prototyping approach. Specifically, we demonstrate how TESY's capability of tightly integrating test data provided by crowd-testers into the prototyping tool leads to more data-driven and task-focused design improvements.

Interactive systems are omnipresent today and the need to create graphical user interfaces (GUIs) is just as ubiq-uitous. For the elicitation and validation of requirements, GUI prototyping is a well-known and effective technique, typically employed after gathering initial user requirements represented in natural language (NL) (e.g., in the form of user stories). Un-fortunately, G UI prototyping often requires extensive resources, resulting in a costly and time-consuming process. Despite various easy-to-use prototyping tools in practice, there is often a lack of adequate resources for developing G UI prototypes based on given user requirements. In this work, we present a novel Large Language Model (LLM)-based approach providing assistance for validating the implementation of functional NL- based require-ments in a GUI prototype embedded in a prototyping tool. In particular, our approach aims to detect functional user stories that are not implemented in a G UI prototype and provides recommendations for suitable GUI components directly imple-menting the requirements. We collected requirements for existing GUIs in the form of user stories and evaluated our proposed validation and recommendation approach with this dataset. The obtained results are promising for user story validation and we demonstrate feasibility for the GUI component recommendations.