This paper contributes the first large-scale dataset of 17,979 hand-drawn sketches of 21 UI element categories collected from 967 participants, including UI/UX designers, front-end developers, HCI, and CS grad students, from 10 different countries. We performed a perceptual study with this dataset and found out that UI/UX designers can recognize the UI element sketches with ~96% accuracy. To compare human performance against computational recognition methods, we trained the state-of-the-art DNN-based image classification models to recognize the UI elements sketches. This study revealed that the ResNet-152 model outperforms other classification networks and detects unknown UI element sketches with 91.77% accuracy (chance is 4.76%). We have open-sourced the entire dataset of UI element sketches to the community intending to pave the way for further research in utilizing AI to assist the conversion of lo-fi UI sketches to higher fidelities.

In agile development, lean UX designers perform rapid prototyping and quick evaluation of prototypes to ensure fast releases. To understand designers' workflow during rapid prototyping, we interviewed 15 lean UX designers. We identified the following pain points in the workflow: 1) Compromise on quality of UI design due to time constraint 2) UI design knowledge being scattered among numerous sources such as websites and books 3) Inability of developers to reproduce the same quality of UI design due to lack of UI design knowledge. To address these issues, we propose a UI design pattern-driven approach for rapid prototyping. To realize this approach, we introduce Kiwi, a library for UI design patterns and guidelines that aims to consolidate UI design knowledge for mobile applications. Each UI design pattern consists of a problem statement, context, rationale, and a proposed solution. Additionally, Kiwi provides downloadable and customizable GUI examples, layout blueprints and front-end code for each pattern. Usability evaluation (SUS) of Kiwi with 21 lean UX designers depict good usability and high learnability.