A key objective in the design of graphical user interfaces (GUIs) is to ensure consistency across screens of the same product. However, designing a compliant layout is time-consuming and can distract designers from creative thinking. This paper studies layout recommendation methods that fulfill such consistency requirements using machine learning. Given a desired element type and size, the methods suggest element placements following real-world GUI design processes. Consistency requirements are given implicitly through previous layouts from which patterns are to be learned, comparable to existing screens of a software product. We adopt two recently proposed methods for this task, a Graph Neural Network (GNN) and a Transformer model, and compare them with a custom approach based on sequence alignment and nearest neighbor search (kNN) . The methods were tested on handcrafted datasets with explicit layout patterns, as well as large-scale public datasets of diverse mobile design layouts. Our results show that our instance-based learning algorithm outperforms both neural network approaches. Ultimately, this work contributes to establishing smarter design tools for professional designers with explainable algorithms that increase their efficacy.

During the design of graphical user interfaces (GUIs), one typical objective is to ensure compliance with pertinent style guides, ongoing design practices, and design systems. However, designing compliant layouts is challenging, time-consuming, and can distract creative thinking in design. This paper presents a method for interactive layout transfer, where the layout of a source design – typically an initial rough working draft – is transferred automatically using a selected reference/template layout while complying with relevant guidelines. Our integer programming (IP) method extends previous work in two ways: first, by showing how to transform a rough draft into the final target layout using a reference template and, second, by extending IP-based approaches to adhere to guidelines. We demonstrate how to integrate the method into a real-time interactive GUI sketching tool. Evaluation results are presented from a case study and from an online experiment where the perceived quality of layouts was assessed.

Searchers often make a choice in a matter of seconds on SERPs. As a result of a dynamic cognitive process, choice is ultimately reflected in motor movement and thus can be modeled by tracking the computer mouse. However, because not all movements have equal value, it is important to understand how do they and, critically, their sequence length impact model performance. We study three different SERP scenarios where searchers (1)~noticed an advertisement, (2)~abandoned the page, and (3)~became frustrated. We model these scenarios with recurrent neural nets and study the effect of mouse sequence padding and truncating to different lengths. We find that it is possible to predict the aforementioned tasks sometimes using just 2 seconds of movement. Ultimately, by efficiently recording the right amount of data, we can save valuable bandwidth and storage, respect the users' privacy, and increase the speed at which machine learning models can be trained and deployed. Considering the web scale, doing so will have a net benefit on our environment.

Most successful search queries do not result in a click if the user can satisfy their information needs directly on the SERP. Modeling query abandonment in the absence of click-through data is challenging because search engines must rely on other behavioral signals to understand the underlying search intent. We show that mouse cursor movements make a valuable, low-cost behavioral signal that can discriminate good and bad abandonment. We model mouse movements on SERPs using recurrent neural nets and explore several data representations that do not rely on expensive hand-crafted features and do not depend on a particular SERP structure. We also experiment with data resampling and augmentation techniques that we adopt for sequential data. Our results can help search providers to gauge user satisfaction for queries without clicks and ultimately contribute to a better understanding of search engine performance.