Choosing colors is a pivotal but challenging component of graphic design. The paper presents an intelligent interaction technique supporting designers’ creativity in color design. It fills a gap in the literature by proposing an integrated technique for color exploration, assignment, and refinement: CoColor. Our design goals were 1) let designers focus on color choice by freeing them from pixel-level editing and 2) support rapid flow between low- and high-level decisions. Our interaction technique utilizes three steps – choice of focus, choice of suitable colors, and the colors’ application to designs – wherein the choices are interlinked and computer-assisted, thus supporting divergent and convergent thinking. It considers color harmony, visual saliency, and elementary accessibility requirements. The technique was incorporated into the popular design tool Figma and evaluated in a study with 16 designers. Participants explored the coloring options more easily with CoColor and considered it helpful.

During the design of graphical user interfaces (GUIs), one typical objective is to ensure compliance with pertinent style guides, ongoing design practices, and design systems. However, designing compliant layouts is challenging, time-consuming, and can distract creative thinking in design. This paper presents a method for interactive layout transfer, where the layout of a source design – typically an initial rough working draft – is transferred automatically using a selected reference/template layout while complying with relevant guidelines. Our integer programming (IP) method extends previous work in two ways: first, by showing how to transform a rough draft into the final target layout using a reference template and, second, by extending IP-based approaches to adhere to guidelines. We demonstrate how to integrate the method into a real-time interactive GUI sketching tool. Evaluation results are presented from a case study and from an online experiment where the perceived quality of layouts was assessed.

The problem of computational design for menu systems has been addressed in some specific cases such as the linear menu (list). The classical approach has been to model this problem as an assignment task, where commands are assigned to menu positions while optimizing for users’ selection performance and grouping of associated items. However, we show that this approach fails with larger, hierarchically organized menus because it does not take into account the ways in which users navigate hierarchical structures. This paper addresses the computational menu design problem by presenting a novel integer programming formulation that yields usable, well-ordered command hierarchies from a single model. First, it introduces a novel objective function based on information foraging theory, which minimizes navigation time in a hierarchical structure. Second, it models the hierarchical menu design problem as a combination of the exact set covering problem and the assignment problem, organizing commands into ordered groups of ordered groups. The approach is efficient for large, representative instances of the problem. In a controlled usability evaluation, the performance of computationally designed menus was ∼25% faster to use than existing commercial designs. We discuss applications of this approach for personalization and adaptation.

Graphical user interfaces (GUIs) have gained primacy among the means of interacting with computing systems, thanks to the way they leverage human perceptual and motor capabilities. However, the design of GUIs has mostly been a manual activity. To design a GUI, the designer must select its visual, spatial, textual, and interaction properties such that the combination strikes a balance among the relevant human factors. While emerging computational-design techniques have addressed some problems related to grid layouts, no general approach has been proposed that can also produce good and complete results covering color-related decisions and other nonlinear design objectives. Evolutionary algorithms are promising and demonstrate good handling of similar problems in other conditions, genetic operators, depending on how they are designed. But even these approaches struggle with elements' overlap and hence produce too many infeasible candidate solutions. This paper presents a new approach based on grid-based genetic operators demonstrated in a non-dominated sorting genetic algorithm (NSGA-III) setting. The operators use grid lines for element positions in a novel manner to satisfy overlap-related constraints and intrinsically improve the alignment of elements. This approach can be used for crossovers and mutations. Its core benefit is that all the solutions generated satisfy the no-overlap requirement and represent well-formed layouts. The new operators permit using genetic algorithms for increasingly realistic task instances, responding to more design objectives than could be considered before. Specifically, we address grid quality, alignment, selection time, clutter minimization, saliency control, color harmony, and grouping of elements.

Grid layouts are used by designers to spatially organise user interfaces when sketching and wireframing. However, their design is largely time consuming manual work. This is challenging due to combinatorial explosion and complex objectives, such as alignment, balance, and expectations regarding positions. This paper proposes a novel optimisation approach for the generation of diverse grid-based layouts. Our mixed integer linear programming (MILP) model offers a rigorous yet efficient method for grid generation that ensures packing, alignment, grouping, and preferential positioning of elements. Further, we present techniques for interactive diversification, enhancement, and completion of grid layouts. These capabilities are demonstrated using GRIDS, a wireframing tool that provides designers with real-time layout suggestions. We report findings from a ratings study (N = 13) and a design study (N = 16), lending evidence for the benefit of computational grid generation during early stages of design.

For graphical user interface (UI) design, it is important to understand what attracts visual attention. While previous work on saliency has focused on desktop and web-based UIs, mobile app UIs differ from these in several respects. We present findings from a controlled study with 30 participants and 193 mobile UIs. The results speak to a role of expectations in guiding where users look at. Strong bias toward the top-left corner of the display, text, and images was evident, while bottom-up features such as color or size affected saliency less. Classic, parameter-free saliency models showed a weak fit with the data, and data-driven models improved significantly when trained specifically on this dataset (e.g., NSS rose from 0.66 to 0.84). We also release the first annotated dataset for investigating visual saliency in mobile UIs.

Computing systems have been facing severe technology challenges in recent years with regard to power consumption, circuit reliability, and high performance. For many years, the issues of power consumption and performance have been addressed with the use of technology scaling.However, as Dennard’s scaling tends toward an end, it has become difficult to further improve the performance under the same power constraints. In addition to power, reliability also becomes a critical issue when the feature size of the complementary metal-oxide–semiconductor (CMOS) technology is reduced below 7 nm. Thus, ensuring the complete accuracy of the signal has become increasingly challenging in recent years.

The graphical user interface (GUI) has become the prime means for interacting with computing systems. It leverages human perceptual and motor capabilities for elementary tasks such as command exploration and invocation, information search, and multitasking. For designing a GUI, numerous interconnected decisions must be made such that the outcome strikes a balance between human factors and technical objectives. Normally, design choices are specified manually and coded within the software by professional designers and developers. This article surveys combinatorial optimization as a flexible and powerful tool for computational generation and adaptation of GUIs. As recently as 15 years ago, applications were limited to keyboards and widget layouts. The obstacle has been the mathematical definition of design tasks, on the one hand, and the lack of objective functions that capture essential aspects of human behavior, on the other. This article presents definitions of layout design problems as integer programming tasks, a coherent formalism that permits identification of problem types, analysis of their complexity, and exploitation of known algorithmic solutions. It then surveys advances in formulating evaluative functions for common design-goal foci such as user performance and experience. The convergence of these two advances has expanded the range of solvable problems. Approaches to practical deployment are outlined with a wide spectrum of applications. This article concludes by discussing the position of this application area within optimization and human-computer interaction research and outlines challenges for future work.

The article addresses a key challenge in the design of content for pervasive displays: how to engage passers-by who have limited time and attention? To achieve this, we apply a novel approach for computational design of interesting display content using tiled layouts. We present a model of display foraging based on information foraging theory to describe the behavior of a rational but time-limited user looking at a display. Accordingly, our work aims to maximize the information gain for tiled displays. This complex problem is divided into two phases: (1) generating designs of tiled layouts and (2) assigning content options to individual tiles based on what predicted by display foraging. Accordingly, a proof-of-concept system was realized then evaluated computationally and empirically with a control study and field study. The results show that the proposed system can engage significantly more people than typical digital signage.

Developing cross-device multi-user interfaces (UIs) is a challenging problem. There are numerous ways in which content and interactivity can be distributed. However, good solutions must consider multiple users, their roles, their preferences and access rights, as well as device capabilities. Manual and rule-based solutions are tedious to create and do not scale to larger problems nor do they adapt to dynamic changes, such as users leaving or joining an activity. In this paper, we cast the problem of UI distribution as an assignment problem and propose to solve it using combinatorial optimization. We present a mixed integer programming formulation which allows real-time applications in dynamically changing collaborative settings. It optimizes the allocation of UI elements based on device capabilities, user roles, preferences, and access rights. We present a proof-of-concept designer-in-the-loop tool, allowing for quick solution exploration. Finally, we compare our approach to traditional paper prototyping in a lab study.

Carrier Ethernet is rapidly being deployed in the metropolitan and core segments of the transport network. One of the emerging flavors of Carrier Ethernet is the IEEE 802.1Qay PBB-TE or Provider Backbone Bridging-Traffic Engineering standard. PBB-TE relies on the assignment of a network-specific Virtual Local Area Network (VLAN) tag, called the Backbone VLAN ID or BVID that is used in conjunction with a backbone Media Access Control (MAC) address for forwarding. The 12-bit BVID along with 48-bit Backbone MAC address are used to forward an Ethernet frame. The assignment of BVIDs in a network is critical, given that there are only 4094 possible assignments, especially for those paths that are overlapping in the network graph and incident at the same destination. While the only way to scale is to reuse BVIDs, this method can lead to a complication if the same BVID is allocated to an overlapping path. To the best of our knowledge, this is the first instance of isolating this problem of limited BVID availability which rises only due to graphical overlap between services. We formulate and solve this as a constrained optimization problem. We present optimal and heuristic algorithms to solve the BVID problem. The optimal approach solves the `static' case, while the heuristic can solve both the `static' and the `dynamic' cases of the BVID allocation problem. Results show that the developed heuristics perform close to the optimal and can be used in commercial settings for both the static and dynamic cases.