Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general.Yet, these descriptions are constantly missing in mobile UIs.We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces.We collected a largescale dataset for widget captioning with crowdsourcing.Our dataset contains 162,859 language phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens.We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learning strategies impact the quality of predicted captions.The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal captioning task that connects language and user interfaces.

The Experience Sampling Method (ESM) enables researchers to capture information about participants' experiences in the moment. Adding an end-of-day retrospective survey also allows participants to elaborate on those experiences. Although the use of photos in retrospective interviews and surveys for memory elicitation is well known, little research has investigated the use of photos in ESM studies. As smartphone adoption increases facilitating ESM studies and making photo sharing easier, researchers need to continuously evaluate the method and investigate the role of photos in such studies. We conducted a large-scale ESM and retrospective survey study via Android smartphones with more than 1,000 US participants, and analyzed participants' photo submissions, including how photo use correlated with participants' data quality and what, if any, value photos added for researchers. Our study sheds light on the role of photos in ESM and retrospective studies that researchers can reference when constructing future study designs.

Search engines make it easy to check facts online, but finding some specific kinds of information sometimes proves to be difficult. We studied the behavioral signals that suggest that a user is having trouble in a search task. First, we ran a lab study with 23 users to gain a preliminary understanding on how users' behavior changes when they struggle finding the information they're looking for. The observations were then tested with 179 participants who all completed an average of 22.3 tasks from a pool of 100 tasks. The large-scale study provided quantitative support for our qualitative observations from the lab study. When having difficulty in finding information, users start to formulate more diverse queries, they use advanced operators more, and they spend a longer time on the search result page as compared to the successful tasks. The results complement the existing body of research focusing on successful search strategies.

We investigated the efficacy of visual and textual web page previews in predicting the helpfulness of web pages related to a specific topic. We ran two studies in the usability lab and collected data through an online survey. Participants (total of 245) were asked to rate the expected helpfulness of a web page based on a preview (four different thumbnail variations: a textual web page summary, a thumbnail/title/URL combination, a title/URL combination). In the lab studies, the same participants also rated the helpfulness of the actual web pages themselves. In the online study, the web page ratings were collected from a separate group of participants. Our results show that thumbnails add information about the relevance of web pages that is not available in the textual summaries of web pages (title, snippet & URL). However, showing only thumbnails, with no textual information, results in poorer performance than showing only textual summaries. The prediction inaccuracy caused by textual vs. visual previews was different: textual previews tended to make users overestimate the helpfulness of web pages, whereas thumbnails made users underestimate the helpfulness of web pages in most cases. In our study, the best performance was obtained by combining sufficiently large thumbnails (at least 200x200 pixels) with page titles and URLs - and it was better to make users focus primarily on the thumbnail by placing the title and URL below the thumbnail. Our studies highlighted four key aspects that affect the performance of previews: the visual/textual mode of the previews, the zoom level and size of the thumbnail, as well as the positioning of key information elements.

Web search services are among the most heavily used applications on the World Wide Web. Perhaps because search is used in such a huge variety of tasks and contexts, the user interface must strike a careful balance to meet all user needs. We describe a study that used eye tracking methodologies to explore the effects of changes in the presentation of search results. We found that adding information to the contextual snippet significantly improved performance for informational tasks but degraded performance for navigational tasks. We discuss possible reasons for this difference and the design implications for better presentation of search results.

Web search engines present search results in a rank ordered list. This works when what a user wants is near the top, but sometimes the information that the user really wants is located at the bottom of the page. This study examined how users' search behaviors vary when target results were displayed at various positions for informational and navigational tasks. We found that when targets were placed relatively low in the first page of search results, people spent more time searching and were less successful in finding the target, especially for informational tasks. Further analysis of eye movements showed that the decrease in search performance was partially due to the fact that users rarely looked at lower ranking results. The large decrease in performance for informational search is probably because users have high confidence in the search engine's ranking; in contrast to navigational tasks, where the target is more obvious from information presented in the results, in informational tasks, users try out the top ranked results even if these results are perceived as less relevant for the task.

Retrospective Think aloud (RTA) is a usability method that collects the verbalization of a user's performance after the performance is over. There has been little work done to investigate the validity and reliability of RTA. This paper reports on an experiment investigating these issues with a form of the method called stimulated RTA. By comparing subjects' verbalizations with their eye movements, we support the validity and reliability of stimulated RTA: the method provides a valid account of what people attended to in completing tasks, it has a low risk of introducing fabrications, and its validity isn't affected by task complexity. More detailed analysis of RTA shows that it also provides additional information about user's inferences and strategies in completing tasks. The findings of this study provide valuable support for usability practitioners to use RTA and to trust the users' performance information collected by this method in a usability study.

Inking and gesturing are two central tasks in pen-based user interfaces. Switching between modes for entry of uninterpreted ink and entry of gestures is required by many pen-based user interfaces. Without an appropriate mode switching technique, pen-based interactions in such situations may be inefficient and cumbersome. In this paper, we investigate five techniques for switching between ink and gesture modes in pen interfaces, including a pen-pressure based mode switching technique that allows implicit mode transition. A quantitative experimental study was conducted to evaluate the performance of these techniques. The results suggest that pressing a button with the non-preferred hand offers the fastest performance, while the technique of holding the pen still is significantly slower and more prone to error than the other techniques. Pressure, while promising, did not perform as well as the non-preferred hand button with our current implementation.

Informal presentations are a lightweight means for fast and convenient communication of ideas. People communicate their ideas to others on paper and whiteboards, which afford fluid sketching of graphs, words and other expressive symbols. Unlike existing authoring tools that are designed for formal presentations, we created SketchPoint to help presenters design informal presentations via freeform sketching. In SketchPoint, presenters can quickly author presentations by sketching slide content, overall hierarchical structures and hyperlinks. To facilitate the transition from idea capture to communication, a note-taking workspace was built for accumulating ideas and sketching presentation outlines. Informal feedback showed that SketchPoint is a promising tool for idea communication.