Artificial intelligence (AI) summaries appear in many Web searches, but there have been errors and questionable quality in the AI responses and source attributions. This dissertation project consists of two studies that explore the impact of generative AI summary features on user perceptions and decision-making. The first experimental study examines the effects of two designs of generative AI search responses (narrative point of view, source authority) on user perceptions and information adoption about consumer health topics. AI summaries were collected, modified, and used in a randomised, within-subjects experiment with crowdsourced workers (N=234). The Information Adoption Model (IAM) informed the conceptual model of the experiment, and the Comprehensive Model of Information Seeking (CMIS) guided the health task design. User perceptions are assessed through self-reported measures of persuasive communication (argument quality, source credibility), information usefulness, domain knowledge, and involvement. The second qualitative study aims to explore how AI summaries and source attributions are utilised alongside traditional Web search results for tasks of personal interest. This is an in-person study involving search sessions, retrospective think-alouds, and semi-structured interviews with graduate students (N=16).

Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device’s system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at https://github.com/google-research/android_world.

Autonomous agents that control user interfaces to accomplish human tasks are emerging. Leveraging LLMs to power such agents has been of special interest, but unless fine-tuned on human-collected task demonstrations, performance is still relatively low. In this work we study whether fine-tuning alone is a viable approach for building real-world UI control agents. To this end we collect and release a new dataset, AndroidControl, consisting of 15,283 demonstrations of everyday tasks with Android apps. Compared to existing datasets, each AndroidControl task instance includes both high and low-level human-generated instructions, allowing us to explore the level of task complexity an agent can handle. Moreover, AndroidControl is the most diverse computer control dataset to date, including 14,548 unique tasks over 833 Android apps, thus allowing us to conduct in-depth analysis of the model performance in and out of the domain of the training data. Using the dataset, we find that when tested in domain fine-tuned models outperform zero and few-shot baselines and scale in such a way that robust performance might feasibly be obtained simply by collecting more data. Out of domain, performance scales significantly more slowly and suggests that in particular for high-level tasks, fine-tuning on more data alone may be insufficient for achieving robust out-of-domain performance.

This study investigated the effect of video abstract (VA) presentation style (slideshow versus animation) on viewer comprehension and user engagement. Video abstracts, short video presentations of journal articles, were selected and used in a randomized between-subjects experiment (N = 290) with Amazon Mechanical Turk (MTurk) participants. The Cognitive Theory of Multimedia Learning (CTML) informed the selection of VAs. Barrett’s Taxonomy of Cognitive and Affective Dimensions of Reading Comprehension (Barrett’s Taxonomy) was used to develop comprehension measures focused on recall and summarization, and user engagement was measured using a questionnaire. The study found that: 1) comprehension outcomes did not vary between slideshow and animation style VAs, 2) animation VAs were perceived to be more engaging than slideshow VAs, and 3) user engagement was weakly negatively correlated with comprehension scores. In other words, animation VAs attracted viewers with their content, but did not lead to increased comprehension. In fact, viewers in the study who reported higher levels of engagement had slightly lower comprehension outcomes.

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AitW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10–13), and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance, and, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/androidinthe_wild.

This paper presents a method for extracting both syntactic and semantic tags. An extended CFG parser works in conjunction with an HMM model, which handles unknown words and partially known words, to yield a complete syntactic and semantic interpretation of the utterance. Four experiments and applications were performed using the paradigm to show the usefulness of the approach in processing spoken sentences.