CHI ’25, Yokohama, Japan

Can you explain how inflation affects interest rates and the broader economy?... According to the Taylor Rule, monetary policy should adjust the federal funds rate in response to deviations of actual inflation from the target rate and output from potential GDP. (User pauses to think and digest, while the LLM continues generating content) This increase in rates raises the cost of borrowing, reduces consumer spending and business investment, and can lead to slower economic growth ... Waiting User Normal User (Unaware of Change) Satisfied User How do I make olive oil garlic pasta?Sure! I'd be more than happy to help you with that.Olive oil garlic pasta is a wonderfully simple yet flavorful dish that's perfect for just about any occasion.(Streaming paused due to limited computational resource, leaving the user idle and disengaged) ... Can you explain how inflation affects interest rates and the broader economy?How do I make olive oil garlic pasta?... According to the Taylor Rule, monetary policy should adjust the federal funds rate in response to deviations of actual inflation from the target rate and output from potential GDP. (Streaming is slowed, allowing the user time to digest the content and freeing up computational resources) ... Sure! I'd be more than happy to help you with that.Olive oil garlic pasta is a wonderfully simple yet flavorful dish that's perfect for just about any occasion.To get started, you'll need a few basic ingredients: spaghetti, garlic, extra virgin olive oil, red pepper flakes, salt, and parsley... (Streaming continues smoothly thanks to sufficient computational resources, and proceeds at a faster pace since the content is low in complexity) Normal User Figure 1: (a) In regular LLM streaming, the streaming speed is not associated with the content being delivered.Complex content may be streamed faster than users can comfortably read, resulting in wasted resource.Conversely, simple content may be streamed too slowly, causing users to wait unnecessarily.(b) In our proposed approach, the streaming speed is adapted based on the estimated content complexity.Complex content is slowed down to support user comprehension and optimize resource usage, while simpler content is streamed more quickly to better align with the user's natural reading speed.

Recent advances in Large Language Models (LLMs) have demonstrated their potential as autonomous agents across various tasks. One emerging application is the use of LLMs in playing games. In this work, we explore a practical problem for the gaming industry: Can LLMs be used to measure game difficulty? We evaluate the feasibility of using LLM agents to test game difficulty, focusing on two widely played games: Wordle and Slay the Spire . Our results reveal an interesting finding: although LLMs may not perform as well as the average human player, their performance, when guided by simple, generic prompting techniques, shows a statistically significant and strong correlation with difficulty indicated by human players. This suggests that LLMs could potentially serve as human-like agents for measuring game difficulty during the development process, as their assessments may align closely with those of human players. Based on our experiments, we also propose general principles and guidelines for integrating LLMs into the game testing workflow.

We introduce MoiréWidgets, a novel approach for tangible interaction that harnesses the Moiré effect—a prevalent optical phenomenon—to enable high-precision event detection on physical widgets. Unlike other electronics-free tangible user interfaces which require close coupling with external hardware, MoiréWidgets can be used at greater distances while maintaining high-resolution sensing of interactions. We define a set of interaction primitives, e.g., buttons, sliders, and dials, which can be used as standalone objects or combined to build complex physical controls. These consist of 3D printed structural mechanisms with patterns printed on two layers—one on paper and the other on a plastic transparency sheet—which create a visual signal that amplifies subtle movements, enabling the detection of user inputs. Our technical evaluation shows that our method outperforms standard fiducial markers and maintains sub-millimeter accuracy at 100 cm distance and wide viewing angles. We demonstrate our approach by creating an audio console and indicate how our approach could extend to other domains.

Sound plays a crucial role in enhancing user experience and immersiveness in Augmented Reality (AR). However, current platforms lack support for AR sound authoring due to limited interaction types, challenges in collecting and specifying context information, and difficulty in acquiring matching sound assets. We present SonifyAR, an LLM-based AR sound authoring system that generates context-aware sound effects for AR experiences. SonifyAR expands the current design space of AR sound and implements a Programming by Demonstration (PbD) pipeline to automatically collect contextual information of AR events, including virtual-content-semantics and real-world context. This context information is then processed by a large language model to acquire sound effects with Recommendation, Retrieval, Generation, and Transfer methods. To evaluate the usability and performance of our system, we conducted a user study with eight participants and created five example applications, including an AR-based science experiment, and an assistive application for low-vision AR users.

Precisely understanding how objects move in 3D is essential for broad scenarios such as video editing, gaming, driving, and athletics. With screen-displayed computer graphics content, users only perceive limited cues to judge the object motion from the on-screen optical flow. Conventionally, visual perception is studied with stationary settings and singular objects. However, in practical applications, we---the observer---also move within complex scenes. Therefore, we must extract object motion from a combined optical flow displayed on screen, which can often lead to mis-estimations due to perceptual ambiguities. We measure and model observers' perceptual accuracy of object motions in dynamic 3D environments, a universal but under-investigated scenario in computer graphics applications. We design and employ a crowdsourcing-based psychophysical study, quantifying the relationships among patterns of scene dynamics and content, and the resulting perceptual judgments of object motion direction. The acquired psychophysical data underpins a model for generalized conditions. We then demonstrate the model's guidance ability to significantly enhance users' understanding of task object motion in gaming and animation design. With applications in measuring and compensating for object motion errors in video and rendering, we hope the research establishes a new frontier for understanding and mitigating perceptual errors caused by the gap between screen-displayed graphics and the physical world.

In this work, we study the problem of generating a set of images from an arbitrary tabular dataset. The set of generated images provides an intuitive visual summary of the tabular data that can be quickly and easily communicated and understood by the user. More specifically, we formally introduce this new dataset to image generation task and discuss a few motivating applications including exploratory data analysis and understanding customer segments for creating better marketing campaigns. We then curate a benchmark dataset for training such models, which we release publicly for others to use and develop new models for other important applications of interest. Further, we describe a general and flexible framework that serves as a fundamental basis for studying and developing models for this new task of generating images from tabular data. From the framework, we propose a few different approaches with varying levels of complexity and tradeoffs. One such approach leverages both numerical and textual data as the input to our image generation pipeline. The pipeline consists of an image decoder and a conditional auto-regressive sequence generation model which also includes a pre-trained tabular representation in the input layer. We evaluate the performance of these approaches through several quantitative metrics (FID for image quality and LPIPS scores for image diversity).

In this work, we develop a Graph Neural Network (GNN) framework for the problem of personalized visualization recommendation. The GNN-based framework first represents the large corpus of datasets and visualizations from users as a large heterogeneous graph. Then, it decomposes a visualization into its data and visual components, and then jointly models each of them as a large graph to obtain embeddings of the users, attributes (across all datasets in the corpus), and visual-configurations. From these user-specific embeddings of the attributes and visual-configurations, we can predict the probability of any visualization arising from a specific user. Finally, the experiments demonstrated the effectiveness of using graph neural networks for automatic and personalized recommendation of visualizations to specific users based on their data and visual (design choice) preferences. To the best of our knowledge, this is the first such work to develop and leverage GNNs for this problem.