Artificial intelligence (AI) technologies offer tremendous potential for product and service innovation, yet finding good use cases remains challenging. Currently, AI projects largely fail due to breakdowns in early stage ideation and problem formulation. Drawing on HCI research that used AI capabilities and examples to facilitate AI concept ideation, this paper investigates how these approaches might be operationalized in industry settings. We collaborated with cross-functional industry teams in insurance, accounting, and consultancy. We conducted a series of AI Design Sprints, where innovators simultaneously consider AI capabilities and human needs. All teams perceived the ideation method highly valuable both for rapidly exploring use cases and building AI literacy within teams. We detail our process, the challenges, and artifacts that proved effective. We share insights on how AI projects get initiated, and how innovation teams identify use cases. Reflecting on these case studies, we discuss opportunities for improving early stage AI innovation.

Navigating new indoor spaces and interacting with the environment presents many challenges for people who are blind or have low vision (BLV). To address these challenges, we prototyped a smartphone-based conversational assistant that helps BLV people navigate and interact with their environment. The prototype utilizes a cognitive architecture to integrate three different technologies: (i) augmented-reality spatial anchors for high-precision localization and access to static information about the environment; (ii) real-time object/people detection for information about the environment and obstacle avoidance; and (iii) a conversational agent}that uses large language models (LLMs) for information extraction, conversational interaction, and turn-by-turn navigation. We assess the impact of different technologies on human performance by measuring user task time and errors. We found that conversational interaction holistically integrates the different technologies to deliver a better user experience while significantly reducing task completion time.

Worker-Centered Design (WCD) has gained prominence over the past decade, offering researchers and practitioners ways to engage worker agency and support collective actions for workers. Yet few studies have systematically revisited WCD itself, examining its implementations, challenges, and practical impact. Through a four-lens analytical framework that examines multiple facets of WCD within food delivery industry, we identify critical tensions and blind spots from a Multi-Laborer System perspective. Our analysis reveals conflicts across labor chains, distorted implementations of WCD, designers’ sometimes limited political-economic understanding, and workers as active agents of change. These insights further inform a Diagnostic-Generative pathway that helps to address recurring risks, including labor conflicts and institutional reframing, while cultivating designers’ policy and economic imagination. Following the design criticism tradition, and through a four-lens reflexive analysis, this study expands the action space for WCD and strengthens its relevance to real-world practice.

Around 90% of CEOs see AI as the “most critical technology for ensuring future profitability and competitiveness.” At the same time, up to 95% of AI projects fail. Currently, little is known about the leaders who approve and guide AI initiatives. We call them AI Deciders. This study investigates how AI Deciders reason about AI benefits and risks, and how their knowledge about AI influences their decisions on what and where to innovate. We interviewed AI Deciders across diverse organizations. We found no ideation. AI Deciders just consider one concept at a time. Design and HCI played no role in deciding what to build. Many AI Deciders overestimate AI’s benefits while underestimating risks. Based on these findings, we identified opportunities for design and HCI to support impactful and responsible AI innovation. This should reduce AI project failure.

AI projects often fail due to financial, technical, ethical, or user acceptance challenges-failures frequently rooted in early-stage decisions.While HCI and Responsible AI (RAI) research emphasize this, practical approaches for identifying promising concepts early remain limited.Drawing on Research through Design, this paper investigates how early-stage AI concept sorting in commercial settings can reflect RAI principles.Through three design experiments-including a probe study with industry practitioners-we explored methods for evaluating risks and benefits using multidisciplinary collaboration.Participants demonstrated strong receptivity to addressing RAI concerns early in the process and effectively identified low-risk, high-benefit AI concepts.Our findings highlight the potential of a design-led approach to embed ethical and service design thinking at the front end of AI innovation.By examining how practitioners reason about AI concepts, our study invites HCI and RAI communities to see early-stage innovation as a critical space for engaging ethical and commercial considerations together.

Innovators transform the world by understanding where services are successfully meeting customers' needs and then using this knowledge to identify failsafe opportunities for innovation. Pre-trained models have changed the AI innovation landscape, making it faster and easier to create new AI products and services. Understanding where pre-trained models are successful is critical for supporting AI innovation. Unfortunately, the hype cycle surrounding pre-trained models makes it hard to know where AI can really be successful. To address this, we investigated pre-trained model applications developed by HCI researchers as a proxy for commercially successful applications. The research applications demonstrate technical capabilities, address real user needs, and avoid ethical challenges. Using an artifact analysis approach, we categorized capabilities, opportunity domains, data types, and emerging interaction design patterns, uncovering some of the opportunity space for innovation with pre-trained models.

Young adults often encounter challenges in career exploration. Self-guided interventions, such as the letter-exchange exercise, where participants envision and adopt the perspective of their future selves by exchanging letters with their envisioned future selves, can support career development. However, the broader adoption of such interventions may be limited without structured guidance. To address this, we integrated Large Language Model (LLM)-based agents that simulate participants' future selves into the letter-exchange exercise and evaluated their effectiveness. A one-week experiment (N=36) compared three conditions: (1) participants manually writing replies to themselves from the perspective of their future selves (baseline), (2) future-self agents generating letters to participants, and (3) future-self agents engaging in chat conversations with participants. Results indicated that exchanging letters with future-self agents enhanced participants' engagement during the exercise, while overall benefits of the intervention on future orientation, career self-concept, and psychological support remained comparable across conditions. We discuss design implications for AI-augmented interventions for supporting young adults' career exploration.

As the population of older adults increases, there is a growing need for support for them to age in place. This is exacerbated by the growing number of individuals struggling with cognitive decline and shrinking number of youth who provide care for them. Artificially intelligent agents could provide cognitive support to older adults experiencing memory problems, and they could help informal caregivers with coordination tasks. To better understand this possible future, we conducted a speed dating with storyboards study to reveal invisible social boundaries that might keep older adults and their caregivers from accepting and using agents. We found that healthy older adults worry that accepting agents into their homes might increase their chances of developing dementia. At the same time, they want immediate access to agents that know them well if they should experience cognitive decline. Older adults in the early stages of cognitive decline expressed a desire for agents that can ease the burden they saw themselves becoming for their caregivers. They also speculated that an agent who really knew them well might be an effective advocate for their needs when they were less able to advocate for themselves. That is, the agent may need to transition from being unremarkable to remarkable. Based on these findings, we present design opportunities and considerations for agents and articulate directions of future research.

The population of older adults experiencing cognitive decline is growing faster than the number of workers who can care for them. Artificially intelligent (AI) agents could assist these older adults, keeping them in their homes longer. For this to happen, older adults must be willing to adopt and rely on agents. Would they trust an agent that might need to report their decline to others? We conducted a speed dating study exploring the impact of agent affiliation (i.e., who the agent should work for). Our healthy and declining participants reacted positively to the idea of agents supporting them. They particularly recognized how the agent would reduce the burden placed on their family caregivers. They viewed affiliation to be dynamic, shifting from the declining older adult and orienting more to their caregivers over the course of cognitive decline. They envisioned the agent modifying its decision-making process to be like their caregivers’.

HCI and design studies have increasingly identified challenges for gig workers and advocated for designs centered around worker justice. However, there’s an existing research gap in understanding how platform designers approach gig worker issues in their practice. Our study engaged ten platform designers from food delivery and ride-hailing platforms to investigate this gap. Through semi-structured interviews, we uncovered their strategies, the extent of authority and responsibilities, and the range of obstacles they encounter in influencing decision-making that could affect gig workers’ experiences with the platforms. While platform designers were aware of gig worker issues, they confronted challenges from business goals, decision-making power, policies, and job security in promoting worker well-being. We discuss the jurisdiction of platform designers and propose that HCI research should further support them, who are deeply engaged in the gig economy and have the potential to participate in addressing social justice issues.

Teaching is one of many professions for which personalized feedback and reflection can help improve dialogue and discussion between the professional and those they serve. However, professional development (PD) is often impersonal as human observation is labor-intensive. Data-driven PD tools in teaching are of growing interest, but open questions about how professionals engage with their data in practice remain. In this paper, we present ClassInSight, a tool that visualizes three levels of teachers’ discussion data and structures reflection. Through 22 reflection sessions and interviews with 5 high school science teachers, we found themes related to dissonance, contextualization, and sustainability in how teachers engaged with their data in the tool and in how their professional vision, the use of professional expertise to interpret events, shifted over time. We discuss guidelines for these conversational support tools to support personalized PD in professions beyond teaching where conversation and interaction are important.

Policies significantly shape computation’s societal impact, a crucial HCI concern. However, challenges persist when HCI professionals attempt to integrate policy into their work or affect policy outcomes. Prior research considered these challenges at the “border” of HCI and policy. This paper asks: What if HCI considers policy integral to its intellectual concerns, placing system-people-policy interaction not at the border but nearer the center of HCI research, practice, and education? What if HCI fosters a mosaic of methods and knowledge contributions that blend system, human, and policy expertise in various ways, just like HCI has done with blending system and human expertise? We present this re-imagined HCI-policy relationship as a provocation and highlight its usefulness: It spotlights previously overlooked system-people-policy interaction work in HCI. It unveils new opportunities for HCI’s futuring, empirical, and design projects. It allows HCI to coordinate its diverse policy engagements, enhancing its collective impact on policy outcomes.

Advances in artificial intelligence (AI) have enabled unprecedented capabilities, yet innovation teams struggle when envisioning AI concepts. Data science teams think of innovations users do not want, while domain experts think of innovations that cannot be built. A lack of effective ideation seems to be a breakdown point. How might multidisciplinary teams identify buildable and desirable use cases? This paper presents a first hand account of ideating AI concepts to improve critical care medicine. As a team of data scientists, clinicians, and HCI researchers, we conducted a series of design workshops to explore more effective approaches to AI concept ideation and problem formulation. We detail our process, the challenges we encountered, and practices and artifacts that proved effective. We discuss the research implications for improved collaboration and stakeholder engagement, and discuss the role HCI might play in reducing the high failure rate experienced in AI innovation.

Clinical practice guidelines, care pathways, and protocols are designed to support evidence-based practices for clinicians; however, their adoption remains a challenge. We set out to investigate why clinicians deviate from the “Wake Up and Breathe” protocol, an evidence-based guideline for liberating patients from mechanical ventilation in the intensive care unit (ICU). We conducted over 40 hours of direct observations of live clinical workflows, 17 interviews with frontline care providers, and 4 co-design workshops at three different medical intensive care units. Our findings indicate that unlike prior literature suggests, disagreement with the protocol is not a substantial barrier to adoption. Instead, the uncertainty surrounding the application of the protocol for individual patients leads clinicians to deprioritize adoption in favor of tasks where they have high certainty. Reflecting on these insights, we identify opportunities for technical systems to help clinicians in effectively executing the protocol and discuss future directions for HCI research to support the integration of protocols into clinical practice in complex, team-based healthcare settings.

Public sector agencies aim to innovate not just for efficiency but also to enhance equity. Despite the growing adoption of data-driven decision-making systems in the public sector, efforts to integrate equity as a primary goal often fall short. This typically arises from inadequate early-stage involvement of underserved stakeholders and prevalent misunderstandings concerning the authentic meaning of equity from these stakeholders' perspectives. Our research seeks to address this gap by actively involving undersevered stakeholders in the process of envisioning the integration of equity within public sector data-driven decisions, particularly in the context of a building department in a Northeastern mid-sized U.S. city. Applying a speed dating method with storyboards, we explore diverse equity-centric futures within the realm of local business development, a domain where small businesses, particularly women-and minority-owned businesses, historically confront inequitable distribution of public services. We explored three essential aspects of equity: monitoring equity, resource allocation prioritization, as well as information and equity. Our findings illuminate the complexities of integrating equity into data-driven decisions, offering nuanced insights about the needs of stakeholders. We found that attempts to monitor and incorporate equity goals into public sector decision-making can unexpectedly backfire, inadvertently sparking community apprehension and potentially exacerbating existing inequities. Small business owners, including those identifying as women-and minority-owned, advocated against the use of demographic-based data in equity-focused data-driven decision-making in the public sector, instead emphasizing factors such as community needs, application complexity, and uncertainties inherent in small businesses. Drawing from these insights, we propose design implications to assist designers of public sector data-driven decision-making systems to better accommodate equity considerations.

The gig economy and digital labor platforms, such as food delivery, have become essential while also troubling the current socioeconomic landscape. Delivery platforms promise entry-level work, flexibility, and other benefits. However, researchers remain divided on if these platforms benefit workers and society at large. This study aims to shed light on the comprehensive challenges in food delivery work, uncovering gig worker-centered design opportunities to improve the lives of food couriers. Adopting an exploratory research process, we analyzed 19 ride-along food delivery videos and performed nine semi-structured interviews with food couriers in Portugal. Our findings illustrated the complexity and challenging nature of delivery work due to the entangled physical, digital, social, natural, and human factors. We captured and discussed gig worker-centered opportunities that surfaced from work challenges, echoing the needs of food couriers about supporting work, justice, inclusion, and work vision.

Multi-modal classroom sensing systems can collect complex behaviors in the classroom at a scale and precision far greater than human observers to capture learning insights and provide personalized teaching feedback. As students are critical stakeholders in the adoption of smart classrooms for the improvement of teaching, open questions remain in understanding student perspectives on the use of their data to provide insights to instructors. We conducted a Speed Dating with storyboards study to explore student values and boundaries regarding the acceptance of classroom sensing systems in STEM college courses. We found that students have several emergent beliefs about teaching and learning that influence their views towards smart classroom technologies. Students also held contextual views on the boundaries of data use depending on the outcome. Our findings have implications for the design and communication of classroom sensing systems that reconcile student and instructor beliefs around teaching and learning.

Advances in artificial intelligence have enabled unprecedented technical capabilities, yet making these advances useful in the real world remains challenging. We engaged in a Research through Design process to improve the ideation of AI products and services. We developed a design resource capturing AI capabilities based on 40 AI features commonly used across various domains. To probe its usefulness, we created a set of slides illustrating AI capabilities and asked designers to ideate AI-enabled user experiences. We also incorporated capabilities into our own design process to brainstorm concepts with domain experts and data scientists. Our research revealed that designers should focus on innovations where moderate AI performance creates value. We reflect on our process and discuss research implications for creating and assessing resources to systematically explore AI’s problem-solution space.

HCI research has explored AI as a design material, suggesting that designers can envision AI’s design opportunities to improve UX. Recent research claimed that enterprise applications offer an opportunity for AI innovation at the user experience level. We conducted design workshops to explore the practices of experienced designers who work on cross-functional AI teams in the enterprise. We discussed how designers successfully work with and struggle with AI. Our findings revealed that designers can innovate at the system and service levels. We also discovered that making a case for an AI feature’s return on investment is a barrier for designers when they propose AI concepts and ideas. Our discussions produced novel insights on designers’ role on AI teams, and the boundary objects they used for collaborating with data scientists. We discuss the implications of these findings as opportunities for future research aiming to empower designers in working with data and AI.

Design produces valuable knowledge by offering new perspectives that reframe problematic situations. Research through Design (RtD) contributes new frames along with design work demonstrating a frame's value. Interestingly, RtD papers rarely describe how reframing happens. This gap in documentation unintentionally implies a romantic account of design, it implies that the first step of an RtD project is to have a brilliant idea. This is especially problematic in cases where the reframing causes a pivot that leads to a new research program. To help address this gap, we describe a case where through a series of three design experiments we experienced a research pivot. We describe how our work to improve web-table navigation for screen-reader users broke our frame. The break led to a new research program focused on constructing a conversational internet. This paper offers our case along with reflection on reporting design work that drives reframing.

Social agents and robots are moving into front-line positions in brick and mortar services, taking on roles where they directly interact with customers. These agents could potentially recognize customers to personalize service. Will customers like this, or might they feel monitored and profiled? Robots could also re-embody (move their “personality” between one body and another) in order to take on multiple roles that are typically performed by different people. Will this make customers feel more taken care of, or will it raise concerns about the robot’s competence and expertise? Our work investigates when robots should and should not recognize customers and re-embody. Our online study used storyboards to present possible future interactions between robots and customers across several different service contexts. Our findings suggest that people generally accept robots identifying customers and taking on vastly different roles. However, in some contexts, these robot behaviors seem creepy and untrustworthy.

The ongoing transition within tech industry towards holistic service journey orchestration pushes user experience design (UXD) closer to service design (SD). At the same time, new digital channels for traditional services pushes SD closer to UXD. The increasing interplay between UXD and SD can cause confusion about their relationship, boundaries, and responsibilities. Little research has examined perceptions of these aspects in industry and academia. We surveyed 197 UXD and SD practitioners and academics about their understanding of the overlaps and boundaries between the two design practices. Our findings reveal several conflicting views around the scope of UXD and SD work, the subsumptive relation between UXD and SD, and the theoretical roots and methods each draws upon. We suggest constructive ways for the two design practices to complement each other. Our study shows that UX work is changing, and understanding this change is important to maintain the relevance of UX research.

Recently, transformer language models have been applied to build both task-and non-taskoriented dialogue systems.Although transformers perform well on most of the NLP tasks, they perform poorly on context retrieval and symbolic reasoning.Our work aims to address this limitation by embedding the model in an operational loop that blends both natural language generation and symbolic injection.We evaluated our system on the multi-domain DSTC8 data set and reported joint goal accuracy of 75.8% (ranked among the first half positions), intent accuracy of 97.4% (which is higher than the reported literature), and a 15% improvement for success rate compared to a baseline with no symbolic injection.These promising results suggest that transformer language models can not only generate proper system responses but also symbolic representations that can further be used to enhance the overall quality of the dialogue management as well as serving as scaffolding for complex conversational reasoning.

Contesting Control offers a wonderful example of how to execute artist-led HCI research. The work is excellent due to the clear and narrow focus on contested control and the design and execution of...

Agents that support spoken interaction (e.g., Amazon Echo) are designed for social spaces like the home, yet designers know little about how they should respond to social activity around them. We set out to reconsider current one-on-one interactions with agents, and explore the design space of future socially sophisticated agents. To do so, we use an iterative co-design process with designers and theatre experts to devise an immersive performance, "Robotic Futures." Theatre is a form of knowing through doing-by examining the interactions that persisted in the devising process and those that fell through, we conclude with a proposition for design considerations for future agents. Based on emerging research in this space, we focus on the characteristics of personally-owned agents in comparison to shared agents, and consider the roles and functions each introduce in their integration in the home.

The presence of voice activated personal assistants (VAPAs) in people's homes rises each year [31]. Industry efforts are invested in making interactions with VAPAs more personal by leveraging information from messages and calendars, and by accessing user accounts for 3rd party services. However, the use of personal data becomes more complicated in interpersonal spaces, such as people's homes. Should a shared agent access the information of many users? If it does, how should it navigate issues of privacy and control? Designers currently lack guidelines to help them design appropriate agent behaviors. We used Speed Dating to explore inchoate social mores around agent actions within a home, including issues of proactivity, interpersonal conflict, and agent prevarication. Findings offer new insights on how more socially sophisticated agents might sense, make judgements about, and navigate social roles and individuals. We discuss how our findings might impact future research and future agent behaviors.

Artificial Intelligence (AI) plays an increasingly important role in improving HCI and user experience. Yet many challenges persist in designing and innovating valuable human-AI interactions. For example, AI systems can make unpredictable errors, and these errors damage UX and even lead to undesired societal impact. However, HCI routinely grapples with complex technologies and mitigates their unintended consequences. What makes AI different? What makes human-AI interaction appear particularly difficult to design? This paper investigates these questions. We synthesize prior research, our own design and research experience, and our observations when teaching human-AI interaction. We identify two sources of AI's distinctive design challenges: 1) uncertainty surrounding AI's capabilities, 2) AI's output complexity, spanning from simple to adaptive complex. We identify four levels of AI systems. On each level, designers encounter a different subset of the design challenges. We demonstrate how these findings reveal new insights for designers, researchers, and design tool makers in productively addressing the challenges of human-AI interaction going forward.

Digital fabrication tools have transformed how people work in micro- and small-scale manufacturing settings. While increasing efficiency and precision, these tools raise concerns around user agency and control. This paper describes an exploratory study investigating the felt work experience and desired futures of professionals who use fabrication tools. We conducted co-design workshops with 23 professionals who use 3D printers, laser cutters, and CNC routers. We probed about current practices; machine awareness and autonomy; and user agency. Our findings reveal that current tools are not very professional. They are unreliable and untrustworthy. Participants desired smarter tools that can actively prevent errors and perform self-calibration and self-maintenance. They had few concerns that more intelligence would impact agency. They desired tools that could negotiate trade-offs between time, cost, and quality; and that can operate as super-human shop assistants. We discuss the implications of these findings as opportunities for research that can improve professionals' work experience.

Service robots often perform their main functions in public settings, interacting with more than one person at a time. How these robots should handle the affairs of individual users while also behaving appropriately when others are present is an open question. One option is to design for flexible agent embodiment: letting agents take control of different robots as people move between contexts. Through structured User Enactments, we explored how agents embodied within a single robot might interact with multiple people. Participants interacted with a robot embodied by a singular service agent, agents that re-embody in different robots and devices, and agents that co-embody within the same robot. Findings reveal key insights about the promise of re-embodiment and co-embodiment as design paradigms as well as what people value during interactions with service robots that use personalization.

Personalization of user experience has a long history of success in the HCI community. More recently the community has focused on adaptive user interfaces, supported by machine learning, that reduce interaction efforts and improves user experience by collapsing transactions and pre-filtering results. However, generally, these more recent results have only been demonstrated in the laboratory environment. In this paper, we share the case of a deployed mobile transit app that adapts based on users’ previous usage. We examine the impact of adaptation, both good and bad, and user abandonment rates. We conducted an 18-month assessment where 2,616 participants (with and without vision impairments) were recruited and participated in an A/B study. Finally, we draw some insights on some unusual effects that appear over the long term.

Interactions with multiple conversational agents and social robots are becoming increasingly common. This raises new design challenges: Should agents and robots be modeled after humans, presenting their entity (i.e., social presence) as bound to a single body, or should they take advantage of non-human capabilities, such as moving their social presence from body to body across service touchpoints and contexts? We conducted a User Enactments study in which participants interacted with agents that had one social presence per body, that could re-embody (move their social presence from body to body), and that could co-embody (move their social presence into a body that already contains another). Reactions showed that participants felt comfortable with re-embodying agents, who created more seamless and efficient experiences. Yet situations that required expertise or concentration raised concerns about non-human behaviors. We report on our insights regarding collaboration and coordination with several agents in multi-step interactions.

Clinical decision support tools (DST) promise improved healthcare outcomes by offering data-driven insights. While effective in lab settings, almost all DSTs have failed in practice. Empirical research diagnosed poor contextual fit as the cause. This paper describes the design and field evaluation of a radically new form of DST. It automatically generates slides for clinicians' decision meetings with subtly embedded machine prognostics. This design took inspiration from the notion of Unremarkable Computing, that by augmenting the users' routines technology/AI can have significant importance for the users yet remain unobtrusive. Our field evaluation suggests clinicians are more likely to encounter and embrace such a DST. Drawing on their responses, we discuss the importance and intricacies of finding the right level of unremarkableness in DST design, and share lessons learned in prototyping critical AI systems as a situated experience.

Machine learning (ML) plays an increasingly important role in improving a user's experience. However, most UX practitioners face challenges in understanding ML's capabilities or envisioning what it might be. We interviewed 13 designers who had many years of experience designing the UX of ML-enhanced products and services. We probed them to characterize their practices. They shared they do not view themselves as ML experts, nor do they think learning more about ML would make them better designers. Instead, our participants appeared to be the most successful when they engaged in ongoing collaboration with data scientists to help envision what to make and when they embraced a data-centric culture. We discuss the implications of these findings in terms of UX education and as opportunities for additional design research in support of UX designers working with ML.

HCI has become particularly interested in using machine learning (ML) to improve user experience (UX). However, some design researchers claim that there is a lack of design innovation in envisioning how ML might improve UX. We investigate this claim by analyzing 2,494 related HCI research publications. Our review confirmed a lack of research integrating UX and ML. To help span this gap, we mined our corpus to generate a topic landscape, mapping out 7 clusters of ML technical capabilities within HCI. Among them, we identified 3 under-explored clusters that design researchers can dig in and create sensitizing concepts for. To help operationalize these technical design materials, our analysis then identified value channels through which the technical capabilities can provide value for users: self, context, optimal, and utility-capability. The clusters and the value channels collectively mark starting places for envisioning new ways for ML technology to improve people's lives.

Machine learning (ML) is now a fairly established technology, and user experience (UX) designers appear regularly to integrate ML services in new apps, devices, and systems. Interestingly, this technology has not experienced a wealth of design innovation that other technologies have, and this might be because it is a new and difficult design material. To better understand why we have witnessed little design innovation, we conducted a survey of current UX practitioners with regards to how new ML services are envisioned and developed in UX practice. Our survey probed on how ML may or may not have been a part of their UX design education, on how they work to create new things with developers, and on the challenges they have faced working with this material. We use the findings from this survey and our review of related literature to present a series of challenges for UX and interaction design research and education. Finally, we discuss areas where new research and new curriculum might help our community unlock the power of design thinking to re-imagine what ML might be and might do.

Everyday, people generate lots of personal data. Driven by the increasing use of online services and widespread adoption of smartphones (owned by 68% of U.S. residents; Anderson, 2015), personal data take many forms, including communications (e.g., e-mail, SMS, Facebook), plans and coordination (e.g., calendars, TripIt, to-do lists), entertainment consumption (e.g., YouTube, Spotify, Netflix), finances (e.g., banking, Amazon, eBay), activities (e.g., steps, runs, check-ins), and even health care (e.g., doctor visits, medications, heart rate). Collectively, these data provide a highly detailed description of an individual. Personal data afford the opportunity for many new kinds of applications that might improve people’s lives through deep personalization, tools to manage personal well-being, and services that support identity construction. However, developers currently encounter challenges working with personal data due to its fragmentation across services. This article evaluates the landscape of personal data, including the systemic forces that created current fragmented collections of data and the process required for integrating data from across services into an application. It details challenges the fragmented ecosystem imposes. Finally, it contributes Phenom, an experimental system that addresses these challenges, making it easier to develop applications that access personal data and providing users with greater control over how their data are used.

Machine learning improves mobile user experience. Interestingly, envisioning apps with adaptive interfaces that reduce navigation and selection effort is not standard UX practice. When implementing an adaptive UI for our mobile transit app, we encountered a number of problems. Our original design did not log necessary information nor did it induce users to provide good labels. On reflection, we realized UX designers should identify and refine UI adaptions when sketching wireframes. To advance on this insight, we reviewed the interfaces of popular apps and extracted six design patterns where UI adaptation can improve in-app navigation. Next, we designed an exemplar set of wireframes, illustrating how UX designers might annotate their interaction flows to communicate planned adaptation and note the information (logs and labels) needed to make the desired inferences.

Teens' financial literacy, their ability to understand financial concepts and make effective decisions, is an important topic in the US. Educational programs targeted at teens have had little impact. Our study investigated if parents should play a role in teaching their teens. We conducted a literature review on teenage financial literacy and a competitive analysis of current tools. We interviewed parents and teens, probing on what teens know and how they learned these concepts. We conducted speed dating sessions, exposing parents and teens to provocative services meant to trigger conversations about financial values. These included a system where parents gave loans and charged interest and a game where teens guessed the prices of family expenses. Results showed a strong interest for tools that allow parents to educate their teens. Concepts worked best when they addressed teens' current needs and when they allowed parents to influence spending based on values.

The mere existence of digital documents on the web makes those documents more accessible to blind people than if they were not digitized. The web provides several opportunities, however, for web authors to describe what content looks like, rather than what it means, and this visual description inconveniences and confuses people using screen readers. We focus on the particular instance of this problem where information that is logically tabular is presented using repeated visual templates. We recruited a group of experienced screen reader users and observed them performing a series of tasks designed to elicit tabular behavior. We find that people who make use of tabular operations perform the tasks faster and more successfully than those who do not. We furthermore find that while many users discover the tables we introduced as part of the study, many others do not or do not choose to make use of the tables. We discuss some techniques for enhancing the discoverability of the introduced tables.

People accumulate huge assortments of virtual possessions, but it is not yet clear how systems and system designers can help people make meaning from these large archives. Early research in HCI has suggested that people generally appear to value their virtual things less than their material things, but theory on material possessions does not entirely explain this difference. To investigate if changes to the form and behavior of virtual things may surface valued elements of a virtual archive, we designed a technology probe that selected snippets from old emails and mailed them as physical postcards to participating households. The probe uncovered features of emails that trigger meaningful reflection, and how contextual information can help people engage in reminiscence. Our study revealed insights about how materializing virtual possessions influences factors shaping how people draw on, understand, and value those possessions. We conclude with implication and strategies for aimed at supporting people in having more meaningful interactions and experiences with their virtual possessions.

Clinical decision support tools (DSTs) are computational systems that aid healthcare decision-making. While effective in labs, almost all these systems failed when they moved into clinical practice. Healthcare researchers speculated it is most likely due to a lack of user-centered HCI considerations in the design of these systems. This paper describes a field study investigating how clinicians make a heart pump implant decision with a focus on how to best integrate an intelligent DST into their work process. Our findings reveal a lack of perceived need for and trust of machine intelligence, as well as many barriers to computer use at the point of clinical decision-making. These findings suggest an alternative perspective to the traditional use models, in which clinicians engage with DSTs at the point of making a decision. We identify situations across patients' healthcare trajectories when decision supports would help, and we discuss new forms it might take in these situations.

Participatory sensing systems use people and their smartphones as a sensing infrastructure, and getting people to make contributions remains a critical challenge. Little work details how system designers should combine different interactions to increase coverage of service location. Tiramisu, a participatory sensing system, invites transit riders to crowdsource real-time arrival information by sharing location traces when they commute. We extended this system with a new feature that allows riders at stops to "spot" buses passing by. To better understand the impact of this new feature, we conducted an observational log analysis, examining changes in coverage and user behavior before and after the new feature. Following the addition of the spotting feature, participants' contributions increased coverage (the number of trips with real-time data) by 98%, and they used the app more than twice as much. The addition of the spotting feature was also followed by a significant increase of trace contributions.

People increasingly live their lives online, accruing large collections of digital possessions, which symbolically represent important relationships, events, and activities. Most HCI research on bereavement focuses onretainingthese significant digital possessions to honor the departed. However, recent work suggests that significant digital possessions may complicate moving on; they function as both comforting and painful reminders but currently provide inflexible methods for disposal. A few works have investigated the disposal of digital objects as a means ofletting go. To better understand this we interviewed 10 psychotherapists who employ rituals of letting go to help patients overcome loss in situations such as a divorce, a breakup, or a stillbirth. Patients disposed of either natural artifacts or symbolic personal possessions through actions such as burning, burying, or placing in a body of water. Therapists noted that people increasingly have digital possessions, and that the act of deletion does not offer the same cathartic sense of release as disposal of material artifacts. Based on the analysis of this grief therapy, we propose a new conceptual framework for rituals of letting go that highlights temporality, visibility, and force. It provides a vocabulary to talk about disposal. We then offer design implications connecting the rituals of letting go to the disposal of digital things. Based on our interviews and analytic framework, we propose novel technologies that better connect the embodied nature of letting go rituals to the process of digital disposal.

Today, many data-driven web pages present information in a way that is difficult for blind and low vision users to navigate and to understand. EnTable addresses this challenge. It re-writes confusing and complicated template-based data sets as accessible tables. EnTable allows blind and low vision users to submit requests for pages they wish to access. The system then employs sighted informants to markup the desired page with semantic information, allowing the page to be re-written using straightforward tags. Screen reader users who browse the web using the EnTable browser extension can report data sets that are confusing, and utilize data sets re-written with the tag based on their own requests or on the requests of other users.

How effective are call and SMS logs in modeling tie strength? Frequency and duration of communication has long been cited as a major aspect of tie strength. Intuitively, this makes sense: people communicate with those that they feel close to. Highly cited research papers have pushed this idea further, using communication as a direct proxy for tie strength. However, this operationalization has not been validated. Our work evaluates this assumption. We collected call and SMS logs and ground truth relationship data from 36 participants. Consistent with theory, we found that frequent or long-duration communication likely indicates a strong tie. However, the use of call and SMS logs produced many errors in separating strong and weak ties, suggesting this approach is incomplete. Follow-up interviews indicate fundamental challenges for inferring tie strength from communication logs.

People worldwide are increasingly acquiring virtual possessions. While virtual possessions have become ubiquitous, little work exists on how people value them, and how their experiences of them differ from material possessions. In this paper, we reflect on and synthesize findings from five studies we conducted over the past five years that investigated people's perceptions of and practices with virtual possessions. Through the higher-level perspective we adopt, we propose three thematic qualities that help characterize people's experiences with virtual possessions, as compared to their material things: placelessness, spacelessness, and formlessness. We draw on these proposed qualities as lenses to help frame future research and practice opportunities for better supporting value construction activities with virtual possessions.

We have developed User Enactments to help support design teams in more successfully investigating radical alterations to technologies' roles, forms and behaviors in uncharted design spaces. To date, no work exists that explicitly unpacks the practical development of a cohesive set of user enactments. Interest is growing in the method with its recent inclusion in a popular design method handbook for practitioners and also as it becomes integrated into graduate-level university curriculum in interaction design. The contribution of this Pictorial is to unpack the thinking and making behind a set of user enactments through visual documentation and annotations.

The rapid adoption of smartphones along with a growing habit for using these devices as alarm clocks presents an opportunity to use this device as a sleep detector. This adds value to UbiComp and personal informatics in terms of user context and new performance data to collect and visualize, and it benefits healthcare as sleep is correlated with many health issues. To assess this opportunity, we collected one month of phone sensor and sleep diary entries from 27 people who have a variety of sleep contexts. We used this data to construct models that detect sleep and wake states, daily sleep quality, and global sleep quality. Our system classifies sleep state with 93.06% accuracy, daily sleep quality with 83.97% accuracy, and overall sleep quality with 81.48% accuracy. Individual models performed better than generally trained models, where the individual models require 3 days of ground truth data and 3 weeks of ground truth data to perform well on detecting sleep and sleep quality, respectively. Finally, the features of noise and movement were useful to infer sleep quality.

We describe the design, implementation and deployment of Photobox, a domestic technology that prints four or five randomly selected photos from the owner's Flickr collection at random intervals each month. We deployed Photobox in three homes for fourteen months, to explore how the slow pace at which it operates could support experiences of anticipation and re-visitation of the past. Findings reveal changes in attitude toward the device, from frustration to eventual acceptance. Participants drew on the photos to reflect on past life events and reactions indicated a renewed interest for their Flickr collection. Photobox also provoked reflection on technology in and around the home. These findings suggest several opportunities, such as designing for anticipation, better supporting reflection on the past, and, more generally, expanding the slow technology research program within the HCI community.

A central tenet of HCI is that technology should be user-centric, with designs being based around social science findings about users. Nevertheless a repeated but critical challenge in design is translating empirical findings into actionable ideas that inform design, or generating implications for design. Despite various design methods aiming to bridge this gap, knowledge informing design is still seen as problematic. However there has been little empirical exploration into what design researchers understand by such design knowledge, the functions and principles behind their creation. We report on interviews with twelve expert HCI design researchers probing the roles and types of design implications, and the process of generating and evaluating them. We synthesize different types of design implications into a framework to guide their generation. Our findings identify a broader range than previously described, additional sources and heuristics supporting their development as well some important evaluation criteria. We discuss the value of these findings for interaction design research.

Participatory sensing systems (PSS) require frequent injection of information that has a short shelf-life. The use of crowds to gather information for PSS is therefore particularly challenging. In this study, we explore the impact of two policies on user contributions. A quid-pro-quo policy exchanges contributions from users for access to critical information in the system. A request policy simply reminds the user that information is needed to make the system function well. Prior research has shown that request for help in crowdsourced system is an effective mechanism to increase contributions. During a large-scale experimental study within a publicly deployed, crowdsourced, transit information system, we analyzed metrics associated with frequency of contribution and commitment to long-term use over a 10-month period. Our results confirmed that quid-pro-quo led to more contribution, but at a cost of faster departure from the study. When a participant was simply requested to contribute, but could still access community-generated data if they ignored a request, was largely ineffective and was statistically similar to the control condition where no request for contribution occurred. Thus crowdsource system designers should consider imposing quid-pro-quo type policies for PSS that concentrate on fewer users, but makes them more productive.

The smartphone contact list has the potential to be a valuable source of data about personal relationships. To understand how we might data mine the information that people store in their contact lists, we collected the contact lists of 54 participants. Initially we found that the majority of contact list features were unused. However, a further examination of the "name" field revealed a broad variety of contact-naming behaviors. We observed contact "name" fields that included affiliations, relationship role labels, multiple names, phone types, and references to companies/services/places. People's appropriation and usage of contact lists have implications for automated attempts to merge or mine contact lists that assume people use the features and structure of the contact list tool as intended. They also offer new opportunities for data mining to better describe relationships between users and their contacts.

Although advances in technology now enable people to communicate ‘anytime, anyplace’, it is not clear how citizens can be motivated to actually do so. This paper evaluates the impact of three principles of psychological empowerment, namely perceived self-efficacy, sense of community and causal importance, on public transport passengers’ motivation to report issues and complaints while on the move. A week-long study with 65 participants revealed that self-efficacy and causal importance increased participation in short bursts and increased perceptions of service quality over longer periods. Finally, we discuss the implications of these findings for citizen participation projects and reflect on design opportunities for mobile technologies that motivate citizen participation.

People worldwide are increasingly acquiring collections of virtual possessions. While virtual possessions have become ubiquitous, little work exists on how people value and form attachments to these things. To investigate, we conducted a study with 48 young adults from South Korea, Spain and the United States. The study probed on participants' perceived value of their virtual possessions as compared to their material things, and the comparative similarities and differences across cultures. Findings show that young adults live in unfinished spaces and that they often experience a sense of fragmentation when trying to integrate their virtual possessions into their lives. These findings point to several design opportunities, such as tools for life story-oriented archiving, and insights on better forms of Cloud storage.

Social computing provides a new way for citizens to engage with their public service. Our research investigates how social computing might support citizens co-design their transit service. We conducted a field study with public transit riders, exploring the issues and controversies that reveal conflicting communities. Our analyses revealed three insights. First, encourage citizens to share what they see as the rationale for current service offerings. Second, encourage citizens to share the consequences of current services and of proposed changes and new designs. Third, focus on producing a shared citizen and service provider understanding of what the goals and mission of the public service should be.

Paid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd work will fail to achieve its potential, focusing on assembly-line piecework. Can we foresee a future crowd workplace in which we would want our children to participate? This paper frames the major challenges that stand in the way of this goal. Drawing on theory from organizational behavior and distributed computing, as well as direct feedback from workers, we outline a framework that will enable crowd work that is complex, collaborative, and sustainable. The framework lays out research challenges in twelve major areas: workflow, task assignment, hierarchy, real-time response, synchronous collaboration, quality control, crowds guiding AIs, AIs guiding crowds, platforms, job design, reputation, and motivation.

People engage with many overlapping social networks and enact diverse social roles across different facets of their lives. Unfortunately, many online social networking services reduce most people's contacts to "friend". A richer computational model of relationships would be useful for a number of applications such as managing privacy settings and organizing communications. In this paper, we take a step towards a richer computational model by using call and text message logs from mobile phones to classifying contacts according to life facet (family, work, and social). We extract various features such as communication intensity, regularity, medium, and temporal tendency, and classify the relationships using machine-learning techniques. Our experimental results on 40 users showed that we could classify life facets with up to 90.5% accuracy. The most relevant features include call duration, channel selection, and time of day for the communication.

Constructive design research is a form of research where design activity is a central research activity. One type of constructive design research is critical design, which seeks to disrupt or transgress social and cultural norms. Critical design's advocates have turned to critical theory as an intellectual resource to support their approach. Interestingly, critical design processes remain under-articulated in the growing design research literature. In this paper, we first explain why critical design is so hard to describe as a design practice or process. We then describe two critical design case studies we undertook and the effects we observed them having when place in the field. After sharing our breakdowns and breakthroughs along the way, we offer reflections on designing for provocativeness, the value of deep relationships between researchers and research participants, and the need to plan for and go with a fluid and emergent research plan --- with the goal of helping clarify critical design as an approach.

Designing radically new technology systems that people will want to use is complex. Design teams must draw on knowledge related to people's current values and desires to envision a preferred yet plausible future. However, the introduction of new technology can shape people's values and practices, and what-we-know-now about them does not always translate to an effective guess of what the future could, or should, be. New products and systems typically exist outside of current understandings of technology and use paradigms; they often have few interaction and social conventions to guide the design process, making efforts to pursue them complex and risky. User Enactments (UEs) have been developed as a design approach that aids design teams in more successfully investigate radical alterations to technologies' roles, forms, and behaviors in uncharted design spaces. In this paper, we reflect on our repeated use of UE over the past five years to unpack lessons learned and further specify how and when to use it. We conclude with a reflection on how UE can function as a boundary object and implications for future work.

Over the past several years, people have acquired more and more virtual possessions. While virtual possessions have become ubiquitous, little work exists to inform designers on how these growing collections should be displayed and how they should behave. We generated four design concepts that changed the form and behavior of these digital things, making them more present within a teen bedroom. We then conducted speed dating sessions to investigate how these new forms and behaviors influence perceptions of value. Sessions revealed how new technologies might better support self-exploration and reflection, as well as how they could complicate identity construction processes. Findings are interpreted to detail opportunities and tensions that can guide future research and practice in this emerging space.

Here we present findings of a needs validation study of 18-25 year old women and their wardrobes. Based on their feedback, we developed a prototype system that allows for borrowing and sharing of information about clothing via Facebook. The novelty of our interface lies in its combining RFID tags and social networking. In doing so we bridge the material and virtual realms and demonstrate the importance of material culture for ubiquitous computing.

Part of being a parent is taking responsibility for arranging and supplying transportation of children between various events. Dual-income parents frequently develop routines to help manage transportation with a minimal amount of attention. On days when families deviate from their routines, effective logistics can often depend on knowledge of the routine location, availability and intentions of other family members. Since most families rarely document their routine activities, making that needed information unavailable, coordination breakdowns are much more likely to occur. To address this problem we demonstrate the feasibility of learning family routines using mobile phone GPS. We describe how we (1) detect pick-ups and drop-offs; (2) predict which parent will perform a future pick-up or drop-off; and (3) infer if a child will be left at an activity. We discuss how these routine models give digital calendars, reminder and location systems new capabilities to help prevent breakdowns, and improve family life.

There have been many location sharing systems developed over the past two decades, and only recently have they started to be adopted by consumers. In this paper, we present the results of three studies focusing on the foursquare check-in system. We conducted interviews and two surveys to understand, both qualitatively and quantitatively, how and why people use location sharing applications, as well as how they manage their privacy. We also document surprising uses of foursquare, and discuss implications for design of mobile social services.

Over the past several years, people have increasingly acquired virtual possessions. We consider these things to include artifacts that are increasingly becoming immaterial (e.g. books, photos, music, movies) and things that have never traditionally had a lasting material form (e.g. SMS archives, social networking profiles, personal behavior logs). To date, little research exists about how people value and form attachments to virtual possessions. To investigate, we conducted a study with 21 teenagers exploring the perceived value of their virtual possessions, and the comparative similarities and differences with their material things. Findings are interpreted to detail design and research opportunities and issues in this emerging space.

Crowd-sourcing social computing systems represent a new material for HCI designers. However, these systems are difficult to work with and to prototype, because they require a critical mass of participants to investigate social behavior. Service design is an emerging research area that focuses on how customers co-produce the services that they use, and thus it appears to be a great domain to apply this new material. To investigate this relationship, we developed Tiramisu, a transit information system where commuters share GPS traces and submit problem reports. Tiramisu processes incoming traces and generates real-time arrival time predictions for buses. We conducted a field trial with 28 participants. In this paper we report on the results and reflect on the use of field trials to evaluate crowd-sourcing prototypes and on how crowd sourcing can generate co-production between citizens and public services.

The UbiComp research community has a long history of interdisciplinary research on technology, engineering, and human behavior. But, one disciplinary voice that often seems to be missing is design. Due to the value of design to integrate knowledge and insights from across disciplines and focus on the future world, it seems that design might play a critical role in helping UbiComp research through its holistic approach. China is the world leader in manufacturing, and it is almost never the place where the innovative, high-tech products and services are designed. With UbiComp in China, now seems the perfect time to discuss bridging east and west and bringing design to UbiComp. This workshop aims to provide a disciplinary forum for researchers and experts from design computer science, psychology and anthropology to exchange ideas on the issue of collaboration. It will build understanding across disciplines and explore the potential opportunity for design in UbiComp research and practice.

As ubiquitous computing becomes increasingly mobile and social, personal information sharing will likely increase in frequency, the variety of friends to share with, and range of information that can be shared. Past work has identified that whom you share with is important for choosing whether or not to share, but little work has explored which features of interpersonal relationships influence sharing. We present the results of a study of 42 participants, who self-report aspects of their relationships with 70 of their friends, including frequency of collocation and communication, closeness, and social group. Participants rated their willingness to share in 21 different scenarios based on information a UbiComp system could provide. Our findings show that (a) self-reported closeness is the strongest indicator of willingness to share, (b) individuals are more likely to share in scenarios with common information (e.g. we are within one mile of each other) than other kinds of scenarios (e.g. my location wherever I am), and (c) frequency of communication predicts both closeness and willingness to share better than frequency of collocation.

While the HCI community has long investigated issues of designing for family and the home, very little attention has focused on the lives of divorced families and the ways in which interactive systems might be better designed to address the very real and growing issues they face. In this paper we present an overview of related research on divorce and families. We then report field evidence from 13 in depth interviews conducted with families of parents and children in joint custody situations, and unpack key emergent problems and tensions. We conclude with a discussion of the design implications and opportunities that give shape to how the HCI community may be able to have a positive effect on this set of potential users. The overarching goal of this research is to better understand how the HCI community might begin to approach designing for this alternative family.

For more than forty years, researchers have detailed how people develop attachments to their material possessions as they create and evolve a sense of self. Over the past several years people have increasingly acquired virtual possessions. These include both possessions that are losing their material integrity (books, photos, music, movies) as well as things that have never had material form (e.g. email archives, social networking profiles, personal behavior logs). However, little is known about how people perceive, value, and form attachments to their virtual possessions. To investigate, we conducted a study with teens exploring their virtual possessions. Preliminary findings reveal three key themes and suggest emerging interaction design opportunities for new forms for people's virtual things.

The field of HCI is experiencing a growing interest in Research through Design (RtD), a research approach that employs methods and processes from design practice as a legitimate method of inquiry. We are interested in expanding and formalizing this research approach, and understanding how knowledge, or theory, is generated from this type of design research. We conducted interviews with 12 leading HCI design researchers, asking them about design research, design theory, and RtD specifically. They were easily able to identify different types of design research and design theory from contemporary and historical design research efforts, and believed that RtD might be one of the most important contributions of design researchers to the larger research community. We further examined three historical RtD projects that were repeatedly mentioned in the interviews, and performed a critique of current RtD practices within the HCI research and interaction design communities. While our critique summarizes the problems, it also shows possible directions for further developments and refinements of the approach.

Researchers have detailed the importance of routines in how people live and work, while also cautioning system designers about the importance of people's idiosyncratic behavior patterns and the challenges they would present to learning systems. We wish to take up their challenge, and offer a vision of how simple sensing technology could capture and model idiosyncratic routines, enabling applications to solve many real world problems.

When designing novel GUI controls, interaction designers are challenged by the "immaterial" materiality of the digital domain; they lack tools that effectively support a reflecting conversation with the material of software as they attempt to conceive, refine, and communicate their ideas. To investigate this situation, we conducted two participatory design workshops. In the first workshop, focused on conceiving, we observed that designers want to invent controls by exploring gestures, context, and examples. In the second workshop, on refining and communicating, designers proposed tools that could refine movement, document context through usage scenarios, and support the use of examples. In this workshop they struggled to effectively communicate their ideas for developers because their ideas had not been fully explored. In reflecting on this struggle, we began to see an opportunity for the output of a design tool to be a boundary object that would allow for an ongoing conversation between the design and the material of software, in which the developer acts as a mediator for software.

As long as people have traveled, they have constructed bags to help them carry more items than their hands will hold. While quite effective at keeping things together, bags do a poor job of communicating when something is missing. We propose that there exists an opportunity for the HCI community to improve the quality of people's lives by creating bags that have knowledge of people's schedules and equipment needs, can sense their contents, and can communicate when something has been forgotten. To investigate this opportunity, we conducted a field study with six dual-income families. Through interviews and observations we investigated their experiences using bags to organize equipment needed for children's enrichment activities. Based on the findings we generated 100 concepts and conducted a needs validation session to better understand the best opportunity to improve people's lives with technical intervention. This paper reports on our field study and needs validation session, and shares insights on the opportunities and implications of a smart activity bag.

The recent advances in web 2.0 technologies and the rapid adoption of smart phones raises many opportunities for public services to improve their services by engaging their users (who are also owners of the service) in co-design: a dialog where users help design the services they use. To investigate this opportunity, we began a service design project investigating how to create repeated information exchanges between riders and a transit agency in order to create a virtual "place" from which the dialog on services could take place. Through interviews with riders, a workshop with a transit agency, and speed dating of design concepts, we have developed a design direction. Specifically, we propose a service that combines vehicle location and "fullness" ratings provided by riders with dynamic route change information from the transit agency as a foundation for a dialog around riders conveying input for continuous service improvement.

RADAR is a multiagent system with a mixed-initiative user interface designed to help office workers cope with email overload. RADAR agents observe experts to learn models of their strategies and then use the models to assist other people who are working on similar tasks. The agents' assistance helps a person to transition from the normal email-centric workflow to a more efficient task-centric workflow. The Email Classifier learns to identify tasks contained within emails and then inspects new emails for similar tasks. A novel task-management user interface displays the found tasks in a to-do list, which has integrated support for performing the tasks. The Multitask Coordination Assistant learns a model of the order in which experts perform tasks and then suggests a schedule to other people who are working on similar tasks. A novel Progress Bar displays the suggested schedule of incomplete tasks as well as the completed tasks. A large evaluation demonstrated that novice users confronted with an email overload test performed significantly better (a 37% better overall score with a factor of four fewer errors) when assisted by the RADAR agents.

A wealth of studies in the field of user experience have tried to conceptualize new measures of product quality and inquire into how the overall goodness of a product is formed on the basis of product quality perceptions. An interesting question relates to how the perception as well as the relative dominance of different product qualities evolve across different phases in the adoption of a product. However, temporality of experience poses substantial challenges to traditional reductive evaluation approaches. In this paper we present an alternative methodological approach for studying how users’ experiences with interactive products develop over time. The approach lies in the elicitation of rich qualitative insights in the form of experience narratives, combined with content-analytical approaches for the aggregation of idiosyncratic insights into generalized knowledge. We describe a tool designed for eliciting rich experience narratives retrospectively, and illustrate this tool by means of a study that inquired into how users’ experiences with mobile phones change over the first 6 months of use. We use the insights of the study to validate and extend a framework of temporality proposed by Karapanos et al. (2009b).

A recent trend in Human-Computer Interaction (HCI) research addresses human needs that go beyond the instrumental, resulting in an increasing body of knowledge about how users form overall evaluative judgments on the quality of interactive products. An aspect largely neglected so far is that of temporality, i.e. how the quality of users' experience develops over time. This paper presents an in-depth, five-week ethnographic study that followed 6 individuals during an actual purchase of the Apple iPhone". We found prolonged use to be motivated by different qualities than the ones that provided positive initial experiences. Overall, while early experiences seemed to relate mostly to hedonic aspects of product use, prolonged experiences became increasingly more tied to aspects reflecting how the product becomes meaningful in one's life. Based on the findings, we promote three directions for CHI practice: designing for meaningful mediation, designing for daily rituals, and designing for the self.

Product attachment theory describes how people learn to love certain possessions through a process of meaning making. It provides a rich and as yet untapped source of inspiration for driving the practice of experience design. However, there are currently no guidelines that describe how to apply this theory in design practice. Taking a research through design approach, I made many different products with the goal of helping people become the person they desire to be through their product interactions. Then, in order to better understand how the different design teams applied attachment theory, I created a set of design patterns that document the application of product attachment theory to the interaction design of each product. I clustered the patterns based on similarities across the different artifacts, and this produced six framing constructs, which work as specific perspectives designers can take when applying product attachment theory in an experience design project.

A key challenge for mixed-initiative systems is to create a shared understanding of the task between human and agent. To address this challenge, we created a mixed-initiative interface called Mixer to aid administrators with automating tedious information-retrieval tasks. Users initiate communication with the agent by constructing a form, creating a structure to hold the information they require and to show context in order to interpret this information. They then populate the form with the desired results, demonstrating to the agent the steps required to retrieve the information. This method of form creation explicitly defines the shared understanding between human and agent. An evaluation of the interface shows that administrators can effectively create forms to communicate with the agent, that they are likely to accept this technology in their work environment, and that the agent's help can significantly reduce the time they spend on repeated information-retrieval tasks.

Presents the introductory welcome message from the conference proceedings.

For years the HCI community has struggled to integrate design in research and practice. While design has gained a strong foothold in practice, it has had much less impact on the HCI research community. In this paper we propose a new model for interaction design research within HCI. Following a research through design approach, designers produce novel integrations of HCI research in an attempt to make the right thing: a product that transforms the world from its current state to a preferred state. This model allows interaction designers to make research contributions based on their strength in addressing under-constrained problems. To formalize this model, we provide a set of four lenses for evaluating the research contribution and a set of three examples to illustrate the benefits of this type of research.

Today many workers spend too much of their time translating their co-workers' requests into structures that information systems can understand. This paper presents the novel interaction design and evaluation of VIO, an agent that helps workers trans late request. VIO monitors requests and makes suggestions to speed up the translation. VIO allows users to quickly correct agent errors. These corrections are used to improve agent performance as it learns to automate work. Our evaluations demonstrate that this type of agent can significantly reduce task completion time, freeing workers from mundane tasks.

Users in an organization frequently request help by sending request messages to assistants that express information intent: an intention to update data in an information system. Human assistants spend a significant amount of time and effort processing these requests. For example, human resource assistants process requests to update personnel records, and executive assistants process requests to schedule conference rooms or to make travel reservations. To process the intent of a request, assistants read the request and then locate, complete, and submit a form that corresponds to the expressed intent. Automatically or semi-automatically processing the intent expressed in a request on behalf of an assistant would ease the mundane and repetitive nature of this kind of work.For a well-understood domain, a straightforward application of natural language processing techniques can be used to build an intelligent form interface to semi-automatically process information intent request messages. However, high performance parsers are based on machine learning algorithms that require a large corpus of examples that have been labeled by an expert. The generation of a labeled corpus of requests is a major barrier to the construction of a parser. In this paper, we investigate the construction of a natural language processing system and an intelligent form system that observes an assistant processing requests. The intelligent form system then generates a labeled training corpus by interpreting the observations. This paper reports on the measurement of the performance of the machine learning algorithms based on real data. The combination of observations, machine learning and interaction design produces an effective intelligent form interface based on natural language processing.

No abstract available.

Large organizations with sophisticated infrastructures have large form-based systems that manage the interaction between the user community and the infrastructure. In many cases, when a user needs to complete a form to accomplish a task, the user e-mails a description of the task to the appropriate form expert. In many cases this description is incomplete and the expert engages in a clarification dialog to determine the details of the task. Since many tasks and descriptions are routine, this e-mail dialog can be replaced with an intelligent user interface. The interface proactively reads e-mail (or IM) messages and assists the user in completing the associated task without involving the expert. To ground our vision in a specific application, we have built an agent that functions as a webmaster assistant. For example, a user emails the request: "Change John Doe's home phone number to 800-555-1212" to the agent. The webmaster agent then replies with the biographical data form displaying information about John Doe with the new phone number pre-filled in the form. The user then simply approves the change.In this paper we describe a prototype website maintenance agent that (i) allows users to express the updates they want to make in human terms (free text input expression of intent), and (ii) allows users to quickly repair any inference errors the agent makes. In addition, we present the results of a proof of concept study that details how interacting with a webmaster agent that makes inference errors is both more efficient (faster) and more effective (errors made to site) than sending a request to a human webmaster. We conclude the paper with a discussion of the application of our work to any form-based system.

The need for personalized summaries of media content has been driven by the recent and anticipated explosive growth in the media world. In this paper, we present a methodology and a supporting user study for generating user profiles and content features that can be used to automatically create personalized summaries of broadcast television content. We determined a mapping, from users' personality traits measured by commonly available personality tests, to computable video features that such personality traits appear to prefer. Three common personality profiles (Myers-Briggs, Merrill Reed, and Brain.exe) were elicited from 59 subjects, together with their preferred summary of news, music, and talk show videos. A factor analysis between the personality traits and the features in preferred summaries indicated that only some traits (e.g., gender, extroversion, control orientation, intuitiveness, etc.) and only some features (e.g., faces, reportage, text, chorus, host, etc.) had predictive value. The mapping of personality to feature also differed by genre. However, in general, extroverted users tended to prefer directly experienced content, while introverted users preferred content mediated through analysis. A validation user study is in progress

The ability to summarize and abstract information is an essential part of intelligent behavior in consumer devices. However, even for the emerging area of video content analysis, user preferences on summarization have not been explored. This paper reports on a panel which asked: who, why and when summarization is needed; what information should be summarized; and what forms should summaries take. In particular, we investigated the requirements with sensitivity to user needs, user context, media content, device capabilities, and the methods by which user and environment profiles can be assembled and exploited. We organize our findings as a wish list containing four major user questions, and we report the current and near-term states of the art and the major technical challenges in satisfying them. Our study suggests that user preferences should be derived from explicit user statements and from implicit trends inferred from viewing histories and summary usage.

No abstract available.

Automatic indexing of large collections of multimedia data is important for enabling retrieval functions. Current approaches mostly draw on a single or dual modality of video content analysis. Here we describe a framework for the integration of multimedia content and context information, which generalizes and systematizes current methods. Content information in the visual, audio, and text domains, is described at different levels of granularity and abstraction. Context describes the underlying structural information that can be used to constrain the possible number of interpretations. We introduce a probabilistic framework that combines (a) Bayesian networks that describe both content and context and (b) hierarchical priors that describe the integration of content and context. We present an application that uses this framework to segment and index TV programs. We discuss experimental results on segment classification on six and a half hours of broadcast video. In our experiments we used audio context information. Classification results for financial segments yield 83% and for celebrity segments 89%.

This paper is concerned with the topic of personalized news assembly at the set-top box, based on augmented video. This is video complemented with additional information that is somehow relevant to the semantic video content. We touch upon the technique that is used for video augmentation, which is video subject detection followed by information searches on the subject. The focus of this paper is on subject detection implemented using traditional text analysis tools; video segmentation is based on these results and visual processing. We describe the architecture of such a system and the benefits to the consumer. Further we discuss a preliminary system that shows the viability of the concept.

Currently available personal video recorders find and store whole TV programs. Our system, Video Scouting, not only finds and stores programs; it automatically segments and indexes story segments from the programs according to viewers' profiles. The extracted descriptions serve the viewers' content information requests for program segment selection, e.g., play the three minute interview with Hillary Clinton. To achieve this, the system combines information from the audio, visual, and transcript domains in a probabilistic framework based on Bayesian networks. We describe the overall architecture, a system implementation, and discuss some experimental results.

We describe integrated multimedia processing for Video Scout, a system that segments and indexes TV programs according to their audio, visual, and transcript information. Video Scout represents a future direction for personal video recorders. In addition to using electronic program guide metadata and a user profile, Scout allows the users to request specific topics within a program. For example, users can request the video clip of the USA president speaking from a half-hour news program. Video Scout has three modules: (i) video pre-processing, (ii) segmentation and indexing, and (iii) storage and user interface. Segmentation and indexing, the core of the system, incorporates a Bayesian framework that integrates information from the audio, visual, and transcript (closed captions) domains. This framework uses three layers to process low, mid, and high-level multimedia information. The high-level layer generates semantic information about TV program topics. This paper describes the elements of the system and presents results from running Video Scout on real TV programs.

Current personal Vido recorders make it very easy for consumers to record whole TV programs. Our research however, focuses on personalizing TV at a sub-program level. We use a traditional Content-Based Information Retrieval system architecture consisting of archiving and retrieval modules. The archiving module employs a three-layered, multimodal integration framework to segment, analyze, characterize, and classify segments. The retrieval module relies on users personal preferences to deliver both full programs and video segments of interest. We tested retrieval concepts with real users and discovered that they see more value in segmenting non-narrative programs (e.g. news) than narrative programs (e.g. movies). We benchmarked individual algorithms and segment classification for celebrity and financial segments as instances of non-narrative content. For celebrity segments we obtained a total precision of 94.1% and recall of 85.7%, and for financial segments a total precision of 81.1% and a recall of 86.9%.