Reflection and metacognition are central to the creative user experience. However, most HCI research on reflection focuses on clear, task-oriented goals such as to reflect on personal data or pedagogical outcomes. This contrasts with the open-ended and challenging to articulate goals of creative user experiences. For the first time, this workshop brings together interdisciplinary researchers, designers, educators, and artists across HCI, Cognitive Science, Design, AI, Learning Sciences, and Digital Art to examine reflection in creative interaction. The workshop will discuss themes, drawn from earlier discussions with HCI researchers and artists, on: how best to capture reflection in creative contexts, how to leverage the arts to support reflection for ethical change, and how to design creative AI that enhances – not hinders – critical thinking. By bringing interdisciplinary perspectives on reflection into discussion, the workshop will develop a guiding taxonomy for reflection in creative interaction to inform future creative practice and tool development.

The need to reduce the environmental impact has led IT providers to reconsider how physical resources are managed and assigned to different applications. However, application developers play a relevant role and should design their applications in a context-aware way to adapt to the availability of renewable energy. This generates a trade-off between environmental impact and user experience. In this work, we explore this trade-off in the context of search engines. To this end, we conducted an interview-based study to understand the impact of sustainability awareness and other factors on the final users’ choices.

This paper explores how a library of conversational patterns defined for Web browsing can be adopted to improve the accessibility of prompt interfaces for Large Language Models. It reports preliminary findings from a formative evaluation with blind and visually impaired users. It also discusses how these findings support transferring the benefits of voice-based Web browsing to the inclusivity of prompt-based interfaces.

Artificial intelligence (AI) algorithms are increasingly making decisions and supporting human decisions. Both autonomous systems and algorithm-in-the-loop decision-support systems use AI algorithm...

Widespread adoption of artificial intelligence (AI) technologies is substantially affecting the human condition in ways that are not yet well understood. Negative unintended consequences abound including the perpetuation and exacerbation of societal inequalities and divisions via algorithmic decision making. We present six grand challenges for the scientific community to create AI technologies that are human-centered, that is, ethical, fair, and enhance the human condition. These grand challenges are the result of an international collaboration across academia, industry and government and represent the consensus views of a group of 26 experts in the field of human-centered artificial intelligence (HCAI). In essence, these challenges advocate for a human-centered approach to AI that (1) is centered in human well-being, (2) is designed responsibly, (3) respects privacy, (4) follows human-centered design principles, (5) is subject to appropriate governance and oversight, and (6) interacts with individuals while respecting human’s cognitive capacities. We hope that these challenges and their associated research directions serve as a call for action to conduct research and development in AI that serves as a force multiplier towards more fair, equitable and sustainable societies.

We introduce the Idea Machine, a creativity support tool that leverages large language models (LLMs) to empower people engaged in idea generation tasks. The tool includes a number of affordances that can be used to enable various levels of automation and intelligent support. Each idea entered into the system can be expanded, rewritten, or combined with other ideas or concepts. An idea suggestion mode can also be enabled to make the system proactively suggest ideas.

We study the effect of contextual information obtained from a user’s digital trace on Web search performance. Contextual information is modeled using Dirichlet–Hawkes processes (DHP) and used in augmenting Web search queries. The context is captured by monitoring all naturally occurring user behavior using continuous 24/7 recordings of the screen and associating the context with the queries issued by the users. We report a field study in which 13 participants installed a screen recording and digital activity monitoring system on their laptops for 14 days, resulting in data on all Web search queries and the associated context data. A query augmentation (QAug) model was built to expand the original query with semantically related terms. The effects of context window and source were determined by training context models with temporally varying context windows and varying application sources. The context models were then utilized to re-rank the QAug model. We evaluate the context models by using the Web document rankings of the original query as a control condition compared against various experimental conditions: (1) a search context condition in which the context was sourced from search history; (2) a non-search context condition in which the context was sourced from all interactions excluding search history; (3) a comprehensive context condition in which the context was sourced from both search and non-search histories; and (4) an application-specific condition in which the context was sourced from interaction histories captured on a specific application type. Our results indicated that incorporating more contextual information significantly improved Web search rankings as measured by the positions of the documents on which users clicked in the search result pages. The effects and importance of different context windows and application sources, along with different query types are analyzed, and their impact on Web search performance is discussed.

Everyday digital tasks can highly benefit from systems that recommend the right information to use at the right time. However, existing solutions typically support only specific applications and tasks. In this demo, we showcase EntityBot, a system that captures context across application boundaries and recommends information entities related to the current task. The user’s digital activity is continuously monitored by capturing all content on the computer screen using optical character recognition. This includes all applications and services being used and specific to individuals’ computer usages such as instant messaging, emailing, web browsing, and word processing. A linear model is then applied to detect the user’s task context to retrieve entities such as applications, documents, contact information, and several keywords determining the task. The system has been evaluated with real-world tasks, demonstrating that the recommendation had an impact on the tasks and led to high user satisfaction.

Recommender systems can support everyday digital tasks by retrieving and recommending useful information contextually. This is becoming increasingly relevant in services and operating systems. Previous research often focuses on specific recommendation tasks with data captured from interactions with an individual application. The quality of recommendations is also often evaluated addressing only computational measures of accuracy, without investigating the usefulness of recommendations in realistic tasks. The aim of this work is to synthesize the research in this area through a novel approach by (1) demonstrating comprehensive digital activity monitoring, (2) introducing entity-based computing and interaction, and (3) investigating the previously overlooked usefulness of entity recommendations and their actual impact on user behavior in real tasks. The methodology exploits context from screen frames recorded every 2 seconds to recommend information entities related to the current task. We embodied this methodology in an interactive system and investigated the relevance and influence of the recommended entities in a study with participants resuming their real-world tasks after a 14-day monitoring phase. Results show that the recommendations allowed participants to find more relevant entities than in a control without the system. In addition, the recommended entities were also used in the actual tasks. In the discussion, we reflect on a research agenda for entity recommendation in context, revisiting comprehensive monitoring to include the physical world, considering entities as actionable recommendations, capturing drifting intent and routines, and considering explainability and transparency of recommendations, ethics, and ownership of data.

Web searches often originate from conversations in which people engage before they perform a search. Therefore, conversations can be a valuable source of context with which to support the search process. We investigate whether spoken input from conversations can be used as a context to improve query auto-completion. We model the temporal dynamics of the spoken conversational context preceding queries and use these models to re-rank the query auto-completion suggestions. Data were collected from a controlled experiment and comprised conversations among 12 participant pairs conversing about movies or traveling. Search query logs during the conversations were recorded and temporally associated with the conversations. We compared the effects of spoken conversational input in four conditions: a control condition without contextualization; an experimental condition with the model using search query logs; an experimental condition with the model using spoken conversational input; and an experimental condition with the model using both search query logs and spoken conversational input. We show the advantage of combining the spoken conversational context with the Web-search context for improved retrieval performance. Our results suggest that spoken conversations provide a rich context for supporting information searches beyond current user-modeling approaches.

In this paper we present the results of two experimental evaluations carried out in the broader context of a research project on Industry 4.0. We envision in large displays operated through gestural interaction one of the key technologies for enabling real-time management of production processes and supporting decision-making. We developed prototype applications with the twofold aim of: (1) designing a vocabulary of one-handed, touchless gestures meant for interaction in a CAD-like environment, and (2) investigating the role of the cursor feedback. Our results provide a basis for the development of a new integrated wearable sensor which will be assessed with industry workers and in a more realistic setting, for better ecological validity.

Conversations among people involve solving disputes, building common ground, and reinforce mutual beliefs and assumptions. Conversations often require external information that can support these human activities. In this paper, we study how a spoken conversation can be supported by a proactive search agent that listens to the conversation, detects entities mentioned in the conversation, and proactively retrieves and presents information related to the conversation. A total of 24 participants (12 pairs) were involved in informal conversations, using either the proactive search agent or a control condition that did not support conversational analysis or proactive information retrieval. Data comprising transcripts, interaction logs, questionnaires, and interviews indicated that the proactive search agent effectively augmented the conversations, affected the conversations' topical structure, and reduced the need for explicit search activity. The findings also revealed key challenges in the design of proactive search systems that assist people in natural conversations.

While ordinary touchscreen-based interfaces on urban explorer applications draw much of a user's attention onto the screen, visual and audio-haptic augmented reality interfaces have emerged as the two main streams for enabling direct focus on the surroundings. However, neither interface alone satisfies users in the highly dynamic urban environment. This research investigates how the two complementary augmentation can coexist on one system and how people adapt to the situation by selecting the more suitable interface. A prototype was deployed in a field experiment in which participants explored points of interest in an urban environment with both interfaces. The engagement with the surroundings was compared with a touchscreen-based application. Most participants spontaneously switched between the two interfaces, which manifests the value of the availability of both interfaces on one system. The results point at the situated advantages of either interface and reveal the users' preferences when both interfaces are available.

Collaborative and co-located information access is becoming increasingly common. However, fairly little attention has been devoted to the design of ubiquitous computing approaches for spontaneous exploration of large information spaces enabling co-located collaboration. We investigate whether an entity-based user interface provides a solution to support co-located search on heterogeneous devices. We present the design and implementation of QueryTogether, a multi-device collaborative search tool through which entities such as people, documents, and keywords can be used to compose queries that can be shared to a public screen or specific users with easy touch enabled interaction. We conducted mixed-methods user experiments with twenty seven participants (nine groups of three people), to compare the collaborative search with QueryTogether to a baseline adopting established search and collaboration interfaces. Results show that QueryTogether led to more balanced contribution and search engagement. While the overall s-recall in search was similar, in the QueryTogether condition participants found most of the relevant results earlier in the tasks, and for more than half of the queries avoided text entry by manipulating recommended entities. The video analysis demonstrated a more consistent common ground through increased attention to the common screen, and more transitions between collaboration styles. Therefore, this provided a better fit for the spontaneity of ubiquitous scenarios. QueryTogether and the corresponding study demonstrate the importance of entity based interfaces to improve collaboration by facilitating balanced participation, flexibility of collaboration styles and social processing of search entities across conversation and devices. The findings promote a vision of collaborative search support in spontaneous and ubiquitous multi-device settings, and better linking of conversation objects to searchable entities.

Online crowds can help infuse creativity into the design process, but traditional strategies for leveraging them, such as large-scale ideation platforms, require time and organizational effort in order to obtain results. We propose a new method for crowd-based ideation that simplifies the process by having smaller crowds join in-person ideators during synchronous creative sessions. Our system Crowdboard allows online crowds to provide real-time creative input during early-stage design activities, such as brainstorming or concept mapping. The system enables in-person ideators to develop ideas on a physical or digital whiteboard which is augmented with real-time creative input from online participants who see and hear a live broadcast of the meeting. We validate Crowdboard via two user studies in which dyads of in-person ideators brainstormed with the help of crowd ideators. Our studies suggest that Crowdboard can effectively enhance ongoing brainstorming sessions, but also revealed key challenges for how to better facilitate interactions among in-person and crowd ideators.

We present visual re-ranking, an interactive visualization technique for multi-aspect information retrieval. In multi-aspect search, the information need of the user consists of more than one aspect or query simultaneously. While visualization and interactive search user interface techniques for improving user interpretation of search results have been proposed, the current research lacks understanding on how useful these are for the user: whether they lead to quantifiable benefits in perceiving the result space and allow faster, and more precise retrieval. Our technique visualizes relevance and document density on a two-dimensional map with respect to the query phrases. Pointing to a location on the map specifies a weight distribution of the relevance to each of the query phrases, according to which search results are re-ranked. User experiments compared our technique to a uni-dimensional search interface with typed query and ranked result list, in perception and retrieval tasks. Visual re-ranking yielded improved accuracy in perception, higher precision in retrieval and overall faster task execution. Our findings demonstrate the utility of visual re-ranking, and can help designing search user interfaces that support multi-aspect search.

Surface computing allows flexible search interaction where users can manipulate the representation of entities recommended for them to create new queries or augment existing queries by taking advantage of increased screen estate and almost physical tactile interaction. We demonstrate a search system based on 1) Direct Manipulation of Entity Representation on Surfaces and 2) Entity Recommendation and Document Retrieval. Entities are modeled as a knowledge-graph and the relevances of entities are computed using the graph structure. Users can manipulate the representation of entities via spatial grouping and assigning preferences on entities. Our contribution can help to design effective information exploration systems that take advantage of large surfaces.

Collaborative idea generation leverages social interactions and knowledge sharing to spark diverse associations and produce creative ideas. Information exploration systems expand the current context by suggesting novel but related concepts. In this paper we introduce InspirationWall, an unobtrusive display that leverages speech recognition and information exploration to enhance an ongoing idea generation session with automatically retrieved concepts that relate to the conversation. We evaluated the system in six idea generation sessions of 20 minutes with small groups of two people. Preliminary results suggest that InspirationWall contrasts the decay of idea productivity over time and can thus represent an effective way to enhance idea generation activities.

Exploratory search confront users with challenges in expressing search intents as the current search interfaces require investigating result listings to identify search directions, iterative typing, and reformulating queries. We present the design of Exploration Wall, a touch-based search user interface that allows incremental exploration and sense-making of large information spaces by combining entity search, flexible use of result entities as query parameters, and spatial configuration of search streams that are visualized for interaction. Entities can be flexibly reused to modify and create new search streams, and manipulated to inspect their relationships with other entities. Data comprising of task-based experiments comparing Exploration Wall with conventional search user interface indicate that Exploration Wall achieves significantly improved recall for exploratory search tasks while preserving precision. Subjective feedback supports our design choices and indicates improved user satisfaction and engagement. Our findings can help to design user interfaces that can effectively support exploratory search on touch devices.

Locating points of interest (POIs) in cities is typically facilitated by visual aids such as paper maps, brochures, and mobile applications. However, these techniques require visual attention, which ideally should be on the surroundings. Non-visual techniques for navigating towards specific POIs typically lack support for free exploration of the city or more detailed guidance. To overcome these issues, we propose a multimodal, wearable system for alerting the user of nearby recommended POIs. The system, built around a tactile glove, provides audio-tactile cues when a new POI is in the vicinity, and more detailed information and guidance if the user expresses interest in this POI. We evaluated the system in a field study, comparing it to a visual baseline application. The encouraging results show that the glove-based system helps keep the attention on the surroundings and that its performance is on the same level as that of the baseline.

The user's understanding of information needs and the information available in the data collection can evolve during an exploratory search session. Search systems tailored for well-defined narrow search tasks may be suboptimal for exploratory search where the user can sequentially refine the expressions of her information needs and explore alternative search directions. A major challenge for exploratory search systems design is how to support such behavior and expose the user to relevant yet novel information that can be difficult to discover by using conventional query formulation techniques. We introduce IntentStreams, a system for exploratory search that provides interactive query refinement mechanisms and parallel visualization of search streams. The system models each search stream via an intent model allowing rapid user feedback. The user interface allows swift initiation of alternative and parallel search streams by direct manipulation that does not require typing. A study with 13 participants shows that IntentStreams provides better support for branching behavior compared to a conventional search system.

In the area of cultural heritage preservation and fruition the development of electronics and information technologies has opened new scenarios of research in the field of survey, representation and communication of Cultural Heritage. The aim is thus to make the fruition of works of art available to as many users as possible by using survey techniques and multimodal interaction. In this paper we propose a multimodal fruition of a virtual representation of a medieval ceiling, built in the XIV century, which covers the ¿Sala Magna¿ in the Steri of Palermo. The research deals with the design of an intelligent relational agent which interacts with the user in a natural way. To address this very issue a vocal guide has been implemented and wrapped around on the 3D model. This guide assists the user during the visit by means of vocal interaction, so the user can easily navigate the 3D model using voice command. An intelligent relational agent embodied into the system, provides user with a more natural and intelligent way of interaction.