This special issue presents four papers featuring extended versions of full papers originally published at the Conference INTERACT 2023, which was held August 28th to September 1st, 2023, at the University of York, York, UK. We will first present the INTERACT Conference and then the selection process of papers that ultimately led to publication of this special issue in Interacting with Computers. The INTERACT Conference is held every 2 years. It started in 1984, making it one of the longest running conferences on human–computer interaction. INTERACT 2023 was the 19th International Conference of Technical Committee 13 (Human–Computer Interaction) of the International Federation for Information Processing (IFIP). IFIP was created in 1960 under the auspices of UNESCO. The IFIP Technical Committee 13 (TC13)1 aims to develop the science and technology of human–computer interaction (HCI). The INTERACT Conference series is an important showcase for researchers and practitioners in the field of HCI. Coming under the open, inclusive umbrella of the IFIP, INTERACT is truly international in its spirit and attracts researchers from many countries and cultures. The venues of the INTERACT conferences over the years bear a testimony to this inclusiveness.

Large language models (LLMs) have taken centre stage in debates on Artificial Intelligence. Yet there remains a gap in how to assess LLMs' conformity to important human values. In this paper, we investigate whether state-of-the-art LLMs, GPT-4 and Claude 2.1 (Gemini Pro and LLAMA 2 did not generate valid results) are moral hypocrites. We employ two research instruments based on the Moral Foundations Theory: (i) the Moral Foundations Questionnaire (MFQ), which investigates which values are considered morally relevant in abstract moral judgements; and (ii) the Moral Foundations Vignettes (MFVs), which evaluate moral cognition in concrete scenarios related to each moral foundation. We characterise conflicts in values between these different abstractions of moral evaluation as hypocrisy. We found that both models displayed reasonable consistency within each instrument compared to humans, but they displayed contradictory and hypocritical behaviour when we compared the abstract values present in the MFQ to the evaluation of concrete moral violations of the MFV.

Abstract The social interaction restrictions imposed by COVID-19 required academic researchers to rethink the design of their research projects, especially those that involved human interaction. Considering this scenario, we report here the experience of conducting four studies in Human–Computer Interaction research projects that were adapted to be executed remotely: a focus group study, a communicability evaluation study, a speculative design study and a user observation study. Considering how the studies were planned, executed and analysed, we reflect on the adaptations needed, and lessons related to our experiences. From these lessons, we believe that, even after all the pandemic restrictions are lifted, we may continue conducting at least part of our research studies remotely to increase our geographical reach, reduce costs and facilitate data capture.

[Background] Many startup environments and even traditional software companies have embraced the use of MVPs (Minimum Viable Products) to allow quickly experimenting solution options. The MVP concept has influenced the way in which development teams apply Software Engineering (SE) practices. However, the overall understanding of this influence of MVPs on SE practices is still poor. [Objective] Our goal is to characterize the publication landscape on practices that have been used in the context of software MVPs. [Method] We conducted a systematic mapping study using a hybrid search strategy that consists of a database search and parallel forward and backward snowballing. [Results] We identified 33 papers, published between 2013 and 2020. We observed some trends related to MVP ideation and evaluation practices. For instance, regarding ideation, we found six different approaches (e.g., Design Thinking, Lean Inception) and mainly informal end-user involvement practices (e.g., workshops, interviews). For evaluation there is an emphasis on end-user validations based on practices such as usability tests, A/B testing, and usage data analysis. However, there is still limited research related to MVP technical feasibility assessment and effort estimation. We also observed a lack of scientific rigor in many of the identified studies. [Conclusion] Our analysis suggests that there are opportunities for solution proposals to address gaps concerning technical feasibility assessment and effort estimation. Also, more effort needs to be invested into empirically evaluating the existing MVP-related practices.

The large-scale adoption of systems that automate classifications using Machine Learning (ML) algorithms raises pressing challenges as they support or make decisions with profound consequences for human beings. It is important to understand how users' trust is affected by ML models' suggestions, even when those models are wrong. Many research efforts have focused on the user's ability to interpret what a model has learned. In this paper, we seek to understand another aspect of ML interpretability: how the presence of classification probabilities affects users' trust in the model outcomes, especially in ambiguous instances. To this end, we conducted an online survey in which we asked participants to evaluate their agreement with an automatic classification made by an ML model before and after presenting them the model classification probabilities. Surprisingly, we found that, in ambiguous instances, respondents agreed more with incorrect model outcomes than with correct ones, requiring further analyses.

Nowadays, we have access to data of unprecedented volume, high dimensionality, and complexity. To extract novel insights from such complex and dynamic data, we need effective and efficient strategies. One such strategy is to combine data analysis and visualization techniques, which are the essence of visual analytics applications. After the knowledge discovery process, a major challenge is to filter the essential information that has led to a discovery and to communicate the findings to other people, explaining the decisions they may have made based on the data. We propose to record and use the trace left by the exploratory data analysis, in the form of user interaction history, to aid this process. With the trace, users can choose the desired interaction steps and create a narrative, sharing the acquired knowledge with readers. To achieve our goal, we have developed the BONNIE ( Building Online Narratives from Noteworthy Interaction Events ) framework. BONNIE comprises a log model to register the interaction events, auxiliary code to help developers instrument their own code, and an environment to view users’ own interaction history and build narratives. This article presents our proposal for communicating discoveries in visual analytics applications, the BONNIE framework, and the studies we conducted to evaluate our solution. After two user studies (the first one focused on history visualization and the second one focused on narrative creation), our solution has showed to be promising, with mostly positive feedback and results from a Technology Acceptance Model ( TAM ) questionnaire.

The increasingly rapid growth of data production and the consequent need to explore data to obtain answers to the most varied questions have promoted the development of tools to facilitate the manipulation and construction of data visualizations. However, building useful data visualizations is not a trivial task: it may involve a large number of subtle decisions from experienced designers. In this paper, we present an approach that uses a set of heuristics to recommend data visualizations associated with questions, in order to facilitate the understanding of the recommendations and assisting the visual exploration process. Our approach was implemented and evaluated through the VisMaker tool. We carried out two studies comparing VisMaker with Voyager 2 and analyzed some aspects of the recommendation approaches through the participants' feedbacks. As a result, we found some advantages of our approach and gathered comments to help improve the development of visualization recommender tools.

Petrobras is Brazil's largest publicly-held company, operating in the oil, natural gas, and energy industry. Internal efforts enabled Petrobras to identify Digital Transformation (DT) opportunities to further promote their operational excellence. While addressing these opportunities typically requires Research and Development (R&D) uncertainties that could lead traditional R&D cooperation terms to be negotiated in years, there are time-to-market constraints for fast-paced deliveries to experiment solution options. Having this in mind, they partnered up with PUC-Rio to establish a new DT initiative. [Goal] The goal of this paper is to present the Lean R&D approach, tailored within the new initiative to meet the aforementioned DT needs. [Method] We designed Lean R&D integrating the following building blocks: (i) Lean Inceptions, to allow stakeholders to jointly outline a Minimal Viable Product (MVP); (ii) parallel technical feasibility assessment and conception phases, allowing to `fail fast'; (iii) scrum-based development management; and (iv) strategically aligned continuous experimentation to test business hypotheses. We report on first experiences of applying Lean R&D in practice. [Results] Lean R&D enabled addressing research-related uncertainties early and to efficiently deliver valuable MVPs within fast-paced four months cycles. [Conclusions] In our first experiences Lean R&D showed itself suitable for supporting DT initiatives. However, more formal case studies are needed. The business strategy alignment and the continuous support of a highly qualified research team were considered key success factors.

Interaction models specify the structure and content of the user interface, the allowed user actions, and the corresponding system responses. There is a need to inspect interaction models, as it avoids the propagation of defects to other artefacts. We created two inspection techniques for interaction models, called MoLVERIC Cards (MCards) and MoLVERIC Check (MCheck). MCards employs a gamification mechanism to motivate practitioners during the inspection. MCheck is a simple technique to be used by practitioners in a traditional way. Both techniques have questions, whose answers assist in identifying defects. We performed three studies to verify whether these techniques support the inspection of interaction models. In the first and second studies, we evaluated MCards and MCheck in comparison with an ad‐hoc technique supported by the conventional inspection approach based on defect types. The results of these studies showed that both techniques support the inspection of interaction models. In the third study, we evaluated MCard in comparison with MCheck to understand the participants'; perceptions of both techniques. The study results showed that MCards was considered more suitable for practitioners interested in using dynamic activities, while MCheck was considered more suitable for practitioners who want to use a more traditional technique.

Recent advances in hardware and software technologies have given rise to a new class of human-computer interfaces that both explores multiple modalities and allows for multiple collaborating users. When compared to the development of traditional single-user WIMP (windows, icons, menus, pointer)-based applications, however, applications supporting the seamless integration of multimodal-multiuser interactions bring new specification and runtime requirements. With the aim of assisting the specification of multimedia applications that integrate multimodal-multiuser interactions, this paper: (1) proposes the MMAM (Multimodal-Multiuser Authoring Model); (2) presents three different instantiations of it (in NCL, HTML, and a block-based syntax); and (3) evaluates the proposed model through a task-based user study. MMAM enables programmers to design and ponder different solutions for applications with multimodal-multiuser requirements. The proposed instantiations served as proofs of concept about the feasibility of our model implementation and provided the basis for practical experimentation, while the performed user study focused on capturing evidence of both the user understanding and the user acceptance of the proposed model. We asked developers to perform tasks using MMAM and then answer a TAM (Technology Acceptance Model)-based questionnaire focused on both the model and its instances. As results, the study indicates that the participants easily understood the model (most of them performed the required tasks with minor or no errors) and found it both useful and easy to use. 94.47% of the participants gave positive answers to the block-based representation TAM questions, whereas 75.17% gave positive answers to the instances-related questions.

Online education has broadened the avenues of research on student behavior and performance. In this paper, we shed light on how student interaction logs on virtual learning environments have been analyzed. We conducted two studies: interviews with instructors and a systematic mapping of 136 papers on Education Data Mining and Learning Analytics, published between 2010 and Feb/2019. Then, we triangulate the results between the instructors' answers and research results found in the mapping. The main contributions of this paper are: (i) an overview of Brazilian instructors' experience with Virtual Learning Environments and requirements to improve student analysis; (ii) an overview and classification of the papers in this topic; (iii) a ranking of importance of the instructors' main statements and paper results; and (iv) guidelines based on the requirements mentioned by instructors for designing tools that support their analyses. Finally, we discuss gaps in existing work that can ground and guide future research, such as tools to evaluate student interaction logs with video lectures and to analyze instructor interaction logs on Virtual Learning Environments.

Traditionally, most multimedia content has been developed to stimulate two of the human senses, i.e., sight and hearing. Due to recent technological advancements, however, innovative services have been developed that provide more realistic, immersive, and engaging experiences to the audience. Omnidirectional (i.e., 360-degree) video, for instance, is becoming increasingly popular. It allows the viewer to navigate the full 360-degree view of a scene from a specific point. In particular, when consumed through head-mounted displays, 360-degree videos provide increased immersion and sense of presence. The use of multi-sensory effects —e.g., wind, vibration, and scent— has also been explored by recent work, which allows an improved experience by stimulating other users' senses through sensory effects that go beyond the audiovisual content. Understanding how these additional multi-sensory effects affect the users' perceived quality of experience (QoE) in 360-degree, however, is still an open research problem at large. As a step to better understand the QoE of immersive sensory experiences, this paper presents a testbed and discusses a user-focused study on a scenario in which the user is immersed in the 360-degree video content and is stimulated through additional sensory effects. Quantitative results indicated that the sensorial effects can considerably increase the sense of presence of 360-degree videos. Qualitative results provided us with a better view of the limitations of current technologies and interesting insights such as the users' sense of surprise.

Business process models help understand the organizational process and the software that supports it.BPMN (Business Process Modeling and Notation) is the standard notation for business process modeling, and it is widely accepted in industry.BPMN models can elucidate the activities carried out by a software during its construction and maintenance.However, during the maintenance of the software that supports an organizational process, usually only the source code of the software undergoes modifications, even when inserting new features.The software design models, including the BPMN models, often become outdated over time and, in future maintenance, they will not help understand the business process in which the software is inserted and which the software aims to support.Such scenario highlights the importance of supporting the maintenance of BPMN models.However, what has been experimentally investigated regarding the maintenance of BPMN models?To answer this question, we performed a systematic mapping, which showed experimental studies, factors and technologies that influence the maintenance of BPMN models.These results present conclusions about the state of the art and gaps that can be explored in this field of research.

Background: Unhandled exceptions affect the reliability of web applications. Several studies have measured the reliability of web applications in use against unhandled exceptions, showing a recurrence of the problem during the maintenance phase. Detecting latent unhandled exceptions automatically is difficult and application-specific. Hence, general approaches to deal with defects in web applications do not treat unhandled exceptions appropriately. Aims: To design and evaluate a method that can support finding, correcting, and preventing unhandled exceptions in web applications. Method: We applied the design science engineering cycle to design a method called Pattern-Driven Maintenance (PDM). PDM relies on identifying defect patterns based on application server logs and producing static analysis rules that can be used for prevention. We applied PDM to two industrial web applications involving different companies and technologies, measuring the reliability improvement and the precision of the produced static analysis rules. Results: In both cases, our approach allowed identifying defect patterns and finding latent unhandled exceptions to be fixed in the source code, enabling to completely eliminate the pattern-related failures and improving the application reliability. The static analysis rules produced by PDM achieved a precision of 59-68% in the first application and 89-100% in the second, where lessons learnt from the first evaluation were addressed. Conclusions: The results strengthen our confidence that PDM can help maintainers to improve the reliability for unhandled exceptions in other existing web applications.

The prevalence of design problems may cause re-engineering or even discontinuation of the system. Due to missing, informal or outdated design documentation, developers often have to rely on the source code to identify design problems. Therefore, developers have to analyze different symptoms that manifest in several code elements, which may quickly turn into a complex task. Although researchers have been investigating techniques to help developers in identifying design problems, there is little knowledge on how developers actually proceed to identify design problems. In order to tackle this problem, we conducted a multi-trial industrial experiment with professionals from 5 software companies to build a grounded theory. The resulting theory offers explanations on how developers identify design problems in practice. For instance, it reveals the characteristics of symptoms that developers consider helpful. Moreover, developers often combine different types of symptoms to identify a single design problem. This knowledge serves as a basis to further understand the phenomena and advance towards more effective identification techniques.

After a sensemaking process using a visual analytics application, a major challenge is to filter the essential information that led to a discovery and to communicate the findings to other people. We propose to take advantage of the interaction trace left by the exploratory data analysis, presenting it with a novel visualization to aid in this process. With the trace, the user can choose the desired noteworthy interaction steps and create a visual narrative of his/her own interaction, sharing the acquired knowledge with readers. To achieve our goal, we have developed the BONNIE (Building Online Narratives from Noteworthy Interaction Events) framework. It comprises a log model to register the interaction events and a visualization environment for users to view their own interaction history and to build their visual narratives. This paper presents our proposal for communicating discoveries in visual analytics applications, the BONNIE visualization environment, and an empirical study we conducted to evaluate our solution.

Building Information Modeling (BIM) provides an integrated 3D environment to manage large-scale engineering projects. The Architecture, Engineering and Construction (AEC) industry explores 4D visualizations over these datasets for virtual construction planning. However, existing solutions lack adequate visual mechanisms to inspect the underlying schedule and make inconsistencies readily apparent. The goal of this paper is to apply best practices of information visualization to improve 4D analysis of construction plans. We first present a review of previous work that identifies common use cases and limitations. We then consulted with AEC professionals to specify the main design requirements for such applications. These guided the development of CasCADe, a novel 4D visualization system where task sequencing and spatio-temporal simultaneity are immediately apparent. This unique framework enables the combination of diverse analytical features to create an information-rich analysis environment. We also describe how engineering collaborators used CasCADe to review the real-world construction plans of an Oil & Gas process plant. The system made evident schedule uncertainties, identified work-space conflicts and helped analyze other constructability issues. The results and contributions of this paper suggest new avenues for future research in information visualization for the AEC industry.

Visual analytics applications (VAApps) rely heavily on visual representations and notations to communicate information and support user interaction. The visual representations themselves are the main communication form used by designers, reflecting their interpretation of what data aspects should be highlighted and how users can explore the data. Ultimately, the combination of representation and interactivity guides the user's data analysis in VAApp. To keep the VAApp development centered in their users, the design team must pay special attention to the visual representations and their underlying interaction mechanisms. Our team is developing HistoryViewer, a VAApp itself for users to explore log data obtained from interactions with other ("source") VAApps (VAAppsrc). During the development of HistoryViewer and the instrumentation of a VAAppsrc, we noticed unanticipated benefits for the development of the VAAppsrc itself. This paper presents our HistoryViewer system, proposes an architecture for VAApps integrated with our log model, discussing some ancillary benefits of such instrumentation. Finally, it reports two empirical studies conducted to gather potential users' opinions about HistoryViewer, which revealed positive attitudes towards the system and pointed to opportunities for improvement.

During interaction design, interaction models are developed to help design adequate user interaction with the system.MoLIC (Modeling Language for Interaction as Conversation) is a language used to represent an interaction model, which can then be used as a basis for building other artifacts, such as mockups.However, inspections are necessary to verify whether the MoLIC diagrams are complete, consistent, unambiguous, and contain few or no defects, to avoid propagating preventable defects to derived artifacts.In this paper, we present MoLVERIC, a technique for the inspection of MoLIC diagrams that uses cards with verification items and employs principles of gamification.Furthermore, we discuss the results of a pilot study conducted to analyze the feasibility of this technique.

Collaborative Systems go beyond user-system interaction, allowing users to cooperate, coordinate and communicate with each other. To support model-based design for collaborative systems, we propose an extension to the Modeling Language for Interaction as Conversation (MoLIC), allowing designers to represent user influences, cooperative tasks and awareness mechanisms to design collaboration. In this paper, we present a case study to evaluate the extended MoLIC during interpretation and design tasks performed by 12 undergraduate students. Our results point to a good acceptance of the extension, but defects introduced in the models indicate that some concepts were not very easy to understand as they were presented to participants.

Preferences have been widely studied in several areas including computer science, as they play an important role in many computational tasks, such as decision making support, providing recommendations and personalizing applications. Although many approaches consider particular preference representation models to be used as input for algorithms to address these tasks, there is a need for identifying a model that provides adequate constructions for users to express their preferences. In this paper, we propose a preference meta-model that provides various preference constructs, which include end-user expressions. We start by describing an exploratory study of how people express their preferences in natural language, which provides the basis for the meta-model. After describing the meta-model, we evaluate it with two user studies in different domains. The results of this evaluation indicate that the preference statements that can be expressed with the meta-model are adequate for allowing users to indicate their preferences to a computational system.

Explanations play an essential role in decision support and recommender systems as they are directly associated with the acceptance of those systems and the choices they make. Although approaches have been proposed to explain automated decisions based on multi-attribute decision models, there is a lack of evidence that they produce the explanations users need. In response, in this paper we propose an explanation generation technique, which follows user-derived explanation patterns. It receives as input a multi-attribute decision model, which is used together with user-centric principles to make a decision to which an explanation is generated. The technique includes algorithms that select relevant attributes and produce an explanation that justifies an automated choice. An evaluation with a user study demonstrates the effectiveness of our approach.

We present DysWebxia, an eBook reader for iOS which modifies the form and the content of the text. This tool is specifically designed for people with dyslexia according to previous research with this target group. The settings are customizable depending on the reading preferences.

Automating tasks can make a smartphone easier to use and more battery efficient. However, currently little work has been done to help end-users to create such automations. In this paper, we explore an approach for automating smartphone tasks by demonstration. We have developed a mobile application called Keep Doing It that continuously records users' interactions with their smartphones. After users performed a task that they would like to automate, they can ask our application to create the automation based on their latest actions. Since users only have to use their smartphones, as they would naturally do, to demonstrate automations, we believe that our approach can lower the barrier for creating smartphone automations. Overall, an initial evaluation of the approach suggests that users would be willing to automate their phones by demonstration.

Sketching is viewed as an efficient way to design the user interface. However, there are few tools that help designers go from sketching the user interface to simulating its behavior with endusers during early evaluation. We have developed a pen-based tool called UISKEI, which goes beyond allowing designers to define simple navigation between user interface snapshots, into allowing them to define more complex interactive behavior with conditional changes in the state of the user interface. We have conducted two studies on how UISKEI compares to similar prototyping techniques: paper and pencil; widget-based (Balsamiq) and other sketch-based user interface prototyping tools (DENIM and SketchiXML). The studies revealed that UISKEI's sketch-based definition of the user interface behavior for later simulation is better than the analogous mechanisms provided by the other tools.

Map-based services have increasingly become more important in our daily life, especially mobile maps. However, the screen constraints on mobile devices significantly increases the mode-switching between panning and zooming when trying to retrieve map data which negatively affects the user experience. The Anchored Zoom (AZ) is one of the techniques that have been proposed to overcome this problem, but it has some limitations concerning the use of its anchor point, i.e. a point of reference that serves as a major tool to control zooming in this technique. This paper presents the Anchored Zoom with Anchor Management (AZAM), a new technique that extends AZ by adding an anchor management to it, i.e., by adding new features that better exploit the use of anchors, improving the choice of new anchors and access to the previous ones. We evaluated both techniques, AZ and AZAM, and we also investigated their relevance in comparison with the popular Pinch to Zoom (PZ) technique. The study reveals that AZAM is superior to AZ in perceived satisfaction and that it serves as a complement to PZ during visual search tasks.

To realize the full potential of pen-based interfaces, it is important to provide adequate support to sketching, comparable to paper and pencil. This kind of support is paramount for highly flexible activities, such as user interface sketching. This paper discusses an implementation of a few recognition strategies to support this kind of activity, presenting accuracy test results for each one.

This paper presents a human-centered vision of unmanned but crewed platforms, combining an onshore collaborative hybrid environment and an intelligent offshore oilfield. Computer science plays a major role in this vision since digital environments are paramount for the job. We discuss the importance of research in the fields of computer graphics, virtual reality and computer supported cooperative work in this context and present the basic technologies needed to accomplish the task. We instantiate our vision appointing the results of some initial experiments using virtual reality in submarine installation operations.

It is widely known that, for most computer applications, their users and contexts of usage are quite heterogeneous and change in time. A variability analysis approach was developed to bridge the gap between requirements engineering and human-computer interaction can help identify these variations and design flexible, adaptable and adaptive solutions that accommodate them at interaction time, i.e., during system usage. The core assumption of the approach is that an in-depth exploration of how user goals can be formulated into diverse user requests expressed in a linguistic structure using cases, designers will be better equipped to create solutions that cope with variations during user-system interaction. This paper shows how a set of adaptation-related questions and related interactive solutions may help to bridge the problem and solution spaces of flexible, adaptable and adaptive systems.

This working session will be an interactive discussion about multimedia content transformation. The basic assumption is that content transformation activities should be provided as non-destructive operations. The final goal of the panel is to gather researchers within the community interested in manipulating multimedia content for providing rich user experiences. The organizers of the panel will moderate and shape the discussion; nevertheless, position papers from the participants are expected.

Much research has been devoted to adaptable and adaptive systems over a decade, with the aim to improve user-system interaction. Despite these efforts, we still lack a systematic approach to deal with variations in users, user goals, and contexts of use. In this paper, we revise some previous work on a language-based variability analysis process, propose a structure for the solution space of adaptable and adaptive systems, and illustrate how design decisions on how to cope with variability can be made in this approach.

Metaphor is not merely a rhetorical device, characteristic of language alone, but rather a fundamental feature of the human conceptual system. A metaphor is understood by finding an analogy mapping between two domains. This paper argues that analogy mappings facilitate conceptual modeling by allowing the designer to reinterpret fragments of familiar conceptual models in other contexts. The contributions of the paper are expressed within the tradition of the Entity-Relation model.

End user programming (EUP) environments are difficult to evaluate empirically. Most users do not engage in programming, and those who do are often discouraged by the complexity of programming tasks. Often the difficulties arise from the programming languages in which users are expected to express themselves. But there are other difficulties associated with designing extensions and adjustments to artifacts that have been originally designed by others. This paper characterizes EUP as a semiotic design process, and presents two principles that can be used to illustrate the distinctions between the various kinds of techniques and approaches proposed in this field. The principles support a preliminary theoretical model of EUP and should thus facilitate the definition and interpretation of empirical evaluation studies. They also define some specific semiotic qualifications that more usable and applicable EUP languages could be expected to have.

This article is about applications that can be customized or extended through their own user interface. This is achieved by the interface's ability to interpret users' non-literal expressions, namely metaphorical and metonymic ones. Such increased interpretive intelligence depends on static and dynamic models of the domain and application, from which new figurative meanings are abducted automatically or semi-automatically. The system performs controlled modifications on the underlying models, based on its interferences about users' intentions as they produce figurative utterances.