The rapid aging of the global population has highlighted the need for technologies to support elderly, particularly in healthcare and emotional well-being. Facial expression recognition (FER) systems offer a non-invasive means of monitoring emotional states, with applications in assisted living, mental health support, and personalized care. This study presents a systematic review of deep learning-based FER systems, focusing on their applications for the elderly population. Following a systematic methodology, we analyzed 39 studies published over the last decade, addressing challenges such as the scarcity of elderly-specific datasets, class imbalances, and the impact of age-related facial expression differences. Our findings show that convolutional neural networks remain dominant in FER, and especially lightweight versions for resource-constrained environments. However, existing datasets often lack diversity in age representation, and real-world deployment remains limited. Additionally, privacy concerns and the need for explainable artificial intelligence emerged as key barriers to adoption. This review underscores the importance of developing age-inclusive datasets, integrating multimodal solutions, and adopting explainable artificial intelligence techniques to enhance system usability, reliability, and trustworthiness. We conclude by offering recommendations for future research to bridge the gap between academic progress and real-world implementation in elderly care.

Similarity is a key property across two or more interactive systems. Indeed, interacting with similar systems can bring benefits (e.g., reduced learning efforts and training time) but may also raise issues (e.g., interference errors and higher cognitive load). Interactive systems may be similar, as they correspond to the same work and tasks performed with different underlying systems, such as the pilots’ tasks in the cockpit of an Airbus A320 and a Boeing 737. They may also be similar by design, as they belong to the same suite as, for instance, the Microsoft Office software suite, where several tools are designed to be used concurrently by the same users. Previous work on similarity has focused on the visual presentation of interactive applications, highlighting commonalities and differences in terms of layout, shape, colours and other features of the user interface. This paper proposes a systematic and formal approach to analyse and assess the similarity between several interactive systems, focusing on their behaviours. To this end, we propose a tool-supported process that exploits both interactive formal system behaviour models and user task models. These models are analysed with the help of formal tools (model checking) to identify commonalities and differences in their exhibited behaviours. Beyond, we compare the specific and generic behavioural properties of each model, which provide semantic and meaningful information about how similar they are and how they differ. The approach is applied to two similar critical command and control interactive systems for flight safety operations in the space domain.

Operators of critical systems (e.g., air traffic controllers, pilots in commercial aircraft, operators managing space vehicle launch…) must acquire specific knowledge before being granted permission to perform their tasks on the job. They acquire such knowledge by following a training program developed for the specific tasks they will be doing. The preparation of their training program requires identifying the knowledge they have to acquire, and in particular, the two main types of knowledge that are declarative knowledge (a set of concepts required to operate a system) and procedural knowledge (a set of tasks required to operate a system). Task analysis techniques were originally proposed for the purpose of identifying and describing the knowledge required by users to interact with a system. These techniques provide support to describe procedural knowledge in detail, and a few of them provide limited support to describe declarative knowledge. In this paper, we propose a model-based technique to exhaustively identify and describe declarative and procedural knowledge, as well as to couple the different types of models in order to ensure the full coverage and consistency of the identification of knowledge required to operate an interactive system. We present the results from the application of the technique on a case study in the space domain.

This special issue presents four papers featuring extended versions of full papers originally published at the Conference INTERACT 2023, which was held August 28th to September 1st, 2023, at the University of York, York, UK. We will first present the INTERACT Conference and then the selection process of papers that ultimately led to publication of this special issue in Interacting with Computers. The INTERACT Conference is held every 2 years. It started in 1984, making it one of the longest running conferences on human–computer interaction. INTERACT 2023 was the 19th International Conference of Technical Committee 13 (Human–Computer Interaction) of the International Federation for Information Processing (IFIP). IFIP was created in 1960 under the auspices of UNESCO. The IFIP Technical Committee 13 (TC13)1 aims to develop the science and technology of human–computer interaction (HCI). The INTERACT Conference series is an important showcase for researchers and practitioners in the field of HCI. Coming under the open, inclusive umbrella of the IFIP, INTERACT is truly international in its spirit and attracts researchers from many countries and cultures. The venues of the INTERACT conferences over the years bear a testimony to this inclusiveness.

Designing new input devices and associated interaction techniques is a key contribution in order to increase the bandwidth between users and interactive applications. In the field of Human-Computer Interaction, research and development services in industry and research laboratories in universities have been, since the invention of the mouse and the graphical user interface, proposing multiple contributions including the integration of multiple input devices in multimodal interaction technique. Those contributions, most of the time, are presented as prototypes or demonstrators providing evidence of the bandwidth increase through user studies. Such contributions however, do not provide any support to software developers for integrating them in real-life systems. When done, this integration is performed in a craft manner, outside required software engineering good practice. This paper proposes a systematic process to integrate novel input devices and associated interaction techniques to better support users' work. It exploits most recent interactive systems architectural model and formal model-based approaches for interactive systems supporting verification and validation when required. The paper focusses on Frame-based input devices which support gesture-based interactions and movements recognition but also addresses their multimodal use. This engineering approach is demonstrated on an interactive application in the area of rehabilitation in healthcare where dependability of interactions and applications is as critical as their usability.

Post-stroke or post-brain injuries rehabilitation is a long-term process defined by therapists and tuned to the specific damages and their consequences for each patient. This process requires the definition of well-defined multiple exercises to be performed multiple times by each patient over a long period of time. One of the key findings in the rehabilitation domain is that exercises should be tasks and goals oriented to increase functional independence, enhance development and prevent disability [24]. Patients performing the exercises must be carefully monitored to identify progress, possible regression and prevent damages due to the inadequate performance of the exercises. The SIVIRE tool [20] provides interactive graphical feedback to patients about the performance of their exercises and deviations from nominal, planed practice but fails at providing high-level descriptions of exercises and at connecting them to high-level rehabilitation goals (e.g. gain autonomy in cleaning a room). The HAMSTERS-XLE tool [16] provides a tuneable editor and a simulator to describe goals and task models including low-level tasks covering perception of the various senses, cognition and motoric actions. This paper presents the integration of these two tools and the benefits it brings both to the patients and to the therapists. The resulting tool and its engineering are presented in detail highlighting some generic integration mechanisms. On a case study, we show how exercises can be defined in HAMSTERS-XLE, transferred to SIVRE for fine tuning, monitored by SIVRE during the multiple practices of the patient and transferred back again to HAMSTERS-XLE to monitor progress, identify issues with the exercises and provide therapists with goal and task-based data about patients' evolutions

Abstract This paper proposes a unified formal framework, Formal Framework For FLUID (F3FLUID), for the development of safety‐critical interactive systems. This framework is based on the Formal Language of User Interface Design (FLUID) pivot modeling language defined in the FORMEDICIS project, which enables high‐level system requirements for interactive systems to be specified in the FLUID language. This modeling language is specifically designed for handling concepts of safety‐critical interactive systems, including domain knowledge. A FLUID model is used as a source model for the generation of several target models in different modeling languages to support the formal verification methods, such as theorem proving and model checking. In this paper, we use the Event‐B modeling language for checking functional behaviors, user interactions, safety properties, and domain properties. A FLUID model is transformed into an Event‐B model, and then, the Rodin tool is used to check the internal consistency with respect to the given safety properties. We illustrate the operational semantics of the FLUID language, and the transformation strategy of FLUID models into Event‐B models, including the tool development. We use the ProB model checker to analyze the temporal properties and to animate the formalized specification. In addition, an interactive cooperative objects (ICOs) model is derived from the Event‐B model for animation, visualization and validation of dynamic behaviors, visual properties, and task analysis. Finally, an industrial case study, complying with the ARINC 661 standard, Multi‐Purpose Interactive Applications (MPIA), is used to illustrate the effectiveness of our F3FLUID framework for the development of safety‐critical interactive systems.

System engineering advocates a thorough under-standing of the engineering domain or certification standards (aeronautics, railway, medical, etc.) associated to the system under design. In this context, engineering domain knowledge plays a predominant role in system design and/or certification. Furthermore, it is a prerequisite to achieve the effectiveness and performance of the designed system. This article proposes a formal method for describing and setting up domain-specific behavioural analyses. It defines a formal verification technique for dynamic properties entailed by engineering domain knowledge where Event-B formal models are annotated and analysed in a non-intrusive way, i.e. without destructive alteration. This method is based on the formalisation of behavioural properties analyses relying on domain knowledge as an ontology on the one hand and a meta-theory for Event-B on the other hand. The proposed method is illustrated using a critical interactive system.

Operators’ well-being is a key factor for the success of industrial production processes. Even though research has studied the well-being aspects of the industry, such as support and improvement of ergonomics, there is still a long way to go to achieve a sustainable and healthy work context for manufacturing industry. We believe the Human-Computer Interaction community can contribute by developing research on worker well-being in real-life settings. This workshop intends to offer a venue for HCI researchers that focus on worker well-being for the manufacturing industry and other industry domains.

In this paper, a task model based on HAMSTERS-XL notation is proposed to represent activities of daily living (ADL). For an efficient representation of ADL, it is also a need to delimit requirements in time and location for each ADL, as well as the sensor data that define the ADL. In this sense, it is also proposed a procedure consisting of a set of step intending to delimit such time and location requirements, regarding both activities and sensors; alongside the events that need to be identified. The presented approach aims at providing an analysis tool regarding the performance of elderly/handicapped people by comparing data recovered from sensors within a smart home environment simulator to the task models. The feed oneself ADL (as well as its possible variations) is picked in order to evaluate our proposal. After proper analysis was carried out, anomalies detected during ADL performance are pointed out in order to detect possible ADL routine modification in a coherent manner through improved task models.

International audience

Training operators of complex interactive systems is a difficult task that involves multiple actors, requires specific competencies and may be extremely costly in terms of time and resources. For instance, an initial training has to be performed in order to bring operators to a desired level of declarative and procedural knowledge. A successful operator will be granted a qualification to operate the system that was used for (or targeted by) the training. However, depending on the nature and the diversity of the operations performed on a daily basis, this initial training may decay and some knowledge may become deprecated. Recurrent training needs to be organised on a regular basis in order to keep operators qualified for the work. Such training differs from initial training and requires taking into account information about how humans learn and forget, what has been performed by operators since last training and the expected required level of knowledge. Trainers need to encompass all this generic information (to all operators) and specific information (to each individual) to define both initial and recurrent training programs. One particularly cumbersome task is the gathering of what happened during operations for each operator in order to minimize recurrent training program to knowledge that might have been forgotten as it was not used in recent operations. We propose a tool-supported task-model approach fed by information of the real work of operators in order to identify complete, relevant and efficient training for operators. These contributions have been applied to the civil aviation domain demonstrating multiple induced benefits.

Along the design process of interactive system multiple intermediate artefacts (such as user interface prototypes, task models, dialog models?) are created, tested, revised, and improved until the development team produces a full-fledged system. However, relevant information for describing the design solution and/or supporting design decisions (such as rational about the design, decisions made, recommendations, etc.) are not explicitly captured in the models/artefacts, hence the need for annotations. Many approaches argue against information duplication to increase maintainability of the artefacts. Nonetheless, annotations created on one artefact are usually relevant to other artefacts/models. So that, there is a need for tools and techniques to coordinate annotations across artefacts/models which is the contribution of the present work. In this paper, we propose a model-based approach that was conceived to handle annotations in a systematic way along the development process of interactive systems. As part of the solution, we propose an annotation model built upon the W3C's Web Annotation Data Model. The feasibility of the approach is demonstrated by means of a tool suite featuring a plugin, which has been deployed and tested over the multi-artefacts. The overall approach is illustrated on the design of an interactive cockpit application performing two design iterations. The contribution brings two main benefits for interactive systems engineering: i) it presents a generic pattern for integrating information in multiple usually heterogenous artefacts throughout the design process of interactive systems; and ii) it highlights the need for tools helping to rationalize and to document the various artefacts and the related decisions made during interactive systems design.

Abstract This editorial introduces the special issue of Interacting with Computer on Heterogeneous Models and Modelling Approaches for Engineering of Interactive Systems. This special issue was proposed to gather the best contributions from a series of workshops organized alongside conferences such as FM’19 (3rd World Congress on Formal Methods) and EICS’19 (11th ACM SIGCHI Symposium on Engineering Interactive Computing Systems). It also encompasses papers submitted directly to this special issue.

Abstract Training operators to efficiently operate critical systems is a cumbersome and costly activity. A training program aims at modifying operators’ knowledge and skills about the system they will operate. The design, implementation and evaluation of a ‘good’ training program is a complex activity that requires involving multi-disciplinary work from multiple stakeholders. This paper proposes the combined use of task descriptions and augmented reality (AR) technologies to support training activities both for trainees and instructors. AR interactions offer the unique benefit of bringing together the cyber and the physical aspects of an aircraft cockpit, thus providing support to training in this context that cannot be achieved by software tutoring systems. On the instructor side, the LeaFT-MixeR system supports the systematic coverage of planed tasks as well as the constant monitoring of trainee performance. On the trainee side, LeaFT-MixeR provides real-time AR information supporting the identification of objects with which to interact, in order to perform the planned task. The paper presents the engineering principles and their implementation to bring together AR technologies and tool-supported task models. We show how these principles are embedded in LeaFT-MixeR system as well as its application to the training of flight procedures in aircraft cockpits.

Abstract Feedback and feedforward are two fundamental mechanisms that support users’ activities while interacting with computing devices. While feedback can be easily solved by providing information to the users following the triggering of an action, feedforward is much more complex as it must provide information before an action is performed. For interactive applications where making a mistake has more impact than just reduced user comfort, correct feedforward is an essential step toward correctly informed, and thus safe, usage. Our approach, Fortunettes, is a generic mechanism providing a systematic way of designing feedforward addressing both action and presentation problems. Including a feedforward mechanism significantly increases the complexity of the interactive application hardening developers’ tasks to detect and correct defects. We build upon an existing formal notation based on Petri Nets for describing the behavior of interactive applications and present an approach that allows for adding correct and consistent feedforward.

Abstract This paper presents a formal development approach for designing interactive applications using a correct-by-construction approach. In this work, we propose a refinement strategy using model-view-controller (MVC) to structure and design Event-B formal models of the interactive application. The proposed MVC-based refinement strategy facilitates the development of an abstract model and a series of refined models by introducing the possible modes, controller’s behaviour and visual components of the interactive application while preserving the required interaction-related safety properties. To demonstrate the effectiveness, scalability, reliability and feasibility of our approach, we use a small example (from automotive domain) and real-life industrial case studies (from aviation). The entire development is realized in Event-B and the associated Rodin tool is used to analyse and verify the correctness of the formalized model. Finally, the developed Event-B models are used to generate source code using EB2ALL tool for going from the specification to the implementation of the interactive application.

The goal of software testing is to detect defects with the objective of removing them at a later stage in the development process. Interactive software development follows the User Centered Design approach that promotes continuous involvement of users both at design and evaluation phases. This process is meant to produce usable interactive software by gathering functional and non-functional requirements related to both user needs and context of use. However, taking into account these potentially very-complex-to-implement requirements increases the complexity of the software that is likely, without appropriate methods and tools, to encompass a large number of defects. One of the limitations of UCD approaches is that it provides no guidance on the engineering of the interactive application, which thus usually embeds numerous defects resulting in failures at the origin of user frustrations and performance drops. Even though a classification of interactive application defects has been proposed, interactive application testers remain only superficially supported in detecting them. This paper defines a model-based approach to engineer the testing activity for interactive applications. It proposes a process that bridges the gap between UCD artefacts and interactive software implementation by the production of a dedicated formal model exploited for testing purposes only. The application of the process is demonstrated on an interactive cockpit WIMP application. Finally, threats to validity (capability of the approach to detect defects and to ensure an acceptable coverage testing of the interactive application) are addressed by a longitudinal study on 61 variants of a simple application developed by 61 different developers. ?

This issue of the Proceedings of the ACM on Human-Computer Interaction features contributions in the intersection of human-computer interaction and software engineering, with further disciplines blending into a rich set of scientific works. 2021 is the first time the annual conference on Engineering Interactive Computing Systems (EICS) is hosted in the Netherlands and in the context of an Industrial Design department. We take this opportunity to focus on the relations and influence of the design discipline on the work of the EICS community. This resulted in a new set of topics for EICS, which were already partly reflected in the many submissions we received in three extensive review rounds throughout 2020 and the beginning of 2021. In this editorial we offer a perspective on what EICS is not yet, looking at the inclusion of and interplay with design as a related discipline.

When interactive systems allow users to interact with critical systems, they are qualified as Critical Interactive Systems, CIS for short. Their design requires the support of different activities and tasks to achieve user goals. Examples of such systems are cockpits, nuclear plant control panels, medical devices, etc. Such critical systems are very difficult to model due to the complexity of the offered interaction capabilities. This paper presents a formal framework, F3FLUID (Formal Framework For FLUID), for designing safety-critical interactive systems. It relies on FL UID as core modelling language. FL UID enables the modelling and use of interactive systems domain concepts and supports an incremental design of such systems. Formal verification, validation and animation of the designed models are supported through different transformations of FLUID models into target formal verification techniques: Event-B for formal verification, ProB model checker for animation and Interactive Cooperative Objects for user validation. The Event-B models are generated from FLUID while ICO and ProB models are produced from Event-B. We exemplify the real-life case study TCAS (Traffic alert and Collision Avoidance System) to demonstrate our framework.

Automation, as a design goal, focusses mainly on the migration of tasks from a human operator to a mechanical or digital system. Designing automation thus usually consists in removing tasks or activities from that operator and in designing systems that will be able to perform them. When these automations are not adequately designed (or correctly understood by the operator), they may result in so called automation surprises [1], [2] that degrade, instead of enhance, the overall performance of the couple (operator, system). Usually, these tasks are considered at a high level of abstraction (related to work and work objectives) leaving unconsidered low-level, repetitive tasks. This paper proposes a decomposition of automation for interactive systems highlighting the diverse objectives it may target at. Beyond, multiple complementary views of automation for interactive systems design are presented to better define the multiform concept of automation. It provides numerous concrete examples illustrating each view and identifies ten rules for designing interactive systems embedding automations.

Use error due to user interface design defects is a major concern in many safety critical domains, for example avionics and health care. Early detection of latent user interface problems can be facilitated by user-centered design methods that integrate formal verification technologies. This article considers the role that formal verification technologies can play in the context of user-centered design by considering the following three existing tools: CIRCUS, PVSio-web, and IVY. These tools have been developed to support the model based analysis of critical user interfaces. They have their foundations in existing formal verification technologies, but each of them is focused towards particular issues relating to user interface design. The article explores the different phases of the user-centered design process and the extent to which each of these tools supports these phases. Criteria are developed for assessing their role at each stage of the design process. The results of the evaluation provide guidance to developers to help choose the most appropriate tool based on their analysis needs while at the same time setting challenges for future developments.

Incorporating touchscreen interaction into cockpit flight systems offers several potential advantages to aircraft manufacturers, airlines, and pilots. However, vibration and turbulence are challenges to reliable interaction. We examine the design space for braced touch interaction, which allows users to mechanically stabilise selections by bracing multiple fingers on the touchscreen before completing selection. Our goal is to enable fast and accurate target selection during high levels of vibration, without impeding interaction performance when vibration is absent. Three variant methods of braced touch are evaluated, using doubletap, dwell, or a force threshold in combination with heuristic selection criteria to discriminate intentional selection from concurrent braced contacts. We carried out an experiment to test the performance of these methods in both abstract selection tasks and more realistic flight tasks. The study results confirm that bracing improves performance during vibration, and show that doubletap was the best of the tested methods.

Feedback is commonly used to explain what happened in an interface. What if questions, on the other hand, remain mostly unanswered. In this paper, we present the concept of enhanced widgets capable of visualizing their future state, which helps users to understand what will happen without committing to an action. We describe two approaches to extend GUI toolkits to support widget-level feedforward, and illustrate the usefulness of widget-level feedforward in a standardized interface to control the weather radar in commercial aircraft. In our evaluation, we found that users require less clicks to achieve tasks and are more confident about their actions when feedforward information was available. These findings suggest that widget-level feedforward is highly suitable in applications the user is unfamiliar with, or when high confidence is desirable.

When task descriptions are precise they can be analysed to yield a variety of insights about interaction, such as the quantity of actions performed, the amount of information that must be perceived, and the cognitive workload involved. Task modelling notations and associated tools provide support for precise task description, but they generally provide a fixed set of constructs, which can limit their ability to model new and evolving application domains and technologies. This article describes challenges involved in using fixed notations for describing tasks. We use examples of recognized tasks analysis processes and their phases to show the need for customization of task notations, and through a series of illustrative examples, we demonstrate the benefits using our extensible task notation and tool (HAMSTERS-XL).

Touchscreen input in commercial aircraft cockpits offers potential advantages, including ease of use, modifiability, and reduced weight. However, tolerance to turbulence is a challenge for their deployment. To better understand the impact of turbulence on cockpit input methods we conducted a comparative study of user performance with three input methods -- touch, trackball (as currently used in commercial aircraft), and a touchscreen stencil overlay designed to assist finger stabilization. These input methods were compared across a variety of interactive tasks and at three levels of simulated turbulence (none, low, and high). Results showed that performance degrades and subjective workload increases as vibration increases. Touch-based interaction was faster than the trackball when precision requirements were low (at all vibrations), but it was slower and less accurate for more precise pointing, particularly at high vibrations. The stencil did not improve touch selection times, although it did reduce errors on small targets at high vibrations, but only when finger lift-off errors had been eliminated by a timeout. Our work provides new information on the types of tasks affected by turbulence and the input mechanisms that perform best under different levels of vibration.

The rise of autonomous and fully autonomous vehicles requires a closer examination of how people will interact with them. In this workshop, we especially address two areas: (1) What can we learn from mobile HCI for the interaction design between human drivers and the autonomous vehicle? (2) What opportunities for mobile HCI arise due to new freedoms for drivers, when they also take on the role of passengers in autonomous vehicles? The workshop will seek to further develop the challenges involved and/or to suggest early solutions through the use of a highly participative format.

Ensuring that an interactive application allows users to perform their activities and reach their goals is critical to the overall usability of the interactive application. Indeed, the effectiveness factor of usability directly refers to this capability. Assessing effectiveness is a real challenge for usability testing as usability tests only cover a very limited number of tasks and activities. This paper proposes an approach towards automated testing of effectiveness of interactive applications. To this end we resort to two main elements: an exhaustive description of users' activities and goals using task models, and the generation of scenarios (from the task models) to be tested over the application. However, the number of scenarios can be very high (beyond the computing capabilities of machines) and we might end up testing multiple similar scenarios. In order to overcome these problems, we propose strategies based on task models manipulations (e.g., manipulating task nodes, operator nodes, information...) resulting in a more intelligent test case generation approach. For each strategy, we investigate its relevance (both in terms of test case generation and in terms of validity compared to the original task models) and we illustrate it with a small example. Finally, the proposed strategies are applied on a real-size case study demonstrating their relevance and validity to test interactive applications.

User interfaces for the command and control of transportation and navigation systems, such as aircraft cockpits, usually integrate several types of interaction elements: physical, hardware or software. Within these cyber-physical environments, operators have to complete their tasks manipulating these different types of elements. However, task description notations do not take into account physical and hardware aspects beyond manipulation of input devices such as mouse and keyboard. This paper identifies generic aspects of cyber-physical interactive systems and proposes extensions to operators' tasks description techniques, to capture them. We argue that representing cyber-physical elements explicitly and systematically in task models contribute to the design and development of usable and reliable transportation systems. These extensions are integrated within the tool-supported notation called HAMSTERS and are illustrated on a case study from the avionics domain.

The overall dependability of an interactive system is one of its weakest components, which is usually its user interface. The presented approach integrates techniques from the dependable computing field and elements of the user-centered design. Risk analysis and fault-tolerance techniques are used in combination with task analysis and modeling to describe and analyze the impact of system faults on human activities and the impact of human deviation or errors on system performance and overall mission performance. A technique for systematic analysis of human errors, effects, and criticality (HEECA) is proposed. It is inspired and adapted from the Failure Mode, Effects, and Criticality Analysis technique. The key points of the approach are: 1) the HEECA technique combining a systematic analysis of the effects of system faults and of human errors; and 2) a task modeling notation to describe and to assess the impact of system faults and human errors on operators' activities and system performance. These key points are illustrated on an example extracted from a case study of the space domain. It demonstrates the feasibility of this approach as well as its benefits in terms of identifying opportunities for redesigning the system, redesigning the operations, and for modifying operators' training.

Over the last few years multi-touch interfaces have made their ways in most environments including mobile technologies, flight entertainment systems, consumer electronics... Such interfaces and associated interaction techniques have demonstrated benefits partly due to the fact that the output device integrates input management thus bridging the (classical) gap between input and output in user interfaces. They have also demonstrated benefits in terms of performance for triggering commands by exploiting multi-fingers interactions thus reducing the number of unnecessary modes. Together with these benefits, multi-touch interfaces bring a set of issues that are still to be solved prior to make them “acceptable” for command and control of (safety) critical interactive systems. Such issues include usability and reliability in a way that is even more salient than in “classical” Windows, Icons, Menus, Pointing device (WIMP) interfaces that have been around for more than 30 years. This paper proposes a notation and its associated tool for describing in a complete and unambiguous way multi-touch interactions thus mainly targeting at reliability. We present from a case study (in the application domain of interactive cockpits of aircrafts) how the notation makes it possible to describe such interactions from the hardware level and how this notation can be integrated into an industrial development process.

Engineering interactive systems for safety critical applications such as in avionic digital cockpits (and more generally Graphical User interfaces) is a challenge from a dependability viewpoint. The dependability of the user interface and its related hardware and software components must be consistent with the criticality of the functions to be controlled and their required DAL levels. This paper proposes a stepwise refinement approach going from systematic identification of failure modes of these systems to their detection via formally defined assertions. The last steps of the approach present how the assertions can be included into the monitoring part of self-checking interactive components and how they can be deployed on an architecture compliant with the ARINC 653 specification, ensuring temporal and spatial segregation, thus detecting errors and preventing failures due to both physical and transient software faults. We present how these contributions have been applied to the Flight Control Unit Backup interactive application which is available in A380 interactive cockpits.

In this showpiece we demonstrate PetriNect, an instance of a generic layered framework that we have developed for the specification and use of executable models of gestural interaction with virtual objects. The framework is built on top of Petshop and uses ICO models, a variant of high-level Petri nets. PetriNect uses the Kinect as input device for allowing the user to interact gesturally with virtual objects. We present two simple proof-of-concept prototype applications that have been developed for the purpose of this showpiece: a simple Pong game, and the interaction with a virtual bookshelf.

Integrating new input devices and their associated interaction techniques into interactive applications has always been challenging and time-consuming, due to the learning curve and technical complexity involved. Modeling devices, interactions and applications helps reducing the accidental complexity. Visual modeling languages can hide an important part of the technical aspects involved in the development process, thus allowing a faster and less error-prone development process. However, even with the help of modeling, a gap remains to be bridged in order to go from models to the actual implementation of the interactive application. In this paper we use ICO, a visual formalism based on high-level Petri nets, to develop a generic layered framework for specifying executable models of interaction using gestural input devices. By way of the CASE tool Petshop we demonstrate the framework's feasibility to handle the Kinect and gesture-based interaction techniques. We validate the approach through two case studies that illustrate how to use executable, reusable and extensible ICO models to develop gesture-based applications.

In the last few years, glass cockpits are being replaced by interactive cockpits to provide a higher level of integration of both command and information display. Due to their event driven nature, interactive systems offer more display and control capabilities but they require specific error detection and fault tolerance techniques to reach a high level of dependability. This paper proposes a model-based approach for adding fault tolerance mechanisms to interactive cockpits. While several mechanisms are considered and presented, the contribution is focused on the formal description of self-checking widgets, being the basis for interactive cockpits.

Designing systems in such a way that as much functions as possible are automated has been the driving direction of research and engineering in aviation, space and more generally in computer science for many years. In the 90's many studies (e.g. [12] related to the notion of mode confusion) have demonstrated that fully automated systems are out of the grasp of current technologies and that additionally migrating functions [2] from the operator to the system might have disastrous impact on operations both in terms of safety and usability. In order to be able to design automation with a hedonic view of the involved factors (safety, usability, reliability, ...) a complete understanding of operator's tasks is required prior to considering migrating them to the system side. This paper proposes a contribution for reasoning about automation designs using a model-based approach exploiting refined task models. These models describe operations with enough details in order to reason about automation and to rationalize automation designs. In this paper we present how such representations can support the assessment of alternative design options for automation. The proposed approach is applied to satellite ground segments.

Representing the behaviour of multimodal interactive systems in a complete, concise and non-ambiguous way is still a challenge for formal description techniques (FDT). Depending on the FDT, multimodal interactive systems feature specific characteristics that are either cumbersome or impossible to capture with classical FDT. This is due to the multiple (potentially synergistic) use of modalities and the strong temporal constraints usually encountered in this kind of systems that have to be dealt with exhaustively if FDT are used. This paper focuses on the requirements for the modelling and construction of fusion engines for multimodal interfaces. It proposes a formal description technique dedicated to the engineering of interactive multimodal systems able to address the challenges of fusion engines. Such benefits are presented on a set of examples illustrating both the constructs and the process.

Fusion engines are fundamental components of multimodal inter-active systems, to interpret input streams whose meaning can vary according to the context, task, user and time. Other surveys have considered multimodal interactive systems; we focus more closely on the design, specification, construction and evaluation of fusion engines. We first introduce some terminology and set out the major challenges that fusion engines propose to solve. A history of past work in the field of fusion engines is then presented using the BRETAM model. These approaches to fusion are then classified. The classification considers the types of application, the fusion principles and the temporal aspects. Finally, the challenges for future work in the field of fusion engines are set out. These include software frameworks, quantitative evaluation, machine learning and adaptation.

The design of real-life complex systems calls for advanced software engineering models, methods, and tools in order to meet critical requirements such as reliability, dependability, safety, or resilience that will avoid putting the company, the mission, or even human life at stake. When such systems encompass a substantial interactive component, the same level of confidence is required towards the human-computer interface. Conventional empirical or semiformal techniques, although very fruitful, do not provide sufficient insight on the reliability of the human-system cooperation, and offer no easy way to, for example, quantitatively and qualitatively compare two design options with respect to that reliability. The aim of this article is to present a user interface description language (called ICOs) for the engineering and development of usable and reliable user interfaces. The CASE tool supporting the ICOs notation (called Petshop) is a Petri nets-based-tool for the design, specification, prototyping, and validation of interactive software. In that environment models (built with the formal description technique ICOs) of the interactive application can be interactively modified and executed. This is used to support prototyping phases (when the models and the interactive application evolve significantly to meet late user requirements, for instance) as well as the operation phase (after the system is deployed). The use of ICOs and PetShop is presented on several large-scale systems such as a multimodal ground segment application for satellite control, an air traffic control interactive application, and an application for new generation of interactive cockpits in large civil aircraft such as Airbus A380 or Boeing 787. The article emphasizes the demonstration of the expressive power of the notation and how it can support the description of various aspects of user interfaces, namely interaction techniques (both WIMP and post-WIMP), interactive components (such as widgets), and the behavioral part of interactive applications such as the dialog and the functional core. It also demonstrates that PetShop provides dedicated support for prototyping activities of behavioral aspects at the various levels of the architecture of interactive systems. While the focus is on past work done on various large-scale applications, the article also highlights why and how ICOs and Petshop are able to address challenges raised by next-generation user interfaces.

This paper is an extended version of the final version of a paper accepted at EHCI-DSVIS 2004, Lecture Notes in Computer Science n°3425

This paper presents the use of a model-based approach for the formal description of real-time embedded multimodal systems. This modeling technique has been used in the field of military fighter aircrafts. The paper presents the formal description techniques, its application on the case study of a multimodal command and control interface for the Rafale aircraft as well as its relationship with architectural model for interactive systems.

This paper presents a design framework for safety critical interactive systems, based on a formal description technique called the ICO (Interactive Cooperative Object) formalism. ICO allows for describing, in a formal way, all the components of highly interactive (also called post-WIMP) applications. The framework is supported by a case tool called PetShop allowing for editing, verifying and executing the formal models. The first section describes why such user interfaces are challenging for most description techniques, as well as the state of the art in this field. Section 3 presents a development process dedicated to the framework. Then, we use a case study in order to recall the basic concepts of the ICO formalism and the recent extensions added in order to take into account post-WIMP interfaces' specificities. Section 5 presents the case tool PetShop and how the case study presented in the previous section has been dealt with. Lastly, we show how PetShop can be used for interactive prototyping.

No abstract available.

CORBA is now established as one of the main contenders in object-oriented middleware.Beyond the definition of this standard for distributed object systems, the Object Management Group (OMG) has specified several object services (Common Object Services, COS) that should foster the interoperability of distributed applications.Based on experiment, the goal of this paper is to show that the OMG's style of specification of the CORBA services is not suited to guarantee that implementers will produce interoperable and substitutable implementations.To illustrate our point, we give an account of an experiment based upon the formal specification of one COS, namely the CORBA Event Service.This formal specification highlights several ambiguities and under-specifications in the OMG document.We then test several commercial and public domain implementations of the CORBA Event Service, in order to assess how the implementers have dealt with these underspecifications.We show that the choices made by the implementers lead to incompatible implementations.We finally suggest a solution to overcome the problem of specification of object services, which satisfies the views of both implementers and users.Specifically, we suggest that the specification of such services be made using a formal description technique, and that implementers be provided with test cases derived from the formal specification.

CORBA is a standard proposed by the Object Management Group (OMG) in order to promote interoperability between distributed object systems. CORBA provides a programming-language neutral Interface Definition Language (IDL) that describes the syntactic aspects of services supported by remote objects. However, CORBA IDL does not provide any means to specify the behavior of objects in an abstract and formal way. In the current practice, behavioral specification is provided either in plain English, or directly in the programming language chosen for the implementation. We propose to extend the CORBA interface definition of distributed objects by a behavioral specification based on high level Petri nets. We detail at the syntactic and semantic level how this formalism supports the features of the CORBA object model. We present a realistic case study to demonstrate our approach. Our technique allows specifying in an abstract, concise and precise way the behavior of CORBA servers, including internal concurrency and synchronization. As the behavioral specification is fully executable, this approach also enables to prototype and test a distributed object system as soon as the behaviors of individual objects have been defined. The paper discusses several implementation issues of the tool that supports the edition of models and their interactive excution. The high level of formality of the chosen formalism allows for mathematical analysis of behavioral specifications. Copyright © 2000 John Wiley & Sons, Ltd.

Software is increasingly being used to control safety-critical systems. Much research since Levesons fundamental article Software Safety: Why, What, and How (ACM Computing Surveys 18, 2 (1986), pp. 125163) has focused on ways to reduce or avoid software failures. However, the reliability of even the best-engineered software can be undermined by its user interface. Indeed, interface design for safety-critical interactive systems poses special challenges to the human-computer interaction community. This special issue addresses the challenge of analyzing, designing, and building reliable and usable safety-critical interactive systems. From a pragmatic point of view a safety-critical system is a system for which the cost of a failure is more important than the cost of developing the system. Safety-critical interactive systems add the human dimension to a software system by putting control into the hands of a human operator. Prominent examples of such control systems include nuclear power plants, railways systems, airplane cockpits, and military systems. Recent years have seen much effort put into the reengineering of the control system that is well represented in this special issueair traffic control. When compared to office automation systems, human-computer interac-tion for safety-critical interactive systems is both familiar and different. For instance, the management of a functionality like undo, that can be seen as a usability issue in an office automation system, can become a critical functionality when the user interacts with a safety-critical system. The three articles in this special issue provide three snapshots for how human-computer interaction issues play out in the broader field of safety-critical interactive systems. In the first article, Is Paper Safer? The Role of Flight Strips in Air Traffic Control, Wendy Mackay provides a detailed ethnographic study on how air traffic controllers work. As in Mackays article, the case study entails en-route air traffic control. An important contribution of this article is a method for an integrated analysis of three important methods of this field: task performance, analysis of user deviation and consequent hazard, and cooperation among users. Each of the three articles deals with the analysis and design phases of safety-critical interactive systems. If changes are to be made to large, complex, safety-critical control systems, the changes must be made early in the development lifecycle, where redesign in response to identified problems is feasible.This special issue arose from a CHI98 Workshop organized by Palanque and Paterno´ (“Designing User Interfaces for Safety-Critical Systems”, SIGCHI Bulletin 30, 4 ). The three articles included in this special issue were selected from more than a score of papers received. The editors thank and acknowledge their debt to the many qualified external reviewers from several countries who have helped select and improve (through their comments) the contributions in this special issue.

This paper aims at clarifying the articulation between the task models and system models encountered in CHI design practices. We demonstrate how the use of a formal task model may enhance the design of interactive systems, by providing quantitative results on which designers may base their decisions. We also demonstrate that it is possible to describe both task and system models within the same formal framework. This enables us firstly to formally prove that task and system models comply with each other, and secondly to perform quantitative analysis on the combination of task and system models. The approach is illustrated by a toy example which, despite its small size, allows us to develop both task and device models, and to perform several iterations of the design process. The device and tasks are modelled using the Interactive Cooperative Objects (ICO) formalism, which is based on Petri nets and on the object-oriented approach. The formality of Petri nets allows for axiomatic validation of isolated and interacting subsystems.