Traditional human-computer interaction takes place through formally-specified systems like structured UIs and programming languages. Recent AI systems promise a new set of informal interactions with computers through natural language and other notational forms. These informal interactions can then lead to formal representations, but depend upon pre-existing formalisms known to both humans and AI. What about novel formalisms and notations? How are new abstractions created, evolved, and incrementally formalized over time—and how might new systems, in turn, be explicitly designed to support these processes? We conduct a comparative historical analysis of notation development to identify some relevant characteristics. These include three social stages of notation development: invention & incubation, dispersion & divergence, and institutionalization & sanctification, as well as three functional stages: descriptive, generative, and evaluative. Within and across these stages, we detail several patterns, such as the role of linking and grounding metaphors, dimensions of meaningful variation, and analogical alignment. Finally, we offer some implications for design.

This work sheds light on whether and how creative writers' needs are met by existing research and commercial writing support tools (WST). We conducted a need finding study to gain insight into the writers' process during creative writing through a qualitative analysis of the response from an online questionnaire and Reddit discussions on r/Writing. Using a systematic analysis of 115 tools and 67 research papers, we map out the landscape of how digital tools facilitate the writing process. Our triangulation of data reveals that research predominantly focuses on the writing activity and overlooks pre-writing activities and the importance of visualization. We distill 10 key takeaways to inform future research on WST and point to opportunities surrounding underexplored areas. Our work offers a holistic and up-to-date account of how tools have transformed the writing process, guiding the design of future tools that address writers' evolving and unmet needs.

We define "visual story-writing" as using visual representations of story elements to support writing and revising narrative texts.To demonstrate this approach, we developed a text editor that automatically visualizes a graph of entity interactions, movement between locations, and a timeline of story events.Interacting with these visualizations results in suggested text edits: for example, connecting two characters in the graph creates an interaction between them, moving an entity updates their described location, and rearranging events on the timeline reorganizes the narrative sequence.Through two user studies on narrative text editing and writing, we found that visuals supported participants in planning high-level revisions, tracking story elements, and exploring story variations in ways that encourage creativity.Broadly, our work lays the foundation for writing support, not just through words, but also visuals.

We characterize and demonstrate how the principles of direct manipulation can improve interaction with large language models. This includes: continuous representation of generated objects of interest; reuse of prompt syntax in a toolbar of commands; manipulable outputs to compose or control the effect of prompts; and undo mechanisms. This idea is exemplified in DirectGPT, a user interface layer on top of ChatGPT that works by transforming direct manipulation actions to engineered prompts. A study shows participants were 50% faster and relied on 50% fewer and 72% shorter prompts to edit text, code, and vector images compared to baseline ChatGPT. Our work contributes a validated approach to integrate LLMs into traditional software using direct manipulation. Data, code, and demo available at https://osf.io/3wt6s.

Extracting underlying data from rasterized charts is tedious and inaccurate; values might be partially occluded or hard to distinguish, and the quality of the image limits the precision of the data being recovered. To address these issues, we introduce a semi-automatic system leveraging vector charts to extract the underlying data easily and accurately. The system is designed to make the most of vector information by relying on a drag-and-drop interface combined with selection, filtering, and previsualization features. A user study showed that participants spent less than 4 minutes to accurately recover data from charts published at CHI with diverse styles, thousands of data points, a combination of different encodings, and elements partially or completely occluded. Compared to other approaches relying on raster images, our tool successfully recovered all data, even when hidden, with a 78% lower relative error.

Documents often have paragraphs packed with numbers that are difficult to extract, compare, and interpret. To help readers make sense of data in text, we introduce the concept of Charagraphs: dynamically generated interactive charts and annotations for in-situ visualization, comparison, and manipulation of numeric data included within text. Three Charagraph characteristics are defined: leveraging related textual information about data; integrating textual and graphical representations; and interacting at different contexts. We contribute a document viewer to select in-text data; generate and customize Charagraphs; merge and refine a Charagraph using other in-text data; and identify, filter, compare, and sort data synchronized between text and visualization. Results of a study show participants can easily create Charagraphs for diverse examples of data-rich text, and when answering questions about data in text, participants were more correct compared to only reading text.

Inferential statistics are typically reported using p-values (NHST) or confidence intervals on effect sizes (estimation). This is done using a range of styles, but some readers have preferences about how statistics should be presented and others have limited familiarity with alternatives. We propose a system to interactively translate statistical reporting styles in existing documents, allowing readers to switch between interval estimates, p-values, and standardized effect sizes, all using textual and graphical reports that are dynamic and user customizable. Forty years of CHI papers are examined. Using only the information reported in scientific documents, equations are derived and validated on simulated datasets to show that conversions between p-values and confidence intervals are accurate. The system helps readers interpret statistics in a familiar style, compare reports that use different styles, and even validate the correctness of reports. Code and data: https://osf.io/x4ue7

Despite an abundance of carefully-crafted tutorials, trial-and-error remains many people’s preferred way to learn complex software. Yet, approaches to facilitate trial-and-error (such as tooltips) have evolved very little since the 1980s. While existing mechanisms work well for simple software, they scale poorly to large feature-rich applications. In this paper, we explore new techniques to support trial-and-error in complex applications. We identify key benefits and challenges of trial-and-error, and introduce a framework with a conceptual model and design space. Using this framework, we developed three techniques: ToolTrack to keep track of trial-and-error progress; ToolTrip to go beyond trial-and-error of single commands by highlighting related commands that are frequently used together; and ToolTaste to quickly and safely try commands. We demonstrate how these techniques facilitate trial-and-error, as illustrated through a proof-of-concept implementation in the CAD software Fusion 360. We conclude by discussing possible scenarios and outline directions for future research on trial-and-error.

Pointing transfer functions remain predominantly expressed in pixels per input counts, which can generate different visual pointer behaviors with different input and output devices; we show in a first controlled experiment that even small hardware differences impact pointing performance with functions defined in this manner. We also demonstrate the applicability of “hardware-independent” transfer functions defined in physical units. We explore two methods to maintain hardware-independent pointer performance in operating systems that require hardware-dependent definitions: scaling them to the resolutions of the input and output devices, or selecting the OS acceleration setting that produces the closest visual behavior. In a second controlled experiment, we adapted a baseline function to different screen and mouse resolutions using both methods, and the resulting functions provided equivalent performance. Lastly, we provide a tool to calculate equivalent transfer functions between hardware setups, allowing users to match pointer behavior with different devices, and researchers to tune and replicate experiment conditions. Our work emphasizes, and hopefully facilitates, the idea that operating systems should have the capability to formulate pointing transfer functions in physical units, and to adjust them automatically to hardware setups.

Documents such as presentations, instruction manuals, and research papers are disseminated using various file formats, many of which barely support the incorporation of interactive content. To address this lack of interactivity, we present Chameleon, a system-wide tool that combines computer vision algorithms used for image identification with an open database format to allow for the layering of dynamic content. Using Chameleon, static documents can be easily upgraded by layering user-generated interactive content on top of static images, all while preserving the original static document format and without modifying existing applications. We describe the development of Chameleon, including the design and evaluation of vision-based image replacement algorithms, the new document-creation pipeline as well as a user study evaluating Chameleon.

Incorporating touchscreen interaction into cockpit flight systems offers several potential advantages to aircraft manufacturers, airlines, and pilots. However, vibration and turbulence are challenges to reliable interaction. We examine the design space for braced touch interaction, which allows users to mechanically stabilise selections by bracing multiple fingers on the touchscreen before completing selection. Our goal is to enable fast and accurate target selection during high levels of vibration, without impeding interaction performance when vibration is absent. Three variant methods of braced touch are evaluated, using doubletap, dwell, or a force threshold in combination with heuristic selection criteria to discriminate intentional selection from concurrent braced contacts. We carried out an experiment to test the performance of these methods in both abstract selection tasks and more realistic flight tasks. The study results confirm that bracing improves performance during vibration, and show that doubletap was the best of the tested methods.

HCI researchers lack low latency and robust systems to support the design and development of interaction techniques using finger identification. We developed a low cost prototype using piezo based vibration sensors attached to each finger. By combining the events from an input device with the information from the vibration sensors we demonstrate how to achieve low latency and robust finger identification. Our prototype was evaluated in a controlled experiment, using two keyboards and a touchpad, showing recognition rates of 98.2% for the keyboard and, for the touchpad, 99.7% for single touch and 94.7% for two simultaneous touches. These results were confirmed in an additional laboratory style experiment with ecologically valid tasks. Last we present new interactions techniques made possible using this technology.

This paper investigates schedulability analysis for thermal-aware real-time systems. Thermal constraints are becoming more and more critical in new generation miniaturized embedded systems, e.g. Medicals implants. As part of this work, we adapt the PFPasapalgorithm proposed in [1] for energy-harvesting systems to thermal-aware ones. We prove its optimality for non-concrete1fixed-priority task sets and propose a response-time analysis based on worst-case response-time upper bounds. We evaluate the efficacy of the proposed bounds via extensive simulation over randomly-generated task systems.

Heterogeneous multicore platforms have become an attractive choice to deploy mixed criticality systems demanding diverse computational requirements. One of the major challenges is to efficiently harness the computational power of these multicore platforms while deploying mixed criticality applications. The problem is acerbated with an additional demand of energy efficiency. It is particularly relevant for the battery powered embedded systems. We propose a partitioning algorithm for unrelated heterogeneous multicore platforms to map mixed criticality applications that ensures the timeliness property and reduces the energy consumption.

The paper addresses the real-time fixed-priority scheduling problem for battery-powered embedded systems whose energy storage unit is replenished by an environmental energy source. In this context, a task may meet its deadline only if its cost of energy can be satisfied early enough. Hence, a scheduling policy for such a system should account for properties of the source of energy, capacity of the energy storage unit and tasks cost of energy. Classical fixed-priority schedulers are no more suitable for this model. Based on these motivations, we propose PFPASAPan optimal scheduling algorithm that handles both energy and timing constraints. Furthermore, we state the worst case scenario for non concrete task sets a non concrete task set is a set of real-time tasks whose offsets are known only at run-time scheduled with this algorithm and build a necessary and sufficient feasibility condition for non concrete task sets. Moreover, a minimal bound of the storage unit capacity that keeps a task set schedulable with PFPASAPis also proposed. Finally, we validate the proposed theory with large scale simulations and compare our algorithm with other existing ones.

In this paper, we propose feasibility and schedulability tests for a real-time scheduling problem under energy constraints. We first introduce the problem and show how to model it using timed automata. We then propose a feasibility test based on CTL model checking and schedulability tests for EDF and Preemptive Fixed Priority algorithms (PFP). Our approach also permits to generate a feasible schedule if one exists or otherwise to find how to correct battery characteristics to make the problem feasible. It is finally possible to generate schedules that optimize some criteria, such as the number of context switches between the battery recharging and discharging modes, the minimal and the maximal energy levels reached during the execution, or the number of preemptions. The approach is illustrated by some experiments using the model checking tool UPAAL [1].

Energy harvesting is the process of generating electrical energy from environmental sources such as solar panels. In recent years, this term has been frequently applied in the context of small autonomous devices such as wireless sensor nodes. The classical scheduling theory is insufficient for this kind of systems and new scheduling problems arise in this context. Until now, the research on this area focused in trying to improve the efficiency of existing algorithms. Our approach is to complete these efforts by a feasibility theory allowing us to understand why classical optimal algorithms are not efficient anymore with energy constraints.In this paper, we try to establish a schedulability test for a fixed priority real-time scheduling problem with energy constraints. We first introduce the problem and describe the model. Then, to illustrate the difficulty of the problem, we focus on a preemptive fixed priority scheduling policy where all the executions are postponed as long as possible. This policy lets the harvester the maximal amount of time to refill the battery. We call this policy PFPALAPfor As Late As Possible. We try to define sufficient and/or necessary schedulability conditions and discuss its potential optimality under some additional assumptions. Then, through simple counter examples, we show that intuitive assumptions are wrong for this scheduling problem, making it very interesting to study.

This paper presents a framework to design real-time event-based applications using Java. The real-time specification/or Java (RTSJ) is well designed for hard periodic real-time systems. Though it also proposes classes to model asynchronous events and deal with sporadic or aperiodic tasks, it remains insufficient. The literature proposes the use of periodic servers called task servers to handle non-periodic traffics in real-time systems. Unfortunately, there is no support for task servers in RTSJ. In order to fix this lack, we propose an RTSJ extension model. To validate our design, we adapt and implement two policies: the polling server and the deferrable server policies. To show how efficient these policies are, we compare implementation results and results obtained with a discrete-event-based simulator.

After having drawn up a state of the art on the theoretical feasibility of a system of periodic tasks scheduled by a preemptive algorithm at fixed priorities, we show in this article that temporal faults can occur all the same within a theoretically feasible system, that these faults can lead to a failure of the system and that we can use the data calculated during control of admission to install detectors of faults and to define a factor of tolerance. We show then the results obtained on a system of periodic tasks coded with Java real-time and carried out with the virtual machine jRate. These results show that the installation of the detectors and the tolerance to the faults makes an improvement of the behavior of the system in the presence of faults