Dietary counseling (DC) is effective for managing non-communicable diseases but faces scalability challenges due to reliance on self-reporting. While automated nutrition assessment has advanced, how to use passively captured dietary data to support dietitians’ decision-making remains underexplored. We use food purchase data (FPD) from loyalty cards as an example of non-standard dietary data in clinical practice. Building on multi-year digital nutrition research, we identify key FPD requirements regarding data availability and nutritional factors through a dietitian workshop (N=7) and survey (N=18). Based on derived insights, we design and evaluate DietCoach, the first dietitian decision support system integrating patient FPD and basic health data. Using DietCoach, 19 dietitians and 6 dietetics students provided dietary recommendations for three patient profiles with different FPD availability. The recommendation variances highlight the importance for system flexibility. Participants found DietCoach user-friendly and acceptable in their workflow, with no observed differences related to patient FPD availability in our evaluation. Our findings provide design insights for integrating opportunistic dietary data into clinical practice.

Hands-free control offers natural and intuitive interaction with devices, particularly in scenarios where traditional input methods are impractical. We introduce an extensible framework that integrates eye tracking, object detection, and gesture recognition to study intended and unintended interactions with Internet of Things (IoT) devices. To develop our framework, we conducted a structured experiment with 9 participants, focusing on identifying natural and intuitive interaction behaviors in different situations. The results showed that users intuitively combined gaze- and head-based gestures, showing the potential of head/gaze combinations as input mechanisms, specifically for directional movements. On this basis, we propose a system for hands-free interaction and control of IoT devices with intuitive gaze- and head-based gestures. We report on our promising findings as well as on limitations with respect to accurately distinguishing intention in real-world conditions. All our code is publicly available, ensuring the reproducibility and extension of our findings.

In pilot training, deviation from standard procedures is a significant concern. To provide student pilots with objective feedback in post-flight debriefing, we captured pilots’ view and gaze with the Pupil Core eye-tracker. Then we conducted a preliminary evaluation to test the feasibility of existing scene segmentation models for gaze-mapping. We used an OpenCV baseline model for coarse inside vs. outside-analysis, a fine-tuned Detectron2 model for specific instrument segmentation, and Segment Anything Models (SAM 2 and SAM 3) for human-in-the-loop analysis. The baseline was fast but fragile, failing in common flight scenarios; the Detectron2 model was powerful but inflexible and unsuitable for general use; and SAM 3 was promising, offering generalizability for post-flight analysis despite noisy digital displays. A qualitative preliminary evaluation of SAM with Visual Flight Rules shows that it can be beneficial in eye movement analysis. We identified poor data quality in bright cockpit environments and ergonomics as main limitations.

Board games often involve strategic decision making and procedural planning tasks. Such tasks require learners to make decisions based on dynamically evolving game state and changing information that is situated in a physical environment. Recommender systems can filter available information and provide learners with personalized and actionable suggestions that simplify their decision making while playing board games. Such recommendations can further be spatially aligned with relevant physical elements through Mixed Reality (MR). We present an MR system called GLAMRec for an engine-building strategy board game. GLAMRec provides personalized, transparent recommendations by integrating user data, real-time game state tracking, and ontology-based reasoning during a complex board game, which we use as a proxy environment for procedural learning tasks. We interviewed six board game designers to improve the GLAMRec and conducted a within-subjects design user study (N=32) to investigate how personalized explanations affect explanation satisfaction, user experience, and trust. We found that personalized recommendations significantly improve explanation satisfaction and hedonic user experience without affecting trust ratings, recommendation compliance, and game performance. These findings suggest that personalization primarily shaped perception of enjoyment rather than measurable learning outcomes or trust.

Personalization of online realities is today ubiquitous to support decision making or reduce information overload.Recently, through the expanding capabilities and pervasiveness of Mixed Reality and Ubiquitous Computing technologies, we observe increasing personalization also of physical reality.This might yield more convenient, efficient and inclusive everyday interactions.However, it may readily lead to serious societal consequences such as the loss of shared worlds and the emergence of perceptual filter bubbles.To mitigate such harms while retaining the benefits of personalization, it is important to understand how ubiquitous personalization systems may operate responsibly.Responding to this need, we propose a conceptual model that overcomes the limitations of established personalization models and expands their applicable scope to physical, virtual, and hybrid environments.We validated our model in relation to existing literature and show how it provides a conceptual foundation for the analysis and study of responsible personalization systems that create individually and societally beneficial Personalized Realities.

Industrial robots become increasingly prevalent, resulting in a growing need for intuitive, comforting human-robot collaboration. We present a user-aware robotic system that adapts to operator behavior in real time while non-intrusively monitoring physiological signals to create a more responsive and empathetic environment. Our prototype dynamically adjusts robot speed and movement patterns to proxemics while measuring operator pupil dilation. Our user study compares this adaptive system to a non-adaptive counterpart, and demonstrates that the adaptive system significantly reduces both perceived and physiologically measured cognitive load while enhancing usability. Participants reported increased feelings of comfort, safety, trust, and a stronger sense of collaboration when working with the adaptive robot. This highlights the potential of integrating real-time physiological data into human-robot interaction paradigms. This novel approach creates more intuitive and collaborative industrial environments where robots effectively 'read' and respond to human cognitive states, and we feature all data and code for future use.

Unhealthy dietary habits are a major preventable risk factor for widespread non-communicable diseases (NCDs). Diet counseling is effective in managing diet-related NCDs, but constrained by its manual nature and limited (clinical) resources. To address these challenges, we propose FoodCoach, a fully automated diet counseling system that monitors people's food purchases using digital receipts from loyalty cards and provides structured dietary recommendations. We introduce the FoodCoach system's dietary recommender algorithm and architecture, alongside evaluation results from a two-arm randomized controlled trial involving 61 participants. The trial results demonstrate the technical feasibility and potential for scalable, fully automated diet counseling, despite not showing a significant change in participants' food purchase healthiness. We further show how to deploy and extend the FoodCoach system in new contexts, provide all relevant source code, and discuss how to verify and enhance the system efficacy. Our core research contributions are: 1) a novel dietary recommender algorithm designed and implemented with clinical nutritional experts, and 2) a scalable system architecture that employs a knowledge graph for enhanced interoperability and applicability to diverse domains and data sources. From a practical perspective, FoodCoach can augment clinical diet counseling through novel insights about patient food purchases and continuous support between consultations. Its cost-effective automated recommendations can also benefit the general public by helping combat NCD.

Electromechanical systems manage physical processes through a network of inter-connected components. Today, programming the interactions required for coordinating these components is largely a manual process. This process is time-consuming and requires manual adaptation when system features change. To overcome this issue, we use autonomous software agents that process semantic descriptions of the system to determine coordination requirements and constraints; on this basis, they then interact with one another to control the system in a decentralized and coordinated manner. Our core insight is that coordination requirements between individual components are, ultimately, largely due to underlying physical interdependencies between the components, which can be (and, in many cases, already are) semantically modeled in automation projects. Agents then use hypermedia to discover, at run time, the plans and protocols required for enacting the coordination. A key novelty of our approach is the use of hypermedia-driven interaction: it reduces coupling in the system and enables its run-time adaptation as features change.

Humans can easily comprehend and explain the dynamics of a scene by observing the evolution of relationships among identified objects over time- while research towards Hybrid Intelligence promotes the integration of human and machine capabilities, this ability is currently beyond the capabilities of automated systems. Toward realizing it in automated scene understanding systems, we present interpretable Object Identification using Contextual Information System (OIC). Given a video, OIC detects, identifies, and tracks objects and their relationships over time to answer questions about the analyzed scene. Moreover, OIC makes predictions and infers the actors' intentions in a scene. To achieve this, our approach generates a scene graph containing classified objects and their semantic relationships. It then computes a Frame Graph by adding Contextually Unique IDentifiers (CUIDs) to each of the detected objects in the scene graph; the CUIDs permit tracking multiple object instances over time, even if the objects are visually identical. The CUIDs are then used to connect objects across a sequence of Frame Graphs, generating a Temporal Graph. This graph is exported as a cue for a pretrained Large Language Model to provide assistance and answer user questions. OIC's modular architecture enables simple comprehension and swapping of its components, making OIC more interpretable and maintainable than end-to-end scene understanding systems. Our quantitative and qualitative evaluation results demonstrate the effectiveness of OIC as a viable next step toward interpretable automated scene understanding systems.

The future of proactive Digital Companions (DCs)-smart agents capable of assisting and protecting their users-lies in their ability to collaborate effectively with users, learn their preferences, and adjust their behavior according to the user's current state and their environment. To achieve this, DCs must carefully strike a balance between acting autonomously, while keeping users informed to minimize inconveniences, thereby enhancing user acceptance. In this article, we conduct a user survey to enrich the architecture of proactive personal DCs that explores the trade-off between full autonomous functioning and ensuring sufficient user control in various critical and non-critical scenarios. Our findings indicate that most of the participants lean towards having control over the actions of DCs, and will actively collaborate with those systems in everyday situations, in which decisions are not urgent. However, participants would not mind yielding their control to DCs in time-sensitive or urgent scenarios. Furthermore, in our survey, participants highlighted the importance of explaining such actions well. Given the results from our survey, we implemented a system prototype with the enriched architecture of an explainable and unobtrusive proactive DC for a smart home environment. The actions of this DC are not always fully autonomous and follow our survey findings.

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they provide insights into user attention, intentions, and activities, and allow novel interaction methods based on this information. However, in physical environments, the implications of using gaze-enabled AR for human activity recognition have not been explored in detail. In an experimental study with the Microsoft HoloLens 2, we collected gaze data from 20 users while they performed three activities: Reading a text, Inspecting a device, and Searching for an object. We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved up to 89.6% activity-recognition accuracy. Based on the recognized activity, our system—GEAR—then provides users with relevant AR feedback. Due to the sensitivity of the personal (gaze) data GEAR collects, the system further incorporates a novel solution based on the Solid specification for giving users fine-grained control over the sharing of their data. The provided code and anonymized datasets may be used to reproduce and extend our findings, and as teaching material.

Humans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they allow novel interaction methods and provide insights into user attention, intentions, and activities. However, only few studies have used gaze-enabled AR displays for human activity recognition (HAR). In an experimental study, we collected gaze data from 10 users on a HoloLens 2 (HL2) while they performed three activities (i.e., read, inspect, search). We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved an up to 98.7% activity-recognition accuracy. On the HL2, we provided users with an AR feedback that is relevant to their current activity. We present the components of our system (GEAR) including a novel solution to enable the controlled sharing of collected data. We provide the scripts and anonymized datasets which can be used as teaching material in graduate courses or for reproducing our findings.

Interactive systems can adapt to individual users to increase productivity, safety, or acceptance. Previous research focused on different factors, such as cognitive workload (CWL), to better understand and improve the human-computer or human-robot interaction (HRI). We present results of an HRI experiment that uses pupillometry to measure users’ responses to robot movements. Our results demonstrate a significant change in pupil dilation, indicating higher CWL, as a result of increased movement speed of an articulated robot arm. This might permit improved interaction ergonomics by adapting the behavior of robots or other devices to individual users at run time.

Over the past two decades, Cyber-Physical Systems (CPS) have emerged as critical components in various industries, integrating digital and physical elements to improve efficiency and automation, from smart manufacturing and autonomous vehicles to advanced healthcare devices. However, the increasing complexity of CPS and their deployment in highly dynamic contexts undermine user trust. This motivates the investigation of methods capable of generating explanations about the behavior of CPS. To this end, Explainable Artificial Intelligence (XAI) methodologies show potential. However, these approaches do not consider contextual variables that a CPS may be subjected to (e.g., temperature, humidity), and the provided explanations are typically not actionable. In this article, we propose an Actionable Contextual Explanation System (ACES) that considers such contextual influences. Based on a user query about a behavioral attribute of a CPS (for example, vibrations and speed), ACES creates contextual explanations for the behavior of such a CPS considering its context. To generate contextual explanations, ACES uses a context model to discover sensors and actuators in the physical environment of a CPS and obtains time-series data from these devices. It then cross-correlates these time-series logs with the user-specified behavioral attribute of the CPS. Finally, ACES employs a counterfactual explanation method and takes user feedback to identify causal relationships between the contextual variables and the behavior of the CPS. We demonstrate our approach with a synthetic use case; the favorable results obtained, motivate the future deployment of ACES in real-world scenarios.

Smart products, such as toy robots, must comply with multiple legal requirements of the countries they are sold and used in. Currently, compliance with the legal environment requires manually customizing products for different markets. In this paper, we explore a design approach for smart products that enforces compliance with aspects of the European Union’s data protection principles within a product’s firmware through a toy robot case study. To this end, we present an exchange between computer scientists and legal scholars that identified the relevant data flows, their processing needs, and the implementation decisions that could allow a device to operate while complying with the EU data protection law. By designing a data-minimizing toy robot, we show that the variety, amount, and quality of data that is exposed, processed, and stored outside a user’s premises can be considerably reduced while preserving the device’s functionality. In comparison with a robot designed using a traditional approach, in which 90% of the collected types of information are stored by the data controller or a remote service, our proposed design leads to the mandatory exposure of only 7 out of 15 collected types of information, all of which are legally required by the data controller to demonstrate consent. Moreover, our design is aligned with the Data Privacy Vocabulary, which enables the toy robot to cross geographic borders and seamlessly adjust its data processing activities to the local regulations.

Surveillance in private and public spaces provides observers with information that can enhance protection and efficiency but usually infringes upon the privacy of the individuals and groups. These informational privacy risks are centered on users' perceived and design-induced threats. They cannot be removed completely but can be minimized using suitable anonymization techniques. To minimize the users' informational privacy threats, we designed a privacy-aware surveillance system that gives the users leverage over the anonymization filters, to physically adjust the opaqueness of the camera lens used in the prototype according to their privacy requirements. We implement our prototype in the context of office space surveillance, where the proposed solution considers privacy requirements in such environments to improve users' trust in the surveillance system and reduce their privacy concerns.

User interfaces rarely adapt to the specific user preferences or the task at hand. We present a method that allows to quickly and inexpensively create personalized interfaces from plain paper. Users can cut out shapes and assign control functions to these paper snippets via a simple configuration interface. After configuration, control takes place entirely through the manipulation of the paper shapes, providing the experience of a tailored tangible user interface. The shapes and assignments can be dynamically changed during use. Our system is based on markerless tracking of the user's fingers and the paper shapes on a surface using an RGBD camera mounted above the interaction space, which is the only hardware sensor required. Our approach and system are backed up by two studies where we determined what shapes and interaction abstractions users prefer, and verified that users can indeed employ our system to build real applications with paper snippet interfaces.

Electrical substations are crucial elements of Smart Grids (SGs), where they are mainly responsible for voltage transformations. However, due to the integration of distributed energy resources in the grid, substations now have to provide additional grid management capabilities which in turn require supervision and automation solutions for large low-voltage grids. A recurring challenge in such deployments are siloed systems that are due to noninteroperable communication protocols across substations: although most substations' communication is based on the International Electrotechnical Commission (IEC) 61850 standard, deployed legacy protocols lag behind modern communication technologies in terms of performance, hindering the full transition to lightweight protocols. This paper demonstrates that IEC 61850 can be fully mapped to the Constrained Application Protocol (CoAP) in combination with the Concise Binary Object Representation (CBOR) format while improving system performance compared to existing alternatives [e.g., WS-SOAP and Hypertext Transfer Protocol (HTTP)]. On average, CoAP+CBOR needs 44% and 18% of the message size and 71% and 85% of the time compared to systems based on HTTP and WS-* Web services, respectively-this is especially relevant for resource-constrained devices and networks in electrical grids. In addition, CoAP is based on the Representational State Transfer (REST) architectural style, which supports system integration and interoperability through uniform identification and interaction. This approach fosters the standard-compliant integration of legacy platforms with modern substations as well as current IoT systems in neighboring domains, such as building management and infrastructure automation systems.

With the advent of Smart Grid systems, the digitalization of electrical grid infrastructures aims to improve energy saving and efficiency. The International Electrotechnical Commission (IEC) is one of the organizations that create and manage norms and standards in areas related to electricity and electronics, such as IEC 61850. In recent years, the research community have proposed mappings of different communication protocols to IEC 61850. However, most of the proposals focused on heavyweight interaction paradigms and protocols such as Common Object Request Broker Architecture (CORBA), Data Distribution Service (DDS), HTTP-REST or Web Services. With the proliferation of the Internet of Things (IoT), new lightweight protocols are appearing opening new perspectives to the standard. Hence, this paper firstly presents a validation of a mapping of the IEC 61850 standard to the Constrained Application Protocol (CoAP) and then compares its performance implementing the IEC's mapping against HTTP-REST and Web Services/SOAP. For comparison purposes, the communication latency, the number of total bytes sent, and the size of the overhead are presented for each of the three approaches. To conclude, future perspectives on the suitability of lightweight protocols to the IEC 61850 are provided.

One of the central research challenges in the Internet of Things and Ubiquitous Computing domains is how users can be enabled to “program” their personal and industrial smart environments by combining services that are provided by devices around them. We present a service composition system that enables the goal-driven configuration of smart environments for end users by combining semantic metadata and reasoning with a visual modeling tool. In contrast to process-driven approaches where service mashups are statically defined, we make use of embedded semantic API descriptions to dynamically create mashups that fulfill the user’s goal. The main advantage of our system is its high degree of flexibility, as service mashups can adapt to dynamic environments and are fault-tolerant with respect to individual services becoming unavailable. To support users in expressing their goals, we integrated a visual programming tool with our system that allows to model the desired state of a smart environment graphically, thereby hiding the technicalities of the underlying semantics. Possible applications of the presented system include the management of smart homes to increase individual well-being, and reconfigurations of smart environments, for instance in the industrial automation or healthcare domains.

With ever more everyday objects becoming “smart” due to embedded processors and communication capabilities, the provisioning of intuitive user interfaces to control smart things is quickly gaining importance. We present a model-based interface description scheme that enables automatic, modality-independent user interface generation. User interface description languages based on our approach carry enough information to suggest intuitive interfaces while still being easily producible for developers. This is enabled by describing the atomic interactive components of a device and capturing the semantics of interactions with the device. We propose a taxonomy of abstract sensing and actuation primitives and present a smartphone application that can act as a ubiquitous device controller. An evaluation of the mobile application in a laboratory setup, home environments, and an educational setting as well as the results of a user study highlight the accessibility of the proposed scheme for application developers and its suitability for controlling smart devices.

In this position paper we describe how technologies from the Web of Things domain could help to simplify and automate the interaction between robotic devices and their surroundings. Specifically, we discuss how Web patterns like Resource-oriented Architectures or Representational State Transfer together with semantic metadata could help to create environments where robots seamlessly interact with other devices in their vicinity: Using services provided by other devices and offering services themselves. One of the main goals in the Web of Things community is to furnish smart environments with sensors and actuators that offer self-described interfaces and are openly accessible from other devices -- Robots should be enabled to make use of this wealth of instrumentation of their surroundings!

As the Internet of Things finds its way into private households, more and more everyday objects communicate with services that are running inside the home and on the Internet. For individuals to trust their smart homes, they should be aware of possibly privacy-sensitive data flows and control commands. In this demo paper, we present a system that combines a real time network analysis tool with an augmented reality user interface to visualize data streams within the home network and to remote services. Our system requires no modifications to a typical home network infrastructure, as it operates by merely observing packets sent over the network.