Current deep learning models are not designed to simultaneously address three fundamental questions: predict class labels to solve a given classification task (the "What?"), simulate changes in the situation to evaluate how this impacts class predictions (the "How?"), and imagine how the scenario should change to result in different class predictions (the "Why not?"). While current approaches in causal representation learning and concept interpretability are designed to address some of these questions individually (such as Concept Bottleneck Models, which address both ``what'' and ``how'' questions), no current deep learning model is specifically built to answer all of them at the same time. To bridge this gap, we introduce CounterFactual Concept Bottleneck Models (CF-CBMs), a class of models designed to efficiently address the above queries all at once without the need to run post-hoc searches. Our experimental results demonstrate that CF-CBMs: achieve classification accuracy comparable to black-box models and existing CBMs (“What?”), rely on fewer important concepts leading to simpler explanations (“How?”), and produce interpretable, concept-based counterfactuals (“Why not?”). Additionally, we show that training the counterfactual generator jointly with the CBM leads to two key improvements: (i) it alters the model's decision-making process, making the model rely on fewer important concepts (leading to simpler explanations), and (ii) it significantly increases the causal effect of concept interventions on class predictions, making the model more responsive to these changes.

Causal opacity denotes the difficulty in understanding the "hidden" causal structure underlying the decisions of deep neural network (DNN) models. This leads to the inability to rely on and verify state-of-the-art DNN-based systems, especially in high-stakes scenarios. For this reason, circumventing causal opacity in DNNs represents a key open challenge at the intersection of deep learning, interpretability, and causality. This work addresses this gap by introducing Causal Concept Graph Models (Causal CGMs), a class of interpretable models whose decision-making process is causally transparent by design. Our experiments show that Causal CGMs can: (i) match the generalisation performance of causally opaque models, (ii) enable human-in-the-loop corrections to mispredicted intermediate reasoning steps, boosting not just downstream accuracy after corrections but also the reliability of the explanations provided for specific instances, and (iii) support the analysis of interventional and counterfactual scenarios, thereby improving the model's causal interpretability and supporting the effective verification of its reliability and fairness.

Federated Learning (FL) enables collaborative model training across multiple clients while preserving data privacy. Traditional FL methods often use a global model to fit all clients, assuming that clients' data are independent and identically distributed (IID). However, when this assumption does not hold, the global model accuracy may drop significantly, limiting FL applicability in real-world scenarios. To address this gap, we propose FLUX, a novel clustering-based FL (CFL) framework that addresses the four most common types of distribution shifts during both training and test time. To this end, FLUX leverages privacy-preserving client-side descriptor extraction and unsupervised clustering to ensure robust performance and scalability across varying levels and types of distribution shifts. Unlike existing CFL methods addressing non-IID client distribution shifts, FLUX i) does not require any prior knowledge of the types of distribution shifts or the number of client clusters, and ii) supports test-time adaptation, enabling unseen and unlabeled clients to benefit from the most suitable cluster-specific models. Extensive experiments across four standard benchmarks, two real-world datasets and ten state-of-the-art baselines show that FLUX improves performance and stability under diverse distribution shifts—achieving an average accuracy gain of up to 23 percentage points over the best-performing baselines—while maintaining computational and communication overhead comparable to FedAvg.

Human Activity Recognition (HAR) benefits various application domains, including health and elderly care. Traditional HAR involves constructing pipelines reliant on centralized user data, which can pose privacy concerns as they necessitate the uploading of user data to a centralized server. This work proposes multi-frequency Federated Learning (FL) to enable: (1) privacy-aware ML; (2) joint ML model learning across devices with varying sampling frequency. We focus on head-worn devices (e.g., earbuds and smart glasses), a relatively unexplored domain compared to traditional smartwatch- or smartphone-based HAR. Results have shown improvements on two datasets against frequency-specific approaches, indicating a promising future in the multi-frequency FL-HAR task. The proposed network’s implementation is publicly available for further research and development.**

Federated Learning (FL), a privacy-aware approach in distributed deep learning environments, enables many clients to collaboratively train a model without sharing sensitive data, thereby reducing privacy risks. However, enabling human trust and control over FL systems requires understanding the evolving behaviour of clients, whether beneficial or detrimental for the training, which still represents a key challenge in the current literature. To address this challenge, we introduce Federated Behavioural Planes (FBPs), a novel method to analyse, visualise, and explain the dynamics of FL systems, showing how clients behave under two different lenses: predictive performance (error behavioural space) and decision-making processes (counterfactual behavioural space). Our experiments demonstrate that FBPs provide informative trajectories describing the evolving states of clients and their contributions to the global model, thereby enabling the identification of clusters of clients with similar behaviours. Leveraging the patterns identified by FBPs, we propose a robust aggregation technique named Federated Behavioural Shields to detect malicious or noisy client models, thereby enhancing security and surpassing the efficacy of existing state-of-the-art FL defense mechanisms. Our code is publicly available on GitHub.

Accurate Cognitive Workload (CW) estimation, crucial in mobile healthcare and human-machine interaction, is impeded by client heterogeneity, data limitations, and privacy concerns, especially in the presence of Out-of-Distribution (OoD) clients. This study proposes a robust framework that is based on Federated Learning to protect data privacy, and utilizes context-based STRNet to enable joint cross-user learning on heterogeneous datasets, enhancing model generalisability. The framework includes a novel Unsupervised Client Personalisation strategy that prevents accuracy loss in OoD clients. We tested our framework on two publicly available CW datasets, COLET and ADABase. The framework improved the accuracy of centralized approaches while preserving data privacy. The framework is model-agnostic, efficient, and enables unsupervised personalisation for each client, bolstering the quality and robustness of the end-to-end deep learning models.

Human physiological monitoring has become easily accessible by integrating wearable devices into our lives, providing valuable real-time data. Methods for Cognitive Workload (CW) estimation utilize such physiological data to quantify CW during task execution. These methods are crucial for various domains, including mobile healthcare, forecasting human errors, and human-machine interaction. However, accurately estimating CW continues to pose a challenge due to the absence of objective ground truth data, context dependency, and the privacy sensitivity of the data. This study tackled the complex task of estimating CW based on privacy-sensitive data (e.g., eye movement, pupil diameter, blink information, and other physiological signals) using Federated Learning (FL) methods to improve user privacy. We compared the outcomes of the FL models with the more conventional centralized approach on two publicly-available datasets COLET and ADABase, which include data from 75 participants overall. The results highlight the efficacy of FL in collaboratively training a global (person-independent) model. The FL models achieved performances on par with centralized state-of-the-art models while preserving data privacy. Recognizing the importance of privacy in user sensing, FL presents a promising approach that enables wearable sensing applications in privacy-sensitive domains.

Generating counterfactuals to discover hypothetical predictive scenarios is the de facto standard for explaining machine learning models and their predictions. However, building a counterfactual explainer that is time-efficient, scalable, and model-agnostic, in addition to being compatible with continuous and categorical attributes, remains an open challenge. To complicate matters even more, ensuring that the contrastive instances are optimised for feature sparsity, remain close to the explained instance, and are not drawn from outside of the data manifold, is far from trivial. To address this gap we propose BayCon: a novel counterfactual generator based on probabilistic feature sampling and Bayesian optimisation. Such an approach can combine multiple objectives by employing a surrogate model to guide the counterfactual search. We demonstrate the advantages of our method through a collection of experiments based on six real-life datasets representing three regression tasks and three classification tasks.

Widespread sensing devices enable a world in which physical spaces become personalised in the presence of mobile users. An important example of such personalisation is the use of pervasive displays to show content that matches the requirements of proximate viewers. Despite prior work on prototype systems that use mobile devices to personalise displays, no significant attempts to trial such systems have been carried out. In this article, we report on our experiences of designing, developing and operating the world’s first comprehensive display personalisation service for mobile users. Through a set of rigorous quantitative measures and 11 potential user/stakeholder interviews, we demonstrate the success of the platform in realising display personalisation, and offer a series of reflections to inform the design of future systems.

Sharing economy services have become increasingly popular. In addition to various well-known for-profit activities in this space (e.g., ride and apartment sharing), many community groups and non-profit organizations offer collections of shared things (e.g., books, tools) that explicitly aim to benefit local communities. We expect that both non-profit and for-profit approaches will see an increased use in the future. To support designers in devising new sharing economy services, we developed the Sharing Economy Design Cards, a design toolkit in the form of a card deck. We present two deployments of the cards: (1) in individual interviews with 16 designers and sharing economy domain experts; and (2) in two workshops with 5 participants each. Our findings show that the use of the cards not only facilitates the creation of future sharing platforms and services in a collaborative setting, but also helps to evaluate existing sharing economy services as an individual activity.

A plethora of sensors embedded in wearable, mobile, and infrastructure devices allow us to seamlessly capture large parts of our daily activities and experiences. It is not hard to imagine that such data could be used to support human memory in the form of automatically generated memory cues, e.g., images, that help us remember past events. Such a vision of pervasive "memory-augmentation systems", however, comes with significant privacy and security implications, chief among them the threat of memory manipulation: without strong guarantees about the provenance of captured data, attackers would be able to manipulate our memories by deliberately injecting, removing, or modifying captured data. This work introduces this novel threat of human memory manipulation in memory augmentation systems. We then present a practical approach that addresses key memory manipulation threats by securing the captured memory streams. Finally we report evaluation results on a prototypical secure camera platform that we built.

Machine ethics is a key challenge in times when digital systems play an increasing role in people's lives. At the core of machine ethics is the handling of personal data and the security of machine operations. Yet, privacy and security engineering are a challenge in today's business world where personal data markets, corporate deadlines, and a lack of perfectionism frame the context in which engineers need to work. Besides these organizational and market challenges, each engineer has his or her specific view on the importance of these values that can foster or inhibit taking them into consideration. We present the results of an empirical study of 124 engineers based on the Theory of Planned Behavior and Jonas' Principle of Responsibility to understand the drivers and impediments of ethical system development as far as privacy and security engineering are concerned. We find that many engineers find the two values important, but do not enjoy working on them. We also find that many struggle with the organizational environment: They face a lack of time and autonomy that is necessary for building ethical systems, even at this basic level. Organizations' privacy and security norms are often too weak or even oppose value-based design, putting engineers in conflict with their organizations. Our data indicate that it is largely engineers' individually perceived responsibility as well as a few character traits that make a positive difference to ethical system development.

An increasing number of non-profit groups and organizations have formed "libraries" of shared things to leverage the collaborative use of underutilized resources (e.g., power tools) for the benefit of local communities. Their key challenges are the transience and anonymity of their members, and how to nurture creative interactions among them. We designed and developed Roaming Objects, an interactive system aimed at supporting the capture and sharing of equipment-use experiences among these members. We deployed the system for two months in a tool-sharing cooperative to explore how it may help to address these challenges. We offer insights into how resource sharing cooperatives and collectives could be better supported, by proposing design opportunities that facilitate sharing both physical objects and digital information about their use.

Today's sensor-rich mobile and wearable devices allow us to seamlessly capture an increasing amount of our daily experiences in digital format. This process can support human memory by producing "memory cues", e.g., an image or a sound that can help trigger our memories of a past event. However, first-person captures such as those coming from wearable cameras are not always ideal for triggering remembrance. One interesting option is thus to combine our own capture streams with those coming from co-located peers, in or even infrastructure sensors (e.g., a surveillance camera) in order to create more powerful memory cues. Given the significant privacy and security concerns of a system that shares personal experience streams with co-located peers, we developed a tangible user interface (TUI) that allows users to in-situ control the capture and sharing of their experience streams through a set of five physical gestures. We report on the design of the device, as well as the results of a user study with 20 participants that evaluated its usability and efficiency in the context of a meeting capture. Our results show that our TUI outperforms a comparable smartphone application, but also uncovers user concerns regarding the need for additional control devices.

Winter sports like skiing and snowboarding are often group activities. Groups of skiers and snowboarders traditionally use paper maps or board-mounted larger-scale maps near ski lifts to aid decision making: which slope to take next, where to have lunch, or what hazards to avoid when going off-piste. To enrich those static maps with personal content (e.g., pictures, prior routes taken, or hazards encountered), we developed SkiAR - a wearable augmented reality system that allows groups of skiers and snowboarders to share such content on a printed panoramic resort map. The contribution of our work is twofold: (1) we developed a system that offers a novel way to review and share personal content in situ while on the slope using a resort map; (2) we report on the results from a qualitative analysis of two user studies to inform the design and validate the usability and perceived usefulness of our prototype.

Mobile and wearable devices allow people to capture different aspects of their life experiences (e.g. family holidays, work meetings, running activities, etc.) in the form of photos, videos, physiological data, etc. An interesting avenue to explore is the usage of such captured experiences to support and augment human memory. Experiences of different events can be used to generate retrieval memory cues in order to trigger recall of those recorded events. In addition, captured experiences can be shared with other (co-located) people of the same event. The focus of this work is on understanding the privacy challenges with regard to using and sharing captured experiences for memory augmentation purposes. With the ultimate goal of an usage control model for the protection of personal memory cues, here we provide insights on: how sharing captured experiences is different from sharing experiences in social media networks, and what are some challenges in designing an usage control model for memory cues.

Several mobile and social media weather apps support the incorporation of human input in increasing their coverage and accuracy of current weather conditions. This practice is also known as participatory sensing: the act of using sensors (i.e. smartphones) carried by volunteers to acquire highly localized measurements of physical phenomena. In order to assess the accuracy of such user contributed weather reports, we created an android app called Atmos that allows for the in situ collection of weather data in the form of descriptive manual input. Based on a yearlong study with Atmos deployed on the Google Play store, we investigate the ability of mobile users to both report current weather conditions accurately, and to predict future weather developments. We found that mobile users can be sufficiently accurate when reporting current conditions, though report accuracy was affected by hour of day. Users were also able to provide accurate short-term predictions, particularly for temperature and wind intensity. We also present results from an online survey that gathered data from 12 countries in order to understand the role weather plays in users' daily life, which helped us design Atmos.

There is increasing interest in using digital signage to deliver highly personalised content. However, display personalization presents a number of architectural design challenges in particular, how best to provide personalisation without unduly compromising viewers' privacy. While previous research has focused on understanding specific elements of the overall vision, our work presents details of the first significant attempt at a system that integrates future pervasive display networks and mobile devices to support display personalisation. We describe a series of usage models and design goals for display personalisation and then present Tacita, a system that supports these models and goals. Our architecture includes mobile, display and cloud-based elements and provides comprehensive personalisation features while preventing the creation of user profiles within the display infrastructure, thus helping to preserve users' privacy. An initial evaluation of our prototype implementation of the architecture is also included and demonstrates the viability of the Tacita approach.

This paper presents BoD Shapes, a novel authentication method for smartphones that uses the back of the device for input. We argue that this increases the resistance to shoulder surfing while remaining reasonably fast and easy-to-use. We performed a user study (n=24) comparing BoD Shapes to PIN authentication, Android grid unlock, and a front version of our system. Testing a front version allowed us to directly compare performance and security measures between front and back authentication. Our results show that BoD Shapes is significantly more secure than the three other approaches. While performance declined, our results show that BoD Shapes can be very fast (up to 1.5 seconds in the user study) and that learning effects have an influence on its performance. This indicates that speed improvements can be expected in long-term use.

End-users have become accustomed to the ease with which online systems allow them to exchange messages, pictures, and other files with colleagues, friends, and family. This con- venience, however, sometimes comes at the expense of hav- ing their data be viewed by a number of unauthorized par- ties, such as hackers, advertisement companies, other users, or governmental agencies. A number of systems have been proposed to protect data shared online; yet these solutions typically just shift trust to another third party server, are platform specific (e.g., work for Facebook only), or fail to hide that confidential communication is taking place. In this paper, we present a novel system that enables users to exchange data over any web-based sharing platform, while both keeping the communicated data confidential and hiding from a casual observer that an exchange of confidential data is taking place. We provide a proof-of-concept implementa- tion of our system in the form of a publicly available Fire- fox plugin, and demonstrate the viability of our approach through a performance evaluation.

The proliferation of digital signage systems has prompted a wealth of research that attempts to use public displays for more than just advertisement or transport schedules, such as their use for supporting communities. However, deploying and maintaining display systems “in the wild” that can support communities is challenging. Based on the authors’ experiences in designing and fielding a diverse range of community-supporting public display deployments, we identify a large set of challenges and issues that researchers working in this area are likely to encounter. Grouping them into five distinct layers -- (1) hardware, (2) system architecture, (3) content, (4) system interaction, and (5) community interaction design -- we draw up the P-LAYERS framework to enable a more systematic appreciation of the diverse range of issues associated with the development, the deployment, and the maintenance of such systems. Using three of our own deployments as illustrative examples, we will describe both our experiences within each individual layer, as well as point out interactions between the layers. We believe our framework provides a valuable aid for researchers looking to work in this space, alerting them to the issues they are likely to encounter during their deployments, and help them plan accordingly.

The proliferation of social networking sites and mobile technology allows us to check on our friends and family, follow what experts in our field think, or simply 'check-in' online. While in many ways advantageous, the ability to be constantly connected is significantly affecting our offline interaction behavior. People sharing a table today might ignore each other for stretches at a time in order to interact with far-away friends through mobile technology instead. The goal of this workshop is to examine how we can build technologies that promote offline interactions. We plan to discuss how offline interactions can be spurred within different social groups and different settings through currently available devices and technologies. We also plan to explore how such technologies can be built and used for different types of offline engagement (e.g., playful vs. serious). The workshop aims to establish a community interested in computer mediated offline interaction.

No abstract available.

In the age of online social networks, local communities still play an essential role in supporting social cohesion. In this paper we present a study that explores the design of "interacting places" -- networked public multimedia services that foster community awareness between local members -- in the context of a student community. In order to have interacting places "fit in" with the existing communication practices of the students, we performed and analyzed a set of semi-structured interviews with n=17 students regarding their use of email, social networking services, and instant messaging to stay in touch with others. A follow-up online survey (n=76) then explored how networked public multimedia services could complement these practices. Following a "communicative ecology" approach -- a conceptual model that represents the technical, social, and discursive contexts of communication -- we draw up guidelines to support the design of both content and channels (applications) for interacting places in student communities.

Public displays are becoming a ubiquitous resource in the urban environments due to significant price drops of large LCD panels. However, most public displays are still displaying simple advertisements in the form of slide shows or movie clips, instead of offering locally customized content that resonates with passers-by. So far, creating such customized content has been expensive. As a possible solution we present the concept and architecture for autopoiesic content, i.e., self-generative content that is automatically created by matching local context information with regular scheduled information into content that is highly localized. In this paper we report on the design, operation, and user experience of FunSquare -- an application that uses autopoiesic content to present localized "fun facts" in order to strengthen the feeling of community.

Recent years have seen a proliferation of secure device pairing methods that try to improve both the usability and security of today's de-facto standard -- PIN-based authentication. Evaluating such improvements is difficult. Most comparative laboratory studies have so far mainly focused on completeness, trying to find the single best method among the dozens of proposed approaches -- one that is both rated the most usable by test subjects, and which provides the most robust security guarantees. This search for the "best" pairing method, however, fails to take into account the variety of situations in which such pairing protocols may be used in real life. The comparative study reported here, therefore, explicitly situates pairing tasks in a number of more realistic situations. Our results indicate that people do not always use the easiest or most popular method -- they instead prefer different methods in different situations, based on the sensitivity of data involved, their time constraints, and the social conventions appropriate for a particular place and setting. Our study also provides qualitative data on factors influencing the perceived security of a particular method, the users' mental models surrounding security of a method, and their security needs.

With the increase of automated teller machine (ATM) frauds, new authentication mechanisms are developed to overcome security problems of personal identification numbers (PIN). Those mechanisms are usually judged on speed, security, and memorability in comparison with traditional PIN entry systems. It remains unclear, however, what appropriate values for PIN-based ATM authentication actually are. We conducted a field study and two smaller follow-up studies on real-world ATM use, in order to provide both a better understanding of PIN-based ATM authentication, and on how alternative authentication methods can be compared and evaluated. Our results show that there is a big influence of contextual factors on security and performance in PIN-based ATM use. Such factors include distractions, physical hindrance, trust relationships, and memorability. From these findings, we draw several implications for the design of alternative ATM authentication systems, such as resilience to distraction and social compatibility.

Ubiquitous technologies offer new opportunities for digitally augmenting children's toys and play experiences. A key question is how augmented toy environments affect children's playful learning, and whether this differs from non-augmented play environments. This paper presents preliminary results of a user study we conducted to evaluate an augmented toy environment that we built --- the Augmented Knights Castle --- in terms of fun and storytelling, particularly when compared with an identical, non-augmented version. All sessions were observed, video-recorded and further feedback was elicited through small group interviews and questionnaires. Findings suggest ways in which digitally augmented play environments promote different kinds of activity from an equivalent non-augmented play environment.

In this paper, we present a study of responses to the idea of being recorded by a ubicomp recording technology called SenseCam. This study focused on real-life situations in two North American and two European locations. We present the findings of this study and their implications, specifically how those who might be recorded perceive and react to SenseCam. We describe what system parameters, social processes, and policies are required to meet the needs of both the primary users and these secondary stakeholders and how being situated within a particular locale can influence responses. Our results indicate that people would tolerate potential incursions from SenseCam for particular purposes. Furthermore, they would typically prefer to be informed about and to consent to recording as well as to grant permission before any data is shared. These preferences, however, are unlikely to instigate a request for deletion or other action on their part. These results inform future design of recording technologies like SenseCam and provide a broader understanding of how ubicomp technologies might be taken up across different cultural and political regions.

Augmented game environments use unobtrusively embedded technology to augment traditional games with virtual information and novel interaction capabilities. This article establishes and discusses a set of guidelines for designing and implementing such environments, based on our experiences in creating digital augmentations of existing play environments. We suggest a two-step process comprised of game flow virtualization and physical artifact augmentation to create augmented game environments based on existing table top games. We will then demonstrate how these guidelines can be put to practice by presenting the augmented version of a miniature war game.

Combining interactive technology with traditional toys promises to significantly enhance the educational value of children's play. Designing such augmented toy environments, however, requires designers to take both the traditional, technology-less nature of the toy, and the novel interactive aspects of the newly accessible virtual environment into account. This article attempts to present a unified set of guidelines for the design and implementation of augmented toy environments, drawing upon existing literature in traditional and educational toy and game design, as well as our own experiences in building mixed reality game environments. We also offer practical advice on the use of these guidelines by reporting on our own augmented toy environment for young children, called the Augmented Knight's Castle, which encourages learning about the Middle Ages in a playful way.

Smart environments and wearables will make the storage and subsequent sharing of digitized multimedia diaries and meeting protocols - whom we meet, or what we say or do - cheap and easy. However, controlling access to this data will become cumbersome if traditional forms of access control are to be used: overly restrictive rules might deny the potential of data sharing, while a lack of control could easily lead to Orwellian surveillance scenarios. This paper presents FragDB, a storage concept based on localized access control, where data storage and retrieval are bound to a specific place, rather than the knowledge of a particular password or certificate. FragDB uses tiny RFID tags embedded in the environment to compute a local key that is used to encrypt and decrypt data in a global storage system. We describe the design and implementation of an initial prototype.

The use of suitable examples is a key to teach abstract, theoretical concepts. Interactive computer software allows us to use such examples to create attractive learning environments that not only appeal to students, but also enhance knowledge transfer in class. However, developing such highly specialized systs is costly, resulting in only few of these tools being developed for higher education. This article introduces Info Traffic, a collection of new learning environments to support the introduction of fundamental concepts of computer science and mathatics in order to be of long-lived value. We describe the didactical concepts behind the interactive and concrete approach of Info Traffic, and illustrate th through two of its learning environments -- one targeted at propositional logic, the other at queueing theory.