ExtendAI makes plan for action extends user's plan by embedding feedback makes sense of plan containing AI's feedback makes final decision RecommendAI makes sense of AI's suggestions makes suggestion for action makes final decision Figure 1: Illustrative comparison of the thought process when interacting with two 'types' of AI -RecommendAI and ExtendAI.

The use of symbols and metaphors can be a fast and effective way of conveying abstract concepts. At the same time, misconceived symbols can lead to misunderstandings and errors. Therefore, when it comes to privacy and security, clear communication is essential to avoid putting users’ personal data at risk. In this paper, we elicit 32 symbols and metaphors associated with privacy and security through two brainstorming sessions (n = 8, each). Six experts further separated this collection into security and privacy-related symbols and generated clusters based on similarity. Using participants’ clusters, we derived underlying themes. As a result, we present a symbol and metaphor space for privacy and security and discuss their perceived meaning. Our findings can serve researchers, designers, and developers to find suitable symbols or metaphors for a given scenario (e.g., to decide on the interaction metaphor for a tangible security mechanism) and to understand if a symbol is ambiguous or how it may be understood (e.g., is an eye associated with privacy configurations?). Our work provides an initial knowledge base supporting effective communication in this field.

We propose a paradigm shift in human-centered security research in which users’ objective behavior and physiological states move into focus. This proposal is motivated by the fact that many personal and wearable devices today come with capabilities that allow researchers to assess users’ behavior and physiology in real-time. We expect substantial advances due to the ability to develop more targeted approaches to human-centered security in which solutions are targeted at individuals’ literacy, skills, and context. To this end, the main contribution of this work is a research space: we first provide an overview of common human-centered attacks that could be better understood and addressed through our approach. Based on this overview, we then showcase how specific security habits can benefit from the knowledge of users’ current state. Our work is complemented by a discussion of the implications and research directions enabled through this novel paradigm.

Exploiting emotions for user interface evaluation became an increasingly important research objective in Human-Computer Interaction. Emotions are usually assessed through surveys that do not allow information to be collected in real-time. In our work, we suggest the use of smartphones for mobile emotion assessment. We use the front-facing smartphone camera as a tool for emotion detection based on facial expressions. Such information can be used to reflect on emotional states or provide emotion-aware user interface adaptation. We collected facial expressions along with app usage data in a two-week field study consisting of a one-week training phase and a one-week testing phase. We built and evaluated a person-dependent classifier, yielding an average classification improvement of 33% compared to classifying facial expressions only. Furthermore, we correlate the estimated emotions with concurrent app usage to draw insights into changes in mood. Our work is complemented by a discussion of the feasibility of probing emotions on-the-go and potential use cases for future emotion-aware applications.

The pervasive availability of media in foreign languages is a rich resource for language learning. However, learners are forced to interrupt media consumption whenever comprehension problems occur. We present BrainCoDe, a method to implicitly detect vocabulary gaps through the evaluation of event-related potentials (ERPs). In a user study (N=16), we evaluate BrainCoDe by investigating differences in ERP amplitudes during listening and reading of known words compared to unknown words. We found significant deviations in N400 amplitudes during reading and in N100 amplitudes during listening when encountering unknown words. To evaluate the feasibility of ERPs for real-time applications, we trained a classifier that detects vocabulary gaps with an accuracy of 87.13% for reading and 82.64% for listening, identifying eight out of ten words correctly as known or unknown. We show the potential of BrainCoDe to support media learning through instant translations or by generating personalized learning content.

Mobile applications that are granted permission to access the device’s camera can access it at any time without necessarily showing the camera feed to the user or communicating that it is being used. This lack of transparency raises privacy concerns, which are exacerbated by the increased adoption of applications that leverage front-facing cameras. Through a focus group we identified three promising approaches for nudging the user that the camera is being accessed, namely: notification bar, frame, and camera preview. We experimented with accompanying each nudging method with vibrotactile and audio feedback. Results from a user study (N=15) show that while using frame nudges is the least annoying and interrupting, but was less understandable than the camera feed and notifications. On the other hand, participants found that indicating camera usage by showing its feed or by using notifications is easy to understand. We discuss how these nudges raise user awareness and the effects on app usage and perception.

Many user interfaces involve attention shifts between primary and secondary tasks, e.g., when changing a mode in a menu, which detracts the user from their main task. In this work, we investigate how eye gaze input affords exploiting the attention shifts to enhance the interaction with handheld menus. We assess three techniques for menu selection: dwell time, gaze button, and cursor. Each represents a different multimodal balance between gaze and manual input. We present a user study that compares the techniques against two manual baselines (dunk brush, pointer) in a compound colour selection and line drawing task. We show that user performance with the gaze techniques is comparable to pointer-based menu selection, with less physical effort. Furthermore, we provide an analysis of the trade-off as each technique strives for a unique balance between temporal, manual, and visual interaction properties. Our research points to new opportunities for integrating multimodal gaze in menus and bimanual interfaces in 3D environments.

Communicating uncertainty has been shown to provide positive effects on user understanding and decision-making. Surprisingly however, most personal health tracking applications fail to disclose the accuracy of their measurements and predictions. In the case of fertility tracking applications (FTAs), inaccurate predictions have already led to numerous unwanted pregnancies and law suits. However, integrating uncertainty into FTAs is challenging: Prediction accuracy is hard to understand and communicate, and its effect on users' trust and behavior is not well understood. We created a prototype for uncertainty visualizations for FTAs and evaluated it in a four-week field study with real users and their own data (N=9). Our results uncover far-reaching effects of communicating uncertainty: For example, users interpreted prediction accuracy as a proxy for their cycle health and as a security indicator for contraception. Displaying predicted and detected fertile phases next to each other helped users to understand uncertainty without negative emotional effects.

A common objective for context-aware computing systems is to predict how user interfaces impact user performance regarding their cognitive capabilities. Existing approaches such as questionnaires or pupil dilation measurements either only allow for subjective assessments or are susceptible to environmental influences and user physiology. We address these challenges by exploiting the fact that cognitive workload influences smooth pursuit eye movements. We compared three trajectories and two speeds under different levels of cognitive workload within a user study (N=20). We found higher deviations of gaze points during smooth pursuit eye movements for specific trajectory types at higher cognitive workload levels. Using an SVM classifier, we predict cognitive workload through smooth pursuit with an accuracy of 99.5% for distinguishing between low and high workload as well as an accuracy of 88.1% for estimating workload between three levels of difficulty. We discuss implications and present use cases of how cognition-aware systems benefit from inferring cognitive workload in real-time by smooth pursuit eye movements.

Mobile text messaging is one of the most important communication channels today, but it suffers from lack of expressiveness, context and emotional awareness, compared to face-to-face communication. We address this problem by augmenting text messaging with information about users and contexts. We present and reflect on lessons learned from three field studies, in which we deployed augmentation concepts as prototype chat apps in users’ daily lives. We studied (1) subtly conveying context via dynamic font personalisation ( TapScript ), (2) integrating and sharing physiological data – namely heart rate – implicitly or explicitly ( HeartChat ) and (3) automatic annotation of various context cues: music, distance, weather and activities ( ContextChat ). Based on our studies, we discuss chat augmentation with respect to privacy concerns, understandability, connectedness and inferring context in addition to methodological lessons learned. Finally, we propose a design space for chat augmentation to guide future research, and conclude with practical design implications.

Textual communication via mobile phones suffers from a lack of context and emotional awareness. We present a mobile chat application, HeartChat, which integrates heart rate as a cue to increase awareness and empathy. Through a literature review and a focus group, we identified design dimensions important for heart rate augmented chats. We created three concepts showing heart rate per message, in real-time, or sending it explicitly. We tested our system in a two week in-the-wild study with 14 participants (7 pairs). Interviews and questionnaires showed that HeartChat supports empathy between people, in particular close friends and partners. Sharing heart rate helped them to implicitly understand each other's context (e.g. location, physical activity) and emotional state, and sparked curiosity on special occasions. We discuss opportunities, challenges, and design implications for enriching mobile chats with physiological sensing.

The human body reveals emotional and bodily states through measurable signals, such as body language and electroencephalography. However, such manifestations are difficult to communicate to others remotely. We propose EmotionActuator, a proof-of-concept system to investigate the transmission of emotional states in which the recipient performs emotional gestures to understand and interpret the state of the sender.We call this kind of communication embodied emotional feedback, and present a prototype implementation. To realize our concept we chose four emotional states: amused, sad, angry, and neutral. We designed EmotionActuator through a series of studies to assess emotional classification via EEG, and create an EMS gesture set by comparing composed gestures from the literature to sign-language gestures. Through a final study with the end-to-end prototype interviews revealed that participants like implicit sharing of emotions and find the embodied output to be immersive, but want to have control over shared emotions and with whom. This work contributes a proof of concept system and set of design recommendations for designing embodied emotional feedback systems.

Obtaining information about audience engagement in presentations is a valuable asset for presenters in many domains. Prior literature mostly utilized explicit methods of collecting feedback which induce distractions, add workload on audience and do not provide objective information to presenters. We present EngageMeter - a system that allows fine-grained information on audience engagement to be obtained implicitly from multiple brain-computer interfaces (BCI) and to be fed back to presenters for real time and post-hoc access. Through evaluation during an HCI conference (Naudience=11, Npresenters=3) we found that EngageMeter provides value to presenters (a) in real-time, since it allows reacting to current engagement scores by changing tone or adding pauses, and (b) in post-hoc, since presenters can adjust their slides and embed extra elements. We discuss how EngageMeter can be used in collocated and distributed audience sensing as well as how it can aid presenters in long term use.

Although mobile devices provide access to a plethora of sensitive data, most users still only protect them with PINs or patterns, which are vulnerable to side-channel attacks (e.g., shoulder surfing). How-ever, prior research has shown that privacy-aware users are willing to take further steps to protect their private data. We propose GazeTouchPIN, a novel secure authentication scheme for mobile devices that combines gaze and touch input. Our multimodal approach complicates shoulder-surfing attacks by requiring attackers to ob-serve the screen as well as the user’s eyes to and the password. We evaluate the security and usability of GazeTouchPIN in two user studies (N=30). We found that while GazeTouchPIN requires longer entry times, privacy aware users would use it on-demand when feeling observed or when accessing sensitive data. The results show that successful shoulder surfing attack rate drops from 68% to 10.4%when using GazeTouchPIN.

Today's workplaces are dynamic and complex. Digital data sources such as email and video conferencing aim to support workers but also add to their burden of multitasking. Psychophysiological sensors such as Electroencephalography (EEG) can provide users with cues about their cognitive state. We introduce BrainAtWork, a workplace engagement and task logger which shows users their cognitive state while working on different tasks. In a lab study with eleven participants working on their own real-world tasks, we gathered 16 hours of EEG and PC logs which were labeled into three classes: central, peripheral and meta work. We evaluated the usability of BrainAtWork via questionnaires and interviews. We investigated the correlations between measured cognitive engagement from EEG and subjective responses from experience sampling probes. Using random forests classification, we show the feasibility of automatically labeling work tasks into work classes. We discuss how BrainAtWork can support workers on the long term through encouraging reflection and helping in task scheduling.

Common user authentication methods on smartphones, such as lock patterns, PINs, or passwords, impose a trade-off between security and password memorability. Image-based passwords were proposed as a secure and usable alternative. As of today, however, it remains unclear how such schemes are used in the wild. We present the first study to investigate how image-based passwords are used over long periods of time in the real world. Our analyses are based on data from 2318 unique devices collected over more than one year using a custom application released in the Android Play store. We present an in-depth analysis of what kind of images users select, how they define their passwords, and how secure these passwords are. Our findings provide valuable insights into real-world use of image-based passwords and inform the design of future graphical authentication schemes.