Human-centered security research traditionally leverages self-reports and high-level behavioral data. However, the increasing ubiquity of sensors integrated into personal, wearable devices (e.g., smartphones, smartwatches) and in users’ environments (e.g., cameras) enables researchers to unobtrusively collect rich physiological and behavioral signals. These real-time data streams can reveal user states—such as attention or workload—that can be employed to design adaptive security mechanisms. In this paper, we present a platform that supports designing, building, and evaluating next-generation user interfaces that leverage physiological and behavioral data for enhanced security. First, we introduce the physio-behavioral security paradigm, highlighting how sensor-based insights into user states can inform individualized security interventions and accurately identify moments of vulnerability. We then outline the requirements, system architecture, and implementation details of the platform, illustrating how multiple data streams (e.g., gaze, heart rate, keystrokes, mouse movements) are integrated and securely processed. Finally, we report on an exploratory deployment in a mid-sized organization, showcasing how the tool captures real-time security behaviors and enables context-aware interventions. The deployment yields insights into factors influencing acceptance across different stakeholders (management, IT department, employees). Our results suggest that adaptive approaches, informed by physiological and behavioral signals, can improve security outcomes and user acceptance.

Vishing – the use of voice calls for phishing – is a form of Social Engineering (SE) attacks. The latter have become a pervasive challenge in modern societies, with over 300,000 yearly victims in the US alone. An increasing number of those attacks is conducted via voice communication, be it through machine-generated ‘robocalls’ or human actors. The goals of ‘social engineers’ can be manifold, from outright fraud to more subtle forms of persuasion. Accordingly, social engineers adopt multi-faceted strategies for voice-based attacks, utilising a variety of ‘tricks’ to exert influence and achieve their goals. Importantly, while organisations have set in place a series of guardrails against other types of SE attacks, voice calls still remain ‘open ground’ for potential bad actors. In the present contribution, we provide an overview of the existing speech technology subfields that need to coalesce into a protective net against one of the major challenges to societies worldwide. Given the dearth of speech science and technology works targeting this issue, we have opted for a narrative review that bridges the gap between the existing psychological literature on the topic and research that has been pursued in parallel by the speech community on some of the constituent constructs. Our review reveals that very little literature exists on addressing this very important topic from a speech technology perspective, an omission further exacerbated by the lack of available data. Thus, our main goal is to highlight this gap and sketch out a roadmap to mitigate it, beginning with the psychological underpinnings of vishing, which primarily include deception and persuasion strategies, continuing with the speech-based approaches that can be used to detect those, as well as the generation and detection of AI-based vishing attempts, and close with a discussion of ethical and legal considerations. • Vishing is an emerging security problem. • Generative artificial intelligence will exacerbate the issue. • Speech-based detection is urgently needed. • Beyond detection performance, effective interventions are warranted.

We developed an Android phone unlock mechanism utilizing facial recognition and specific mimics to access a specially secured portion of the device, designed for plausible deniability. The widespread adoption of biometric authentication methods, such as fingerprint and facial recognition, has revolutionized mobile device security, offering enhanced protection against shoulder-surfing attacks and improving user convenience compared to traditional passwords. However, a downside is the potential for third-party coercion to unlock the device. While text-based authentication allows users to reveal a hidden system by entering a special password, this is challenging with face authentication. We evaluated our approach in a role-playing user study involving 50 participants, with one participant acting as the attacker and the other as the suspect. Suspects successfully accessed the secured area, mostly without detection. They further expressed interest in this feature on their personal phones. We also discuss open challenges and opportunities in implementing such authentication mechanisms.

The use of symbols and metaphors can be a fast and effective way of conveying abstract concepts. At the same time, misconceived symbols can lead to misunderstandings and errors. Therefore, when it comes to privacy and security, clear communication is essential to avoid putting users’ personal data at risk. In this paper, we elicit 32 symbols and metaphors associated with privacy and security through two brainstorming sessions (n = 8, each). Six experts further separated this collection into security and privacy-related symbols and generated clusters based on similarity. Using participants’ clusters, we derived underlying themes. As a result, we present a symbol and metaphor space for privacy and security and discuss their perceived meaning. Our findings can serve researchers, designers, and developers to find suitable symbols or metaphors for a given scenario (e.g., to decide on the interaction metaphor for a tangible security mechanism) and to understand if a symbol is ambiguous or how it may be understood (e.g., is an eye associated with privacy configurations?). Our work provides an initial knowledge base supporting effective communication in this field.

While eye-tracking has become a core asset for many computing environments, mobile phones, as a prime computing device, are lacking a practical platform to conduct eye-tracking studies and develop gaze applications easily. In this work, we aim to tackle this issue by investigating a system concept that allows for the deployment of remote eye-trackers for mobile devices. We describe a toolkit that supports eye-tracking in mobile apps based on a simple phone, PC, and remote eye tracker setup. We evaluate our approach through a technical evaluation of accuracy and precision in various user contexts important for mobility (sitting, standing, walking, lying). Our results show that eye-trackers can be easily used with high accuracy, and how it is impacted through body posture and motions of the user. Our work paves the way for enabling easy-to-use eye-tracking studies on mobile devices.

While Virtual Reality continues to increase in fidelity, it remains an open question how to effectively reflect the user's movements and provide congruent feedback in virtual environments. We present VRsneaky, a system for producing auditory movement feedback, which helps participants orient themselves in a virtual environment by providing footstep sounds. The system reacts to the user's specific gait features and adjusts the audio accordingly. In a user study with 28 participants, we found that VRsneaky increases users' sense of presence as well as awareness of their own posture and gait. Additionally, we find that increasing auditory realism significantly influences certain characteristics of participants' gait. Our work shows that gait-aware audio feedback is a means to increase presence in virtual environments. We discuss opportunities and design requirements for future scenarios where users walk through immersive virtual worlds.