Art has long been a powerful means by which disabled people and other minority group have established counter-narratives, pushing back against damaging stereotypes. These efforts become particularly meaningful when bringing together the work and stories of multiple disabled people, helping to move away from individual, while striving to preserve the uniqueness of everyone’s contribution. In this paper, we document a collective digital art project led by an internationally acclaimed disabled artist involving 16 local disabled people in Japan, featuring a series of workshops where participants created self-portraits using iPads subsequently included in a larger inflatable sculpture. The generated art piece was exhibited to the local community during a series of events organised for the International Day of People with Disability. We reflect on our experiences to suggest key methodological learnings around peer-led collaborative art for counter-narrative creation and examine the role of technology in the context of curation.

Soft floating robots (SFRs) represent a shift from rigid machines, offering gravity-defying, compliant, and tactile embodiments for indoor cohabitation. However, their development remains fragmented across isolated prototypes, lacking a coherent design vocabulary. Without a systematic understanding of their interactional capabilities, designers struggle to leverage SFRs’ unique affordances, and these systems often remain limited to novelty applications that are difficult to integrate into everyday life. To address this, we propose a design space for interaction with SFRs. Informed by an exploratory study with 12 experts from HCI, Design, and Robotics, we identify ten design dimensions spanning physical, interactive, and behavioral properties, along with a range of application scenarios. We further present proof-of-concept design examples to demonstrate how this design space can support diverse interaction possibilities. This work contributes a structured framework for understanding and designing interactions with SFRs, supporting their integration into everyday indoor environments.

Vocal training is difficult because the muscles that control pitch, resonance, and phonation are internal and invisible to learners. This paper investigates how Electromyography (EMG) and ultrasonic imaging (UI) can make these muscles observable for training purposes. We report three studies. First, we analyze the EMG and UI data from 16 singers (beginners, experienced & professionals), revealing differences among three vocal groups of the muscle control proficiency. Second, we use the collected data to create a system that visualizes an expert’s muscle activity as reference. This system is tested in a user study with 12 novices, showing that EMG highlighted muscle activation nuances, while UI provided insights into vocal cord length and dynamics. Third, to compare our approach to traditional methods (audio analysis and coach instructions), we conducted a focus group study with 15 experienced singers. Our results suggest that EMG is promising for improving vocal skill development and enhancing feedback systems. We conclude the paper with a detailed comparison of the analyzed modalities (EMG, UI and traditional methods), resulting in recommendations to improve vocal muscle training systems.

Smooth pursuits are natural eye movements that occur when we track a moving target with our gaze. While they were explored as a gaze-based input method using external screens to display moving stimuli, we propose BodyPursuits, a novel HCI method that eliminates additional screens. Stimuli are generated by users tracing smooth trajectories with their hand in mid-air while fixating their gaze on their thumb. We conducted a user study to collect eye-tracking and baseline IMU data from 20 participants performing 10 BodyPursuits gestures. Based on 1800 samples and noise data, we train a TinyHAR classifier. It achieves a macro-average F1 score of 0.772. With UEQ results indicating a positive user experience and RTLX scores showcasing low subjective workload for all 10 gestures, we successfully demonstrated BodyPursuits’ potential as viable interaction method. We envision BodyPursuits could be integrated into EOG earphones to detect mid-air hand gestures without external screens or cameras.

Running and jogging are popular activities for many visually impaired individuals thanks to the relatively low entry barriers. Research in HCI and beyond has focused primarily on leveraging technology to enable visually impaired people to run independently. However, depending on their residual vision and personal preferences, many chose to run with a sighted guide. This study presents a comprehensive analysis of the partnership between visually impaired runners and sighted guides. Using a combination of interaction and thematic analysis on video and interview data from 6 pairs of runners and guides, we unpack the complexity and directionality of three layers of vocal communication (directive, contextual, and recreational) and distinguish between intentional and unintentional corporeal communication. Building on the understanding of the importance of synchrony we also present some exploratory data looking at physiological synchrony between 2 pairs of runners with different level of experience and articulate recommendations for the HCI community.

In this paper, we propose a method for utilizing musical artifacts and physiological data as a means for creating a new form of live music experience that is rooted in the physiology of the performers and audience members. By utilizing physiological data (namely Electrodermal Activity (EDA) and Heart Rate Variability (HRV)) and applying this data to musical artifacts including a robotic koto (a traditional 13-string Japanese instrument fitted with solenoids and linear actuators), a Eurorack synthesizer, and Max/MSP software, we aim to develop a new form of semi-improvisational and significantly indeterminate performance practice. It has since evolved into a multi-modal methodology which honors improvisational performance practices and utilizes physiological data which offers both performers and audiences an ever-changing and intimate experience.

Robotic avatars can help disabled people extend their reach in interacting with the world. Technological advances make it possible for individuals to embody multiple avatars simultaneously. However, existing studies have been limited to laboratory conditions and did not involve disabled participants. In this paper, we present a real-world implementation of a parallel control system allowing disabled workers in a café to embody multiple robotic avatars at the same time to carry out different tasks. Our data corpus comprises semi-structured interviews with workers, customer surveys, and videos of café operations. Results indicate that the system increases workers’ agency, enabling them to better manage customer journeys. Parallel embodiment and transitions between avatars create multiple interaction loops where the links between disabled workers and customers remain consistent, but the intermediary avatar changes. Based on our observations, we theorize that disabled individuals possess specific competencies that increase their ability to manage multiple avatar bodies.

The use of wearable sensor technology opens up exciting avenues for both art and HCI research, providing new ways to explore the invisible link between audience and performer. To be effective, such work requires close collaboration between performers and researchers. In this article, we report on the co-design process and research insights from our work integrating physiological sensing and live performance. We explore the connection between the audience’s physiological data and their experience during the performance, analyzing a multi-modal dataset collected from 98 audience members. We identify notable moments based on HRV and EDA, and show how the audience’s physiological responses can be linked to the choreography. The longitudinal changes in HRV features suggest a strong connection to the choreographer’s intended narrative arc, while EDA features appear to correspond with short-term audience responses to dramatic moments. We discuss the physiological phenomena and implications for designing feedback systems and interdisciplinary collaborations.

Ethereal Phenomena is an interactive installation in which a user meditates in front of a digital illustration based on Tibetan thangka art that reacts to the way they breathe. The biofeedback of the breath makes the artwork an extension of the user’s body and a reflection of their mind. By combining art, sound, and biofeedback, Ethereal Phenomena becomes a unique meditative experience that aims at producing physical and mental wellness and that has the potential to induce a state of self-transcendence or pure consciousness.

What we wear (our clothes and wearable accessories) can represent our mood at the moment. We developed Emolleia to explore how to make aesthetic wears more expressive to become a novel form of non-verbal communication to express our emotional feelings. Emolleia is an open wearable kinetic display in form of three 3D printed flowers that can dynamically open and close at different speeds. With our open-source platform, users can define their own animated motions. In this paper, we described the prototype design, hardware considerations, and user surveys (n=50) to evaluate the expressiveness of 8 pre-defined animated motions of Emolleia. Our initial results showed animated motions are feasible to communicate different emotional feelings especially at the valence and arousal dimensions. Based on the findings, we mapped eight pre-defined animated motions to the reported, user-perceived valence, arousal and dominance and discussed possible directions for future work.

As the population ages, many will acquire visual impairments. To improve design for these users, it is essential to build awareness of their perspective during everyday routines, especially for design students.

Two-factor authentication (2FA) is a recommended or imposed authentication mechanism for valuable online assets. However, 2FA mechanisms usually exhibit user experience issues that create user friction and even lead to poor acceptance, hampering the wider spread of 2FA. In this article, we investigate user perceptions of 2FA through in-depth interviews with 42 participants, revealing key requirements that are not well met today despite recently emerged 2FA solutions. First, we investigate past experiences with authentication mechanisms emphasizing problems and aspects that hamper good user experience. Second, we investigate the different authentication factors more closely. Our results reveal particularly interesting preferences regarding the authentication factor “ownership” in terms of properties, physical realizations, and interaction. These findings suggest a path toward 2FA mechanisms with considerably better user experience, promising to improve the acceptance and hence, the proliferation of 2FA for the benefit of security in the digital world.

We present our first steps towards an umbrella-based novel multimedia platform using physiological data as an integrated feedback loop. In this paper, we demonstrate the viability of using an umbrella handle as a form factor to measure electrodermal activity(EDA) and heart rate(HR) in real-time. We compared the performance of the device with that of a more conventional finger sensor placement. Although the finger sensor placement is more widespread and considered to be more reliable, yet we are able to derive meaningful data from the umbrella handle in both stationary and dynamic contexts in the presented feasibility study.

Insufficient social cues between distributed learners in online learning could result in lack of engagement and social bonds. With the development of wearable sensing, sharing physiological data can be used to enhance mutual understanding and connectedness among sharers. Our work aims to explore the potential of sharing heart rate (HR) and heart rate variability (HRV) collected from distributed learners to enhance their online learning experiences. We implemented a physiological streaming system and conducted a field study with 11 learners in online classes. This paper describes the study and discusses our interview findings by contrasting the influence of visualized collective physiological data from viewpoints of data contributors and viewers. Our exploratory results suggest streaming collective HR and HRV from multiple distributed learners could be used in online classes to improve engagement and sense of community.

We present ThermoQuest, a self-contained wearable head-mounted display system for enhancing Virtual Reality experiences with temperature feedback. It’s constructed with commodity hardware elements, featuring 6 Peltier elements on the rim of the headset touching the users face. We explain the design and implementation of an affordable wearable thermal VR prototype build with commodity hardware. In a user study with 15 participants, we show evidence for a significant difference in reported presence in VR between thermal VR and control conditions (p > 0.017) in a counterbalanced experimental setup. We end with a discussion and use cases of the presented VR prototype and similar systems.

In stage performances, an invisible wall in front of the stage often weakens the connections between the audience and performers. To amplify this performative connection, we present the concept ”Boiling Mind”. Our design concept is based on streaming sensor data related to heart and electrodermal activities from audience members and integrating this data into staging elements, such as visual projections, music, and lighting. Thus, the internal states of the audience directly influence the staging. Artists can have a more direct perception of the inner reactions of audience members and can create physical expressions in response to them. In this paper, we present the wearable sensing system as well as design considerations of mapping heart and electrodermal activity to changes in the staging elements. We evaluated our design and setup over three live performances.

Internet voting can afford more inclusive and inexpensive elections. The flip side is that the integrity of the election can be compromised by adversarial attacks and malfunctioning voting infrastructure. Individual verifiability aims to protect against such risks by letting voters verify that their votes are correctly registered in the electronic ballot box. Therefore, voters need to carry out additional tasks making human factors crucial for security. In this article, we establish a categorization of individually verifiable Internet voting schemes based on voter interactions. For each category in our proposed categorization, we evaluate a voting scheme in a user study with a total of 100 participants. In our study, we assessed usability, user experience, trust, and further qualitative data to gain deeper insights into voting schemes. Based on our results, we conclude with recommendations for developers and policymakers to inform the choices and design of individually verifiable Internet voting schemes.

Two-factor authentication is a widely recommended security mechanism and already offered for different services. However, known methods and physical realizations exhibit considerable usability and customization issues. In this paper, we propose 3D-Auth, a new concept of two-factor authentication. 3D-Auth is based on customizable 3D-printed items that combine two authentication factors in one object. The object bottom contains a uniform grid of conductive dots that are connected to a unique embedded structure inside the item. Based on the interaction with the item, different dots turn into touch-points and form an authentication pattern. This pattern can be recognized by a capacitive touchscreen. Based on an expert design study, we present an interaction space with six categories of possible authentication interactions. In a user study, we demonstrate the feasibility of 3D-Auth items and show that the items are easy to use and the interactions are easy to remember.

Human-Computer Integration (HInt) is an emerging paradigm in which computational and human systems are closely interwoven. Integrating computers with the human body is not new. however, we believe that with rapid technological advancements, increasing real-world deployments, and growing ethical and societal implications, it is critical to identify an agenda for future research. We present a set of challenges for HInt research, formulated over the course of a five-day workshop consisting of 29 experts who have designed, deployed and studied HInt systems. This agenda aims to guide researchers in a structured way towards a more coordinated and conscientious future of human-computer integration.

We propose the use of face-related gestures involving the movement of the face, eyes, and head for augmented reality (AR). This technique allows us to use computer systems via hands-free, discreet interactions. In this paper, we present an elicitation study to explore the proper use of facial gestures for daily tasks in the context of a smart home. We used Amazon Mechanical Turk to conduct this study (N=37). Based on the proposed gestures, we report usage scenarios and complexity, proposed associations between gestures/tasks, a user-defined gesture set, and insights from the participants. We also conducted a technical feasibility study (N=13) with participants using smart eyewear to consider their uses in daily life. The device has 16 optical sensors and an inertial measurement unit (IMU). We can potentially integrate the system into optical see-through displays or other smart glasses. The results demonstrate that the device can detect eight temporal face-related gestures with a mean F1 score of 0.911 using a convolutional neural network (CNN). We also report the results of user-independent training and a one-hour recording of the experimenter testing two of the gestures.

The tactile and kinesthetic sensation of pushing a button is usually lost when interacting with modern devices like touchscreens and/or virtual reality platforms. We present FingerFlex, a standalone glove wearable actuating the metacarpophalangeal joint (MCP) of each finger via shape memory alloy (SMA). SMA actuation is subtle, silent, and light, making it ideal for actuation of the fingers which we use to simulate the sensation of pressing a button.

In this paper, we present AromaCue, an initial design for a scent-based toolkit to cope with stressful situations using scent conditioning. The AromaCue toolkit consists of two parts: a breath training device using multiple stimuli and a wearable scent-emitting device (with a stress ball as an activator). Scent can trigger emotional memories. In the initial design, we utilized the properties of various scents as retrieval cues for consciousness breathing. In a comparative experiment, eight participants showed a significant heart rate decrease several minutes after a stressor (the Stroop test) when a scent cue was present, but not without it. One week of user study revealed significant improvement on Depression Anxiety and Stress Scale (DASS-21) scores. Therefore, AromaCue can help users to cope with stress.

Research in sociology shows that effective conversation relates to people’s spatial and orientational relationship, namely the proxemics (distance, eye contact, synchrony) and the F-formation (orientation and arrangement). In this work, we introduce novel conversational paradigms that effects conventional F-formation by introducing the concept of multi-directional conversation. Multiplex Vision is a head-mounted device capable of providing a 360° field-of-view (FOV) and facilitating multi-user interaction multi-directionally, thereby providing novel methods on how people can interact with each other. We propose 3 possible new forms of interactions from our prototype: one-to-one, one-to-many, and many-to-many. To facilitate them, we manipulate 2 key variables, which are the viewing parameter and the display parameter. To gather feedback for our system, we conducted a study to understand information transfer between various modes, as well as a user study on how different proposed paradigms effect conversation. Finally, we discuss present and future use cases that can benefit from our system.

As the day progresses, cognitive functions are subject to fluctuations. While the circadian process results in diurnal peaks and drops, the homeostatic process manifests itself in a steady decline of alertness across the day. Awareness of these changes allows the design of proactive recommender and warning systems, which encourage demanding tasks during periods of high alertness and flag accident-prone activities in low alertness states. In contrast to conventional alertness assessments, which are often limited to lab conditions, bulky hardware, or interruptive self-assessments, we base our approach on eye blink frequency data known to directly relate to fatigue levels. Using electrooculography sensors integrated into regular glasses' frames, we recorded the eye movements of 16 participants over the course of two weeks in-the-wild and built a robust model of diurnal alertness changes. Our proposed method allows for unobtrusive and continuous monitoring of alertness levels throughout the day.

We tend to synchronize our movements to the person we are talking to during face-to-face conversation. Higher interpersonal synchrony is linked to greater empathy and more effortless interactions. This paper presents a first method and a corresponding dataset to explore synchrony in natural conversation by capturing eye and head movement using commodity smart eyewear. We present a 17 hour dataset, using Electrooculography and inertial sensing, of 42 people in conversation (21 dyads: 10 in Japanese, 10 in English, 1 in Chinese). Initial results on 18 dyads show significant interpersonal synchrony of blink and head nod behaviour during conversation (at frequencies of 0.2 to 0.5 Hz). We also find that people are more likely to synchronise blinks at around 1 Hz when conversing back-to-back than when face-to-face.

In this work we present the smart eyewear demo setup consisting of the software platform for cognitive and social interaction assessments in the wild, with several application cases and a demonstration of activity recognition in real-time. The platform is designed to work with Jins MEME, smart EOG enabled glasses, The user software is capable data logging, posture tracking and recognition of several activities, such as talking, reading and blinking. In this work we present several applications and studies that the platform has been used for.

Reading in public spaces can often be tricky if we wish to keep the contents away from the prying eye. We propose Private Reader, an eye-tracking approach towards maintaining privacy while reading by rendering only the portion of text that is gazed by the reader. We conducted a user study by evaluating for both the reader and observer in terms of privacy, reading comfort, and reading speed for three reading modes; normal, underscored, and scrambled text. "Scrambled" performs best in terms of perceived effort and frustration for the shoulder surfer. Our contribution is threefold; we developed a system to preserve privacy by rendering only the text at gaze-point of the reader, we conducted a user study to evaluate user preferences and subjective task load, and we suggested several scenarios where Private Reader is useful in public spaces.

We present a platform to prototype spatial user interfaces for smart watches relative to the user's body. We show the general feasibility and present 2 applications. The first application scenario enable users "pin" applications in the air around them and get back to them by moving the smart watch display at the same position again. The second application shows a zoom use case for maps or other larger displays/visualizations. The smart watch acts like a digital "magnifying glass" enabling the user to see details he cares about.

This paper presents a sensory supplementation experience in virtual reality based on the thermal and vibrotactile feedback. We have implemented a physical system that provides hot and cold sensations around the hand and forearm of the person wearing it. After being equipped with the system the user is exposed to the virtual reality environment with the goal to deflect projectiles coming their way while relying on the thermal sensations to detect them. Presented experience makes a case for using thermal sensory supplementation as an interaction modality in virtual reality environments.

We propose PanoFlex, an adaptive method for projecting panoramic vision using a dynamic distortion method based on eye gaze. We stream real-time video from a 360° camera and project the view on a plane to the user. The user controls the distortion of this equirectangular projection using eye gaze. For our first user study, we compare our method with conventional equirectangular projection considering the impact on spatial perception. For our second study, we perform a simulator sickness evaluation when the user performs regular daily activities. We found that PanoFlex did not carry any significant negative impact towards the user’s spatial perception, perceived task load, and simulator sickness compared to the more conventional equirectangular view.

Gaze-based interactions promise to be fast, intuitive and effective in controlling virtual and augmented environments. Yet, there is still a lack of usable 3D navigation and observation techniques. In this work: 1) We introduce a highly advantageous orbital navigation technique, AnyOrbit, providing an intuitive and hands-free method of observation in virtual environments that uses eye-tracking to control the orbital center of movement; 2) The versatility of the technique is demonstrated with several control schemes and use-cases in virtual/augmented reality head-mounted-display and desktop setups, including observation of 3D astronomical data and spectator sports.

Gaze-based interactions promise to be fast, intuitive and effective in controlling virtual and augmented environments. Yet, there is still a lack of usable 3D navigation and observation techniques. In this work: 1) We introduce a highly advantageous orbital navigation technique, AnyOrbit, providing an intuitive and hands-free method of observation in virtual environments that uses eye-tracking to control the orbital center of movement; 2) The versatility of the technique is demonstrated with several control schemes and use-cases in virtual/augmented reality head-mounted-display and desktop setups, including observation of 3D astronomical data and spectator sports.

This paper presents a new approach to implement wearable haptic devices using Shape Memory Alloy (SMA) wires. The proposed concept allows building silent, soft, flexible and lightweight wearable devices, capable of producing the sense of pressure on the skin without any bulky mechanical actuators. We explore possible design considerations and applications for such devices, present user studies proving the feasibility of delivering meaningful information and use nonlinear autoregressive neural networks to compensate for SMA inherent drawbacks, such as delayed onset, enabling us to characterize and predict the physical behavior of the device.

Navigation and mobility mechanics for virtual environments aim to be realistic or fun, but rarely prioritize the accuracy of movement. We propose PinchMove, a highly accurate navigation mechanic utilizing pinch gestures and manipulation of the viewport for confined environments that prefers accurate movement. We ran a pilot study to first determine the degree of simulator sickness caused by this mechanic, and a comprehensive user study to evaluate its accuracy in a virtual environment. We found that utilizing an 80° tunneling effect at a maximum speed of 15.18° per second was deemed suitable for PinchMove in reducing motion sickness. We also found our system to be at average, more accurate in enclosed virtual environments when compared to conventional methods. This paper makes the following three contributions: 1) We propose a navigation solution in near-field virtual environments for accurate movement, 2) we determined the appropriate tunneling effect for our method to minimize motion sickness, and 3) We validated our proposed solution by comparing it with conventional navigation solutions in terms of accuracy of movement. We also propose several use- case scenarios where accuracy in movement is desirable and further discuss the effectiveness of PinchMove.

We present a concept of AURA, an urban personal projection to initiate the communication. In this work, we focus on how to break the ice with strangers through technology in urban public space. The Aura, as an enliven spiritual pet, floats around user's feet. We introduce a simple interaction scenario, attracting a person who comes within personal distance of the user who carries the AURA. To break the ice between strangers, a projected butterfly as the Aura is moved toward a person who comes within 2m of the user, and then back and forth to attract that person. We believe that externalized interactive representation of the user in the form of a spiritual pet can ease and facilitate the communication, serve as a conversation starter, and make the interactions between people more fun.

We introduce MetaArms, wearable anthropomorphic robotic arms and hands with six degrees of freedom operated by the user's legs and feet. Our overall research goal is to re-imagine what our bodies can do with the aid of wearable robotics using a body-remapping approach. To this end, we present an initial exploratory case study. MetaArms' two robotic arms are controlled by the user's feet motion, and the robotic hands can grip objects according to the user's toes bending. Haptic feedback is also presented on the user's feet that correlate with the touched objects on the robotic hands, creating a closed-loop system. We present formal and informal evaluations of the system, the former using a 2D pointing task according to Fitts' Law. The overall throughput for 12 users of the system is reported as 1.01 bits/s (std 0.39). We also present informal feedback from over 230 users. We find that MetaArms demonstrate the feasibility of body-remapping approach in designing robotic limbs that may help us re-imagine what the human body could do.

This paper presents two innovative design solutions for thermal feedback displays in virtual environments. First solution is aiming to eliminate or decrease the time delay between the user action and onset of the thermal feedback using Machine Learning for user motion prediction. Second is the design of compact but efficient water cooling system necessary to provide cold sensations using peltier elements. Presented thermal display is wearable and battery powered.

Nonverbal cues, especially eye gaze, plays an important role in our daily communication, not just as an indicator of interest, but also as a method to convey information to another party. In this work, we propose a simulation of human eye gaze in Virtual Reality content to improve immersion of interaction between user and virtual agent. We developed an eye-tracking integrated interactive narrative content with a focus on player's interaction with gaze aware virtual agent, which is capable of reacting towards the player's gaze to simulate real human-to-human communication in VR environment and conducted an initial study to measure user's reaction.

We are presenting a Virtual Reality training system for Karate kata based on motion capture and Virtual Reality technologies. The system is built as a game, in which the player needs to learn and repeat different kata to progress and reach the next level. Different levels are represented by obi (belts) of different color, corresponding real Karate obi. We capture players' motion with a Kinect camera and enable interaction with game objects. A database is integrated in the game so that different players can use, save and track their training progress.

Navigating in a natural way in augmented reality (AR) and virtual reality (VR) spaces is a large challenge. To this end, we present ArmSwingVR, a locomotion solution for AR/VR spaces that preserves immersion, while being low profile compared to current solutions, particularly walking-in-place (WIP) methods. The user simply needs to swing their arms naturally to navigate in the direction where the arms are swung, without any feet or head movement. The benefits of ArmSwingVR are that arm swinging feels natural for bipedal organisms second only to leg movement, no additional peripherals or sensors are required, it is less obtrusive to swing our arms as opposed to WIP methods, and requires less energy allowing prolong uses for AR/VR. A conducted user study found that our method does not sacrifice immersion while also being more low profile and less energy consumption compared to WIP.

This paper suggests a demonstration to make users smile using electr ical muscle stimulation (EMS). We present our first insights in how to use an unobtr usive EMS setup to stimulate smiles and showed first findings from our entr y in this year's student innovation competition at an inter national HCI conference. We also outline our approach and follow up exper iments to potentially use this technology to improve user's mood.

This article presents a novel smart eyewear that recognizes the wearer’s facial expressions in daily scenarios. Our device uses embedded photo-reflective sensors and machine learning to recognize the wearer’s facial expressions. Our approach focuses on skin deformations around the eyes that occur when the wearer changes his or her facial expressions. With small photo-reflective sensors, we measure the distances between the skin surface on the face and the 17 sensors embedded in the eyewear frame. A Support Vector Machine (SVM) algorithm is then applied to the information collected by the sensors. The sensors can cover various facial muscle movements. In addition, they are small and light enough to be integrated into daily-use glasses. Our evaluation of the device shows the robustness to the noises from the wearer’s facial direction changes and the slight changes in the glasses’ position, as well as the reliability of the device’s recognition capacity. The main contributions of our work are as follows: (1) We evaluated the recognition accuracy in daily scenes, showing 92.8% accuracy regardless of facial direction and removal/remount. Our device can recognize facial expressions with 78.1% accuracy for repeatability and 87.7% accuracy in case of its positional drift. (2) We designed and implemented the device by taking usability and social acceptability into account. The device looks like a conventional eyewear so that users can wear it anytime, anywhere. (3) Initial field trials in a daily life setting were undertaken to test the usability of the device. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions with an unobtrusive wearable device.

Orbits combines a visual display and an eye motion sensor to allow a user to select between options by tracking a cursor with the eyes as the cursor travels in a circular path around each option. Using an off-the-shelf Jins MEME pair of eyeglasses, we present a pilot study that suggests that the eye movement required for Orbits can be sensed using three electrodes: one in the nose bridge and one in each nose pad. For forced choice binary selection, we achieve a 2.6 bits per second (bps) input rate at 250ms per input. We also inntroduce Head Orbits, where the user fixates the eyes on a target and moves the head in synchrony with the orbiting target. Measuring only the relative movement of the eyes in relation to the head, this method achieves a maximum rate of 2.0 bps at 500ms per input. Finally, we combine the two techniques together with a gyro to create an interface with a maximum input rate of 5.0 bps.

This paper presents a novel smart eyewear that uses embedded photo reflective sensors and machine learning to recognize a wearer's facial expressions in daily life. We leverage the skin deformation when wearers change their facial expressions. With small photo reflective sensors, we measure the proximity between the skin surface on a face and the eyewear frame where 17 sensors are integrated. A Support Vector Machine (SVM) algorithm was applied for the sensor information. The sensors can cover various facial muscle movements and can be integrated into everyday glasses. The main contributions of our work are as follows. (1) The eyewear recognizes eight facial expressions (92.8% accuracy for one time use and 78.1% for use on 3 different days). (2) It is designed and implemented considering social acceptability. The device looks like normal eyewear, so users can wear it anytime, anywhere. (3) Initial field trials in daily life were undertaken. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions in the form of an unobtrusive wearable device.

Reading is a very common learning activity, a lot of people perform it everyday even while standing in the subway or waiting in the doctors office. However, we know little about our everyday reading habits, quantifying them enables us to get more insights about better language skills, more effective learning and ultimately critical thinking. This paper presents a first contribution towards establishing a reading log, tracking how much reading you are doing at what time. We present an approach capable of estimating the words read by a user, evaluate it in an user independent approach over 3 experiments with 24 users over 5 different devices (e-ink reader, smartphone, tablet, paper, computer screen). We achieve an error rate as low as 5% (using a medical electrooculography system) or 15% (based on eye movements captured by optical eye tracking) over a total of 30 hours of recording. Our method works for both an optical eye tracking and an Electrooculography system. We provide first indications that the method works also on soon commercially available smart glasses.

Reading-life log is a research field of analyzing our activities of reading documents to know more about readers and documents. In this paper we propose an implementation of reading-life log which is to estimate the English language skill by analyzing the activities of reading English documents. As input for the analysis, we employ eye movement information, because we consider the eye movement of skillful readers is far different from that of novices. From the experiments, we have found that the following two features are informative: (1) the sum of fixation duration, and (2) the sum of the velocity of saccades. By using these features the proposed method is to estimate the class of English skill from among low, middle and high, which are defined based on the scores of English standardized test called TOEIC. From the experimental results with 11 subjects and 10 documents, we have been successful to estimate the class with the accuracy of 90.9%.

We introduce the Wordometer, a novel method to estimate the number of words a user reads using a mobile eye tracker and document image retrieval. We present a reading detection algorithm which works with over 91 % accuracy over 10 test subjects using 10-fold cross validation. We implement two algorithms to estimate the read words using a line break detector. A simple version gives an average error rate of 13,5 % for 9 users over 10 documents. A more sophisticated word count algorithm based on support vector regression with an RBF kernel reaches an average error rate from only 8.2 % (6.5 % if one test subject with abnormal behavior is excluded). The achieved error rates are comparable to pedometers that count our steps in our daily life. Thus, we believe the Wordometer can be used as a step counter for the information we read to make our knowledge life healthier.

The document analysis community spends substantial resources towards computer recognition of any type of text (e.g. characters, handwriting, document structure etc.). In this paper, we introduce a new paradigm focusing on recognizing the activities and habits of users while they are reading. We describe the differences to the traditional approaches of document analysis. We present initial work towards recognizing reading activities. We report our initial findings using a commercial, dry electrode Electroencephalography (EEG) system. We show the feasibility to distinguish reading tasks for 3 different document genres with one user and near perfect accuracy. Distinguishing reading tasks for 3 different document types we achieve 97 % with user specific training. We present evidence that reading and non-reading related activities can be separated over 3 users using 6 classes, perfectly separating reading from non-reading. A simple EEG system seems promising for distinguishing the reading of different document genres.

In this paper, we investigate how smartphone applications, in particular web browsers, are used on mobile phones. Using a publicly available widget for smart phones, we recorded app usage and the phones' acceleration and orientation from 1,330 devices. Combining app usage and sensor data we derive the device's typical posture while different apps are used. Analyzing motion data shows that devices are moved more while messaging and navigation apps are used as opposed to browser and other common applications. The time distribution between landscape and portrait depicts that most of the landscape mode time is used for burst interaction (e.g., text entry), except for Media apps, which are mostly used in landscape mode. Additionally, we found that over 31% of our users use more than one web browser. Our analysis reveals that the duration of mobile browser sessions is longer by a factor of 1.5 when browsers are explicitly started through the system's launcher in comparison to being launched from within another app. Further, users switch back and forth between apps and web browsers, which suggest that a tight and smooth integration of web browsers with native apps can improve the overall usability. From our findings we derive design guidelines for app developers.

We present an elaborate experimental study of the influence of device placement on the GPS location accuracy in three different mobile appliances: iphone 3gs, nokia n95 and nokia 810. On a total of some 52 km of walking traces we shaw that there are statistically significant difference in the errors between 5 on body locations: hand, front pocket of trousers, inner pocket of the jacket, inside a backpack. While the fact that such difference exists was well known before, this paper is the first to systematically study the effect.

Opportunistic sensing allows to efficiently collect information about the physical world and the persons behaving in it. This may mainstream human context and activity recognition in wearable and pervasive computing by removing requirements for a specific deployed infrastructure. In this paper we introduce the newly started European research project OPPORTUNITY within which we develop mobile opportunistic activity and context recognition systems. We outline the project's objective, the approach we follow along opportunistic sensing, data processing and interpretation, and autonomous adaptation and evolution to environmental and user changes, and we outline preliminary results.

We present a set of heuristics that significantly increase the robustness of motion sensor-based activity recognition with respect to sensor displacement. In this paper placement refers to the position within a single body part (e.g, lower arm). We show how, within certain limits and with modest quality degradation, motion sensorbased activity recognition can be implemented in a displacement tolerant way. We first describe the physical principles that lead to our heuristic. We then evaluate them first on a set of synthetic lower arm motions which are well suited to illustrate the strengths and limits of our approach, then on an extended modes of locomotion problem (sensors on the upper leg) and finally on a set of exercises performed on various gym machines (sensors placed on the lower arm). In this example our heuristic raises the displaced recognition rate from 24% for a displaced accelerometer, which had 96% recognition when not displaced, to 82%.