In the era of ubiquitous computing, people expect applications to work across different devices. To provide a seamless user experience it is therefore crucial that interfaces and interactions are consistent across different device types. In this paper, we present a method to create gesture sets that are consistent and easily transferable. Our proposed method entails 1) the gesture elicitation on each device type, 2) the consolidation of a unified gesture set, and 3) a final validation by calculating a transferability score. We tested our approach by eliciting a set of user-defined gestures for reading with Rapid Serial Visual Presentation (RSVP) of text for three device types: phone, watch, and glasses. We present the resulting, unified gesture set for RSVP reading and show the feasibility of our method to elicit gesture sets that are consistent across device types with different form factors.

Devices with touchscreens have an inherent latency. When a user's finger drags an object across the screen the object follows with a latency of around 100ms for current devices. Previous work showed that latencies down to 25ms reduce users' performance and that even 10ms latency is noticeable. In this paper we demonstrate an approach that reduces latency using a predictive model. Extrapolating the finger's movement we predict where the finger will be in the next moment. Comparing different prediction approaches we show for three different tasks that prediction using neural networks is more precise than linear and polynomial extrapolation. Furthermore, we show through a Fitts' Law dragging experiment that reducing touch latency can significantly increases users' performance. As the approach is software-based it can easily be integrated into existing mobile applications and systems.

Recent work demonstrated the exciting opportunities that thermal imaging offers for the development of interactive systems. It was shown that a thermal camera can sense when a user touches a surface, performs gestures in the camera's direct field of view and, in addition, performs gestures outside the camera's direct field of view through thermal reflection. In this paper, we investigate the material properties that should be considered for detecting interaction using thermal imaging considering both in- and outdoor settings. We conducted a study to analyze the recognition performance for different gestures and different surfaces. Using the results, we derive guidelines on material properties of surfaces for detecting on-surface as well as mid-air interaction using a thermal camera. We discuss the constrains that should be taken into account using thermal imaging as the sensing technology. Finally, we present a material space based on our findings. The space depicts surfaces and the required properties that enable the different interaction techniques.

The advent of wearables (e.g., smartwatches, smartglasses, and digital jewelry) anticipates the need for text entry methods on very small devices. We conduct fundamental research on this topic using 3 qwerty-based soft keyboards for 3 different screen sizes, motivated by the extensive training that users have with qwerty keyboards. In addition to ZoomBoard (a soft keyboard for diminutive screens), we propose a callout-based soft keyboard and ZShift, a novel extension of the Shift pointing technique. We conducted a comprehensive user study followed by extensive analyses on performance, usability, and short-term learning. Our results show that different small screen sizes demand different types of assistance. In general, manufacturers can benefit from these findings by selecting an appropriate qwerty soft keyboard for their devices. Ultimately, this work provides designers, researchers, and practitioners with new understanding of qwerty soft keyboard design space and its scalability for tiny touchscreens.

Pixel densities are increasing rapidly. We can observe this trend in particular for mobile devices like smartphones and tablets. Previous work revealed an effect of pixel density on subjective feedback and objective performance only for low resolution cathode ray tube screens. It is unclear if this effect persists for the four times higher pixel densities of current mobile devices. Therefore, we conducted a study to compare four pixel densities with 359, 180, 120, and 90 pixels per inch. While participants performed three tasks involving images, text and videos on a tablet, we measured perceived effort, perceived visual quality, task completion time, error rate, and body pose. Our results show that the effect of the pixel density highly depends on the content. We found that only for text, the four pixel densities have clearly different perceived media qualities. Pixel density seems to have a smaller effect on perceived media quality for images and videos and we found no effect on objective measures. Results show that text should be displayed in high resolution, while this is less important for images and videos.

Order Picking is not only one of the most important but also most mentally demanding and error-prone tasks in the industry. Both stationary and wearable systems have been introduced to facilitate this task. Existing stationary systems are not scalable because of the high cost and wearable systems have issues being accepted by the workers. In this paper, we introduce a mobile camera-projector cart called OrderPickAR, which combines the benefits of both stationary and mobile systems to support order picking through Augmented Reality. Our system dynamically projects in-situ picking information into the storage system and automatically detects when a picking task is done. In a lab study, we compare our system to existing approaches, i.e, Pick-by-Paper, Pick-by-Voice, and Pick-by-Vision. The results show that using the proposed system, order picking is almost twice as fast as other approaches, the error rate is decreased up to 9 times, and mental demands are reduced up to 50%.

Common user authentication methods on smartphones, such as lock patterns, PINs, or passwords, impose a trade-off between security and password memorability. Image-based passwords were proposed as a secure and usable alternative. As of today, however, it remains unclear how such schemes are used in the wild. We present the first study to investigate how image-based passwords are used over long periods of time in the real world. Our analyses are based on data from 2318 unique devices collected over more than one year using a custom application released in the Android Play store. We present an in-depth analysis of what kind of images users select, how they define their passwords, and how secure these passwords are. Our findings provide valuable insights into real-world use of image-based passwords and inform the design of future graphical authentication schemes.

Eating is not only one of the most fundamental human needs but also among the most regular activities. Acquiring food, preparing meals, and socializing around food are deeply rooted in all human cultures. In this paper we show how food can not only serve to satisfy hunger but also become a new display technology. Through food augmentation, a dinner could communicate its ingredients, convey messages, or provide instructions such as the recipe of a meal. We show how to augment a large range of food with laser. We conducted a series of focus groups to gather people's first impressions and derive a broad range of meaningful augmentation scenarios. We discuss the perceived benefits, opportunities, and concerns. Additionally, we evaluated a number of scenarios through an online survey. The most readily accepted augmentation scenarios include adding practical information, increasing awareness about the food, and augmenting food items with a natural skin.

Thermal cameras have recently drawn the attention of HCI researchers as a new sensory system enabling novel interactive systems. They are robust to illumination changes and make it easy to separate human bodies from the image background. Far-infrared radiation, however, has another characteristic that distinguishes thermal cameras from their RGB or depth counterparts, namely thermal reflection. Common surfaces reflect thermal radiation differently than visual light and can be perfect thermal mirrors. In this paper, we show that through thermal reflection, thermal cameras can sense the space beyond their direct field-of-view. A thermal camera can sense areas besides and even behind its field-of-view through thermal reflection. We investigate how thermal reflection can increase the interaction space of projected surfaces using camera-projection systems. We moreover discuss the reflection characteristics of common surfaces in our vicinity in both the visual and thermal radiation bands. Using a proof-of-concept prototype, we demonstrate the increased interaction space for hand-held camera-projection system. Furthermore, we depict a number of promising application examples that can benefit from the thermal reflection characteristics of surfaces.

Notifications are a core feature of mobile phones. They inform users about a variety of events. Users may take immediate action or ignore them depending on the importance of a notification as well as their current context. The nature of notifications is manifold, applications use them both sparsely and frequently. In this paper we present the first large-scale analysis of mobile notifications with a focus on users' subjective perceptions. We derive a holistic picture of notifications on mobile phones by collecting close to 200 million notifications from more than 40,000 users. Using a data-driven approach, we break down what users like and dislike about notifications. Our results reveal differences in importance of notifications and how users value notifications from messaging apps as well as notifications that include information about people and events. Based on these results we derive a number of findings about the nature of notifications and guidelines to effectively use them.

Touch gestures become steadily more important with the ongoing success of touch screen devices. Compared to traditional user interfaces, gestures have the potential to lower cognitive load and the need for visual attention. However, nowadays gestures are defined by designers and developers and it is questionable if these meet all user requirements. In this paper, we present two exploratory studies that investigate how users would use unistroke touch gestures for shortcut access to a mobile phone's key functionalities. We study the functions that users want to access, the preferred activators for gesture execution, and the shapes of the user-invented gestures. We found that most gestures trigger applications, letter-shaped gestures are preferred, and the gestures should be accessible from the lock screen, the wallpaper, and the notification bar. We conclude with a coherent, unambiguous set of gestures for the 20 most frequently accessed functions, which can inform the design of future gesture-controlled applications.

Falling hardware prices and ever more displays being connected to the Internet will lead to large public display networks, potentially forming a novel communication medium. We envision that such networks are not restricted to display owners and advertisers anymore, but allow also passersby (e.g., customers) to exchange content, similar to traditional public notice areas, such as bulletin boards. In this context it is crucial to understand emerging practices and provide easy and straight forward interaction techniques to be used for creating and exchanging content. In this paper, we present Digifieds, a digital public notice area we built to investigate and compare possible interaction techniques. Based on a lab study we show that using direct touch at the display as well as using the mobile phone as a complementing interaction technology are most suitable. Direct touch at the display closely resembles the interaction known from classic bulletin boards and provides the highest usability. Mobile phones preserve the users' privacy as they exchange (sensitive) data with the display and at the same time allow content to be created on-the-go or to be retrieved.

In this paper, we investigate how smartphone applications, in particular web browsers, are used on mobile phones. Using a publicly available widget for smart phones, we recorded app usage and the phones' acceleration and orientation from 1,330 devices. Combining app usage and sensor data we derive the device's typical posture while different apps are used. Analyzing motion data shows that devices are moved more while messaging and navigation apps are used as opposed to browser and other common applications. The time distribution between landscape and portrait depicts that most of the landscape mode time is used for burst interaction (e.g., text entry), except for Media apps, which are mostly used in landscape mode. Additionally, we found that over 31% of our users use more than one web browser. Our analysis reveals that the duration of mobile browser sessions is longer by a factor of 1.5 when browsers are explicitly started through the system's launcher in comparison to being launched from within another app. Further, users switch back and forth between apps and web browsers, which suggest that a tight and smooth integration of web browsers with native apps can improve the overall usability. From our findings we derive design guidelines for app developers.

Secure user authentication on mobile phones is crucial, as they store highly sensitive information. Common approaches to authenticate a user on a mobile phone are based either on entering a PIN, a password, or drawing a pattern. However, these authentication methods are vulnerable to the shoulder surfing attack. The risk of this attack has increased since means for recording high-resolution videos are cheaply and widely accessible. If the attacker can videotape the authentication process, PINs, passwords, and patterns do not even provide the most basic level of security. In this project, we assessed the vulnerability of a magnetic gestural authentication method to the video-based shoulder surfing attack. We chose a scenario that is favourable to the attack-er. In a real world environment, we videotaped the interactions of four users performing magnetic signatures on a phone, in the presence of HD cameras from four different angles. We then recruited 22 participants and asked them to watch the videos and try to forge the signatures. The results revealed that with a certain threshold, i.e, th=1.67, none of the forging attacks was successful, whereas at this level all eligible login attempts were successfully recognized. The qualitative feedback also indicated that users found the magnetic gestural signature authentication method to be more secure than PIN-based and 2D signature methods.

High-speed Internet connectivity makes browsing a convenient task. However, there are many situations in which surfing the web is still slow due to limited bandwidth, slow servers, or complex queries. As a result, loading web pages can take several seconds, making (mobile) browsing cumbersome. We present an approach which makes use of the time spent on waiting for the next page, by bridging the wait with extra cached or preloaded content. We show how the content (e.g., news, Twitter) can be adapted to the user's interests and to the context of use, hence making mobile surfing more comfortable. We compare two approaches: in time-multiplex mode, the entire screen displays bridging content until the loading is finished. In space-multiplex mode, content is displayed alongside the requested content while it loads. We use an HTTP proxy to intercept requests and add JavaScript code, which allows the bridging content from websites of our choice to be inserted. The approach was evaluated with 15 participants, assessing suitable content and usability.

Even with the rise of the World Wide Web, TV has remained the most pervasive entertainment medium and is nowadays often used together with other media, which allow for active participation. The idea of connecting non-collocated TV viewers via telecommunication technologies, referred to as Social TV, has recently received considerable attention. Such systems typically include set-top boxes for supporting collaboration. In this research we investigate if real-time opinion sharing about TV shows through a nonverbal (non-textual) iconic UI on mobile phones is reasonable. For this purpose we developed a mobile app, made it available to a large number of users through the Android Market, and conducted an uncontrolled user study in the wild during the soccer world cup 2010. The results of the study indicate that TV viewers who used the app had more fun and felt more connected to other viewers. We also show that by monitoring this channel it is possible to collect sentiments relevant to the broadcasted content in real-time. The collected data exemplify that the aggregated sentiments correspond to important moments, and hence can be used to generate a summary of the event.

Traditional public notice areas (PNAs) are nowadays a popular means to publicly exchange information and reach people of a local community. The high usability led to a wide-spread use in stores, cafes, supermarkets, and public institutions. With public displays permeating public spaces and with display providers and owners being willing to share parts of their display space we envision traditional PNAs to be complemented or even replaced by their digital counterparts in the future, hence contributing to making public displays a novel communication medium. In this paper we report on the design and development of Digifieds (derived from digital classified), a digital public notice area. We deployed and evaluated Digifieds in an urban environment in the context of the UbiChallenge 2011 in Oulu, Finland over the course of 6 months. The deployment allowed the users' view to be studied with regard to the envisioned content, preferred interaction techniques, as well as privacy concerns, and to compare them against traditional PNAs.

People spend a significant amount of time in their cars (US: 86 minutes/day, Europe: 43 minutes/day) while commuting, shopping, or traveling. Hence, the variety of entertainment in the car increases, and many vehicles are already equipped with displays, allowing for watching news, videos, accessing the Internet, or playing games. At the same time, the urbanization caused a massive increase of traffic volume, which led to people spending an ever-increasing amount of their time in front of red traffic lights. An observation of the prevailing forms of entertainment in the car reveals that content such as text, videos, or games are often a mere adaptation of content produced for television, public displays, PCs, or mobile phones and do not adapt to the situation in the car. In this paper we report on a web survey assessing which forms of entertainment and which types of content are considered to be useful for in-car entertainment by drivers. We then introduce an algorithm, which is capable of learning standing times in front of traffic lights based on GPS information only. This, on one hand, allows for providing content of appropriate length, on the other hand, for directing the attention of the driver back to-wards the street at the right time. Finally, we present a prototype implementation and a qualitative evaluation.

In this paper, we explore the potential of combining shared and interactive displays (e.g. a multi-touch table) with personal devices (e.g. mobile phones) as an important class of heterogeneous multi-display environments. Within six case studies applications and interactions were invented and implemented that utilize the potential of such heterogeneous multi-display environments. We were in particular interested how to design systems that include interaction across different displays and how to manage public and private information in a group setting. One case study, a digital card game, highlights these design challenges. We explore different natural ways of interaction, including touching the table as well as motion gestures with mobile phones. With this application we provide a use case to discuss gestures combining mobile phones with tabletop surfaces, as well as to explore a private-public display setting. First results showed that combining tables and mobile phones provide a suitable and understandable way for interaction in these settings.

Despite the availability of other mobile messaging applications, SMS has kept its position as a heavily used communication technology. However, there are many situations in which it is inconvenient or inappropriate to check a message's content immediately. In this paper, we introduce the concept of audio previews of SMS. Based on a real-time analysis of the content of a message, we provide auditory cues in addition to the notification tone upon receiving an SMS. We report on a field trial with 20 participants and show that the use of audio-enhanced SMS affects the reading and writing behavior of users. Our work is motivated by the results of an online survey among 347 SMS users of whose we analyzed 3400 text messages.

Context-awareness is one of the rising trends of future mobile technology, and due to advances in technology development, new application and service concepts are being developed and demonstrated in an ever-increasing manner. This workshop brings together researchers and practitioners working on humancomputer interaction (HCI) aspects of context-aware mobile technology and communities to present their insights and research on new concepts, interaction design for mobile context-awareness, usability challenges, collaborative context-aware services and applications for supporting communities, and other topics related to HCI with mobile context-aware technology.

In their role as personal communication devices, mobile phones are a natural choice for sharing and communicating emotions. However, their functionalities are currently very limited in power to express affective messages. In this paper, we describe the design of a system that allows users to easily compose melodies and share them via mobile phones. We show that by using these melodies information about the current emotional state of the sender can be expressed and recognized synchronously by the receiver in a simple, quick, and unobtrusive way. Further, we reveal that self-composed melodies have a stronger impact than pre-composed or downloaded messages, similar to crafted pieces of art offered to a beloved person. We then present findings from a user study that assesses the implementation of a functional prototype and the adequacy of the system for emotional communication.

In this paper, we introduce the design and implementation of a digital card game that combines mobile phone gestures and multi-touch tabletop interactions. Conceptually based on interactions in traditional card games, our Poker game application explores different natural ways of interaction, including touching the table as well as tilting, throwing, and shaking a mobile phone. By exemplarily translating traditional gestures into the digital domain, we provide a use case to discuss useful gestures combing mobile phones with tabletop surfaces, as well as to explore a private-public display setting for digital card games at interactive tables. Additionally, the mobile phone provides a tangible feeling similar to physical cards. The preliminary user study showed that users preferred using mobile phones for interaction compared to direct interaction on the multi-touch table.

In this paper, we introduce Flashlight interaction, a new approach to light-based interaction between mobile phones and large screens by using the phone camera flashlight. Using a mobile phone as an interaction device with large screens has been explored in various projects. Mobile phones are enhanced with different sensors and provide opportunities to be used as an interaction device with other devices such as a large display. Our approach supports to set up a private-public display setting and provide simple means for interaction without a wireless connectivity. Our user study results indicate that the interaction is easy to perform and understand due to the direct mapping between the phone movement and the response on the screen.