We show a new type of side-channel leakage in which the built-in magnetometer sensor in Apple's mobile devices captures touch events of users. When a conductive material such as the human body touches the mobile device screen, the electric current passes through the screen capacitors generating an electromagnetic field around the touch point. This electromagnetic field leads to a sharp fluctuation in the magnetometer signals when a touch occurs, both when the mobile device is stationary and held in hand naturally. These signals can be accessed by mobile applications running in the background without requiring any permissions. We develop iSTELAN, a three-stage attack, which exploits this side-channel to infer users' application and touch data. iSTELAN translates the magnetometer signals to a binary sequence to reveal users' touch events, exploits touch event patterns to fingerprint the type of application a user is using, and models touch events to identify users' touch event types performed on different applications. We demonstrate the iSTELAN attack on 22 users while using 7 popular app types and show that it achieves an average accuracy of 90% for disclosing touch events, 74% for classifying application type used, and 73% for detecting touch event types.

Although existing surveillance cameras can identify people, their utility is limited by the unavailability of any direct camera-to-human communication. This paper proposes a real-time end-to-end system to solve the problem of digitally associating people in a camera view with their smartphones, without knowing the phones' IP/MAC addresses. The key idea is using a person's unique "context features", extracted from videos, as its sole address. The context address consists of: motion features, e.g. walking velocity; and ambience features, e.g. magnetic trend and Wi-Fi signal strengths. Once receiving a broadcast packet from the camera, a user's phone accepts it only if its context address matches the phone's sensor data. We highlight three novel components in our system: (1) definition of discriminative and noise-robust ambience features; (2) effortless ambient sensing map generation; (3) a context feature selection algorithm to dynamically choose lightweight yet effective features which are encoded into a fixed-length header. Real-world and simulated experiments are conducted for different applications. Our system achieves a sending ratio of 98.5%, an acceptance precision of 93.4%, and a recall of 98.3% with ten people. We believe this is a step towards direct camera-to-human communication and will become a generic underlay to various practical applications.

Facial authentication mechanisms are gaining traction on smartphones because of their convenience and increasingly good performance of face recognition systems. However, mainstream systems use traditional 2D face recognition technologies, which are vulnerable to various spoofing attacks. Existing systems perform liveness detection via specialized hardware, such as infrared dot projectors and dedicated cameras. Although effective, such methods do not align well with the smartphone industry's desire to maximize screen space.

This video presents a real-time end-to-end system which enables cameras to send personalized messages to people in a public area without knowing any addresses of their mobile phones. For facilitating this communication, we solve the problem of digitally associating people in the camera view with their smartphones without prior knowledge of the phones' IP/MAC addresses. The system doesn't need any dedicated devices and doesn't request people to wear digital tags. It utilizes users' motion patterns and leverages the diversity in motion features as the address for communication. The cameras broadcast a message to all the phones in the camera view using the target's motion features as the destination. Then a user's phone can locally compare the "motion address" of the packet against its own sensor data and will accept the packet if it's a "good" match. To protect the privacy of users' sensor data, we keep the users' personal sensing data on their phones instead of asking them to upload the data to server. Moreover, to prevent users' walking behavior from being revealed to public, we transform the raw motion features via principal component analysis (PCA) while maintaining their distinguishing power. On the whole, our system achieves 98%, 95%, 90%, 90%, 87% matching correctness for 2, 4, 6, 8 and 10 users respectively.

The rapid adoption of smartphones with different types of advanced sensors has led to an increasing trend in the usage of mobile crowdsensing applications, e.g., to create hyper-local weather maps. However, the high energy consumption of crowdsensing, chiefly due to expensive network communication, has been found to be detrimental to the wide-spread adoption. We propose a framework, called Sense-Aid, that can provide energy-efficient mobile crowdsensing service, coexisting with the cellular network. There are two key innovations in Sense-Aid beyond prior work (Piggyback Crowdsensing-Sensys13)---the middleware running on the cellular network edge to orchestrate multiple devices present in geographical proximity to suppress redundant data collection and communication. It understands the state of each device (radio state, battery state, etc.) to decide which ones should be selected for crowdsensing activities at any point in time. It also provides a simple programming abstraction to help with the development of crowdsensing applications. We show the benefit of Sense-Aid by conducting a user study consisting of 60 students in our campus, compared to a baseline periodic data collection method and Piggyback Crowdsensing. We find that energy saving is 93.3% for Sense-Aid compared with Piggyback Crowdsensing in a representative case which requires 2 devices to provide barometric values within an area of a circle whose radius is 1 kilometer and requires periodic data collection every 5 minutes for a 90-minute test. The selection algorithm of Sense-Aid also ensures reasonable fairness in the use of the different devices.

This paper aims to track the 3D posture of the entire arm - both wrist and elbow - using the motion and magnetic sensors on smartwatches. We do not intend to employ machine learning to train the system on a specific set of gestures. Instead, we aim to trace the geometric motion of the arm, which can then be used as a generic platform for gesture-based applications. The problem is challenging because the arm posture is a function of both elbow and shoulder motions, whereas the watch is only a single point of (noisy) measurement from the wrist. Moreover, while other tracking systems (like indoor/outdoor localization) often benefit from maps or landmarks to occasionally reset their estimates, such opportunities are almost absent here.

Indoor localization using mobile sensors has gained momentum lately. Most of the current systems rely on an extensive calibration step to achieve high accuracy. We propose SemanticSLAM, a novel unsupervised indoor localization scheme that bypasses the need for war-driving. SemanticSLAM leverages the idea that certain locations in an indoor environment have a unique signature on one or more phone sensors. Climbing stairs, for example, has a distinct pattern on the phone's accelerometer; a specific spot may experience an unusual magnetic interference while another may have a unique set of Wi-Fi access points covering it. SemanticSLAM uses these unique points in the environment as landmarks and combines them with dead-reckoning in a new Simultaneous Localization And Mapping (SLAM) framework to reduce both the localization error and convergence time. In particular, the phone inertial sensors are used to keep track of the user's path, while the observed landmarks are used to compensate for the accumulation of error in a unified probabilistic framework. Evaluation in two testbeds on Android phones shows that the system can achieve 0.53 meters human median localization errors. In addition, the system can detect the location of landmarks with 0.83 meters median error. This is 62 percent better than a system that does not use SLAM. Moreover, SemanticSLAM has a 33 percent lower convergence time compared to the same systems. This highlights the promise of SemanticSLAM as an unconventional approach for indoor localization.

Imagine a user typing on a laptop keyboard while wearing a smart watch. This paper asks whether motion sensors from the watch can leak information about what the user is typing. While its not surprising that some information will be leaked, the question is how much? We find that when motion signal processing is combined with patterns in English language, the leakage is substantial. Reported results show that when a user types a word $W$, it is possible to shortlist a median of 24 words, such that $W$ is in this shortlist. When the word is longer than $6$ characters, the median shortlist drops to $10$. Of course, such leaks happen without requiring any training from the user, and also under the (obvious) condition that the watch is only on the left hand. We believe this is surprising and merits awareness, especially in light of various continuous sensing apps that are emerging in the app market. Moreover, we discover additional "leaks" that can further reduce the shortlist -- we leave these exploitations to future work.

This paper develops techniques using which humans can be visually recognized. While face recognition would be one approach to this problem, we believe that it may not be always possible to see a person?s face. Our technique is complementary to face recognition, and exploits the intuition that human motion patterns and clothing colors can together encode several bits of information. Treating this information as a "temporary fingerprint", it may be feasible to recognize an individual with reasonable consistency, while allowing her to turn off the fingerprint at will.

This paper describes WalkCompass, a system that exploits smartphone sensors to estimate the direction in which a user is walking. We find that several smartphone localization systems in the recent past, including our own, make a simplifying assumption that the user's walking direction is known. In trying to relax this assumption, we were not able to find a generic solution from past work. While intuition suggests that the walking direction should be detectable through the accelerometer, in reality this direction gets blended into various other motion patterns during the act of walking, including up and down bounce, side-to-side sway, swing of arms or legs, etc. Moreover, the walking direction is in the phone's local coordinate system (e.g., along Y axis), and translation to global directions, such as 45 degree North, can be challenging when the compass is itself erroneous. WalkCompass copes with these challenges and develops a stable technique to estimate the user's walking direction within a few steps. Results drawn from 15 different environments demonstrate median error of less than 8 degrees, across 6 different users, 3 surfaces, and 3 holding positions. While there is room for improvement, we believe our current system can be immediately useful to various applications centered around localization and human activity recognition.

We present a demonstration of WalkCompass, a system to appear in the MobiSys 2014 main conference. WalkCompass exploits smartphone sensors to estimate the direction in which a user is walking. We find that several smartphone localization systems in the recent past, including our own, make a simplifying assumption that the user's walking direction is known. In trying to relax this assumption, we were not able to find a generic solution from past work. While intuition suggests that the walking direction should be detectable through the accelerometer, in reality this direction gets blended into various other motion patterns during the act of walking, including up and down bounce, side-to-side sway, swing of arms or legs, etc. WalkCompass analyzes the human walking dynamics to estimate the dominating forces and uses this knowledge to find the heading direction of the pedestrian. In the demonstration we will show the performance of this system when the user holds the smartphone on the palm. A collection of YouTube videos of the demo is posted at http://synrg.csl.illinois.edu/projects/ localization/walkcompass.

No abstract available.

This video presents a demo of indoor localization in multiple settings. In the demo, a user walks with a smartphone and the user's location is shown on the phone's screen in real time. Our system, called Unsupervised Indoor Localization (UnLoc) utilizes the sensor data from smartphones to learn "invisible landmarks" in the environment. Example landmarks could be a unique magnetic fluctuation experienced when the phone is near a water-cooler, or a distinct gyroscope rotation when the user turns a corner. We use these indoor "landmarks" to periodically reset the user's location. To track the user between these landmarks, we use an optimized variant of dead reckoning, ultimately leading to a robust location tracking system. We call our system UnLoc, since the landmarks are generated in an unsupervised manner, requiring no manual effort or floorplan of the building. The demo describes the high level intuitions, shows UnLoc in operation, and shares experiences from running UnLoc in various real-world environments.

Local search users today decide what business to visit solely based on distance information, and business ratings that can be sparse or stale. We believe that when users search for local businesses, such as bars or restaurants, they need to know more about the ambience of each business, such as how crowded it is, how loud and of what type the music it plays is, as well as how loud the human chatter in the business is. Unfortunately, this information doesn't exist today. In this paper, we propose to automatically crowdsource such rich, local business ambience metadata through real user check-in events. Every time a user checks into a business, the phone is in user's hands, and the phone's sensors can sense the business environment. We leverage the phone's microphone during this time to infer the occupancy and human chatter levels, the music type, as well as the music and noise levels in the business. As people check-in to businesses throughout the day, business metadata can be automatically updated over time, enabling a new generation of local search experience. Using approximately 150 audio traces collected from real businesses of various types over a period of 3 months, we show that by properly extracting the temporal and frequency signatures of the audio signal, it is feasible to train models that can simultaneously infer occupancy, human chatter, music, and noise levels in a business, with higher than 79% accuracy.

The proliferation of location-based services and applications calls for provisioning of location service as a first class system component that can return accurate location fix in short response time and is energy efficient. In this paper, we present the design, implementation and evaluation of WheelLoc - a continuous system location service for outdoor scenarios. Unlike previous localization efforts that try to directly obtain a point location fix, WheelLoc adopts an indirect approach: it seeks to capture a user mobility trace first and to obtain any point location by time- and speed-aware interpolation or extrapolation. WheelLoc avoids energy-expensive sensors completely and relies solely on commonly available cheap sensors such as accelerometer and magnetometer. With a set of novel techniques and the leverage of publicly available road maps and cell tower information, WheelLoc is able to meet those requirements of a first class component. Experimental results confirmed the effectiveness of WheelLoc. It can return a location estimate within 40ms with an accuracy about 40 meters, consumes only 240mW energy, and effectively strikes a better energy-accuracy tradeoff than GPS duty-cycling.

We propose UnLoc, an unsupervised indoor localization scheme that bypasses the need for war-driving. Our key observation is that certain locations in an indoor environment present identifiable signatures on one or more sensing dimensions. An elevator, for instance, imposes a distinct pattern on a smartphone's accelerometer; a corridor-corner may overhear a unique set of WiFi access points; a specific spot may experience an unusual magnetic fluctuation. We hypothesize that these kind of signatures naturally exist in the environment, and can be envisioned as internal landmarks of a building. Mobile devices that "sense" these landmarks can recalibrate their locations, while dead-reckoning schemes can track them between landmarks. Results from 3 different indoor settings, including a shopping mall, demonstrate median location errors of 1:69m. War-driving is not necessary, neither are floorplans the system simultaneously computes the locations of users and landmarks, in a manner that they converge reasonably quickly. We believe this is an unconventional approach to indoor localization, holding promise for real-world deployment.

We propose UnLoc [1], an unsupervised indoor localization scheme that bypasses the need for war-driving. Our key observation is that certain locations in an indoor environment present an identifiable signature on one or more sensing dimensions. An elevator, for instance, imposes a distinct pattern on a smartphone's accelerometer; a specific spot may experience an unusual magnetic fluctuation. This form of urban sensing and activity recognition has already been demonstrated in literature [2, 3], but not yet applied in pure localization applications. We hypothesize that these kind of signatures naturally exist in the environment and can be envisioned as internal landmarks of a building. Mobile devices that "sense" these landmarks can recalibrate their locations, while dead-reckoning schemes can track them between landmarks. Neither war-driving nor floorplans are necessary - the system simultaneously computes the locations of users and landmarks, in a manner so that they converge reasonably quickly. We believe this is an unconventional approach to indoor localization, holding promise for real-world deployment.