The proliferation of mobile and IoT devices has sparked growing interest in multi-device interaction, where a single app operates across multiple devices to leverage their diverse capabilities. However, the current methods of app development and usage remain bound to a single-device paradigm, rendering multi-device apps difficult to implement and deploy. This paper presents COMET , a novel mobile app framework that enables the dynamic distribution of app components across multiple devices at runtime, supporting a wide range of multi-device scenarios, including collaborative applications, smart environments, and interactions across heterogeneous personal devices. COMET enables developers to specify distributable components using lightweight annotations and employs build-time code instrumentation to automate the deployment and execution of selected components onto remote devices. This approach supports multi-device interaction with minimal developer effort and without requiring system-level modifications. To realize this, COMET addresses four key challenges: (i) the static partitioning of app components and extraction of their dependencies at build time, (ii) efficient execution of these components on remote devices, (iii) preservation of intercomponent communication across devices, and (iv) synchronization of distributed components that share a global state. We implemented a prototype of COMET on Android and evaluated it using real-world apps. Our evaluation with eight case-study apps shows that component distribution completes within 251.8 ms. A user study with 15 participants further demonstrates that COMET provides intuitive multi-device interaction with acceptable responsiveness.

Back-of-Device (BoD) interfaces have emerged as a promising solution to free up screen real estate in smartphones by offloading

On-device machine learning applications are increasingly deployed in dynamic and open system environments, where resource availability fluctuates unpredictably. This variability, coupled with limited computing resources, poses significant challenges in achieving high responsiveness. Existing on-device machine learning frameworks typically rely on static and coarse-grained resource allocation, leading to performance degradation under resource contention. To address this, we propose FlexOn, a novel framework that combines fine-grained model segmentation and dynamic resource selection to rapidly adapt to highly dynamic runtime conditions and effectively mitigate unpredictable resource contention. A prototype built on LiteRT demonstrates significant improvements in both average and tail latencies of up to 54% and 58%, respectively, across three different embedded platforms under dynamically varying resource availability. To the best of our knowledge, this is the first work that addresses the resource contention in open embedded systems for better machine learning inference responsiveness.

PIN authentication is the first line of defense for protecting private data on many smartphone applications, such as lock screens, messengers, and banking apps. However, existing PIN authentication systems have several constraints regarding security, usability, and robustness. To go beyond their limitations, this paper presents TaPIN, a reliable system that authenticates smartphone users with the collaborative use of PINs and tap biometrics. A user is first instructed to enter her PIN by tapping a smartphone screen for authentication. During the PIN entry, the user's fingertip collides with the screen, producing user-specific vibration and sound signals. TaPIN then senses the tap-induced signals and the collision properties, e.g., pressures and sizes, using the smartphone's built-in sensors and leverages them as biometric features. That is, it authenticates the user by verifying not only the entered PIN but also the collected features. Our experiments with 20 real-world users demonstrate that this two-factor authentication system is easy to use, more secure than existing methods, and deployable without dedicated hardware. For example, it accurately authenticates users with an average EER of 1.9% in stationary environments and maintains a reasonable level of security regardless of devices, tap styles, and noise.

Sign language is the primary communication medium for people who are deaf or have hearing loss. However, given the divergent range of sensory abilities of these individuals, there is a communication gap that needs to be addressed. In this paper, we present action-concentrated embedding (ACE), which is a novel sign token embedding framework. Additionally, to provide a more structured foundation for sign language analysis, we introduce a dedicated notation system tailored for sign language that endeavors to encapsulate the nuanced gestures and movements that are integral with sign communication. The proposed ACE approach tracks a signer’s actions based on human posture estimation. Tokenizing these actions and capturing the token embedding using a short-time Fourier transform encapsulates the time-based behavioral changes. Hence, ACE offers input embedding to translate sign language into natural language sentences. When tested against a disaster sign language dataset using automated machine translation measures, ACE notably surpasses prior research in terms of translation capabilities, improving the performance by up to 5.79% for BLEU-4 and 5.46% for ROUGE-L metric.

Samsung Pay, a widely-used mobile payment service, enables users to pay using just their smartphone thanks to Magnetic Secure Transmission (MST). This technology facilitates communication between smartphones and magnetic card terminals by transmitting payment tokens through magnetic waves. Intriguingly, such magnetic waves inherently produce a distinct sound pattern (calledMST sound) containing payment information, which opens up new opportunities for both potential attackers and payment users. That is, MST sound can serve either as a new side channel for attackers to eavesdrop on MST transactions or as an easily accessible communication channel that enhances the payment experience for users. Inspired by these possibilities, we aim to deeply explore the potential of MST sound across these two dimensions, presenting two frameworks with different objectives: MagSnoop and M2APay. The first is the inference framework, which accurately, robustly, and efficiently infers payment tokens by listening to MST sounds. The second is the payment framework, which helps users establish a secure communication channel between MST-supported smartphones and microphone-equipped smartphones by shielding the vulnerability inherent in MST sound. Our experiments with prototypes of these frameworks achieved high accuracy in token inference and data transmission. Furthermore, both MagSnoop and M2APay are capable of accurately decoding tokens in diverse payment environments, including noisy environments and real-world scenarios.

As various computing devices, such as smartphones, IoT devices, smart speakers etc, becomes omnipresent in our daily lives, interest in ubiquitous computing interfaces is increasing. In response to this, various studies have introduced on-surface input techniques that leverage the surface of surrounding objects as touch interfaces. However, most of them struggle to support ubiquitous interaction due to their dependency on specific hardware or environments. In this work, we propose${{\sf S \text{-}UbiTap}}$, an input method that turns any flat solid surface into a touch input space by listening to sound (i.e., with microphones already present in the commodity devices). More specifically, we develop a novel touch localization technique that leverages the physical phenomenon, calleddispersion, which is the characteristic of sound as it travels through solid surfaces, and address the challenges that limit existing acoustic-based solutions in terms of portability, accuracy, usability, robustness, scalability, and responsiveness. Our extensive experiments with a prototype of${{\sf S \text{-}UbiTap}}$show that we can support sub-centimeter accuracy on various types of surfaces with minor user calibration effort. In addition, the accuracy is maintained even when the size of the touch input space increases. In our experience with real-world users,${{\sf S \text{-}UbiTap}}$significantly improves usability and robustness, thus enabling the emergence of more exciting applications.

Mobile apps offer a variety of features that greatly enhance user experience. However, users still often find it difficult to use mobile apps in the way they want. For example, it is not easy to use multiple apps simultaneously on a small screen of a smartphone. In this paper, we present A-Mash, a mobile platform that aims to simplify the way of interacting with multiple apps concurrently to the level of using a single app only. A key feature of A-Mash is that users can mash up the UIs of different existing mobile apps on a single screen according to their preferences. To this end, A-Mash 1) extracts UIs from unmodified existing apps (dynamic UI extraction) and 2) embeds extracted UIs from different apps into a single wrapper app (cross-process UI embedding), while 3) making all these processes hidden from the users (transparent execution environment). To the best of our knowledge, A-Mash is the first work to enable UIs of different unmodified legacy apps to seamlessly integrate and synchronize on a single screen, providing an illusion as if they were developed as a single app. A-Mash offers great potential for a number of useful usage scenarios. For instance, a user can mashup UIs of different IoT administration apps to create an all-in-one IoT device controller or one can mashup today's headlines from different news and magazine apps to craft one's own news headline collection. In addition, A-Mash can be extended to an AR space, in which users can map UI elements of different mobile apps to physical objects inside their AR scenes. Our evaluation of the A-Mash prototype implemented in Android OS demonstrates that A-Mash successfully supports the mashup of various existing mobile apps with little or no performance bottleneck. We also conducted in-depth user studies to assess the effectiveness of the A-Mash in real-world use cases.

Samsung Pay, one of the most representative mobile payment services, allows mobile users to make payment transactions almost anywhere using only their smartphone. This is thanks to MST (Magnetic Secure Transmission) that supports communication between smartphones and payment terminals for magnetic cards by transferring payment tokens via magnetic waves. Several attack methods have targeted this new technology by eavesdropping on magnetic fields to intercept the tokens, but with the use of dedicated hardware. This paper raises new security concerns for mobile payment users in a different, yet more effective way; by introducing MagSnoop, a novel framework that infers payment tokens from listening to MST sounds generated during the activation of MST payment transactions. More specifically, we first explore the principle, causing the generation of MST sounds, and the fundamental characteristics of these sounds. We then use these observations to infer payment tokens with a high degree of accuracy, robustness, applicability, and data efficiency. Our experiments with a prototype of MagSnoop demonstrate that it can support high accuracy in token inference (more than 77.8%). In addition, MagSnoop can maintain a reasonable level of accuracy regardless of the payment environments (e.g., 69.2% with a noise level of 50 dBA) and even in the real world (an inference success rate of 68.0% with 15 real-world users).

In this paper, we present TapAuth, a novel smartphone user authentication system that leverages vibrations generated from a user's tap inputs. TapAuth is based on the observation that the vibrations have their unique characteristic depending on the user due to differences in finger structure. Specifically, TapAuth collects audio and motion data using only built-in sensors, while users are entering PIN code. Then, it authenticates the user by verifying the validity of the code and comparing the collected data with a pre-built training dataset. Our experimental results with 20 real-world users show that TapAuth can achieve high accuracy.

A recent trend in electric fans is the support of smarter features based on Internet connectivity, such as remote control. This work opens new opportunities to make these devices more intelligent, particularly by proposing WindTrack, a novel angular positioning technique for electric fans. This would encourage the emergence of a new class of smart fans that use spatial information for automatic wind direction control. Many works have extensively studied angular positioning, but had limitations realizing such smart fans in terms of deployability (due to hardware requirements), accuracy, and usability. WindTrack overcomes these limitations by leveraging the acoustic phenomenon of refraction caused by wind. More specifically, it transmits and receives sound signals by using a smartphone located close to a user, while operating a fan in an oscillating mode. Therefore, WindTrack only requires Internet-connected fans for collaboration with smartphones. It then identifies the user's angular position based on the relationship between the propagation characteristics of the received signals and the oscillating fan. We further improve the robustness of WindTrack by using multiple speakers and microphones built-in smartphones and listening to wind sounds captured when wind passes across microphones. Extensive experiments on a WindTrack prototype demonstrate that it can achieve an angular positioning accuracy of a few degrees, especially in various environments and with no modification to commercial smartphones and Internet-connected fans.

With the omnipresence of computing devices in our daily lives, interests in ubiquitous computing interfaces have grown. In response to this, various studies have introduced on-surface input techniques which use the surfaces of surrounding objects as a touch interface. However, these methods are yet struggling to support ubiquitous interaction due to their dependency on specific hardware or environments. In this paper, we propose UbiTap, an input method that turns solid surfaces into a touch input space, through the use of sound (i.e., with microphones already present in the commodity devices). More specifically, we develop a novel touch localization technique which leverages the physical phenomenon, referred to as dispersion, a characteristic of sound as it travels through solid surfaces, so as to address challenges which limit existing acoustic-based solutions in terms of portability, accuracy, usability, robustness, and responsiveness. Our extensive experiments with a prototype of UbiTap show that we can support sub-centimeter accuracy on various surfaces with minor user calibration effort. In our experience with real-world users, UbiTap significantly improves usability and robustness, thus enabling the emergence of more exciting applications.

Web browsing is a key application on mobile devices. However, mobile browsers are largely optimized for performance, imposing a significant burden on power-hungry mobile devices. In this work, we aim to reduce the energy consumed to load web pages on smartphones, preferably without increasing page load time and compromising user experience. To this end, we first study the internals of web page loading on smartphones and identify its energy-inefficient behaviors. Based on our findings, we then derive general design principles for energy-efficient web page loading, and apply these principles to the open-source Chromium browser and implement our techniques on commercial smartphones. Experimental results show that our techniques are able to achieve a 24.4% average system energy saving for Chromium on a latest-generation big.LITTLE smartphone using WiFi (a 22.5% saving when using 3G), while not increasing average page load time. We also show that our proposed techniques can bring a 10.5% system energy saving on average with a small 1.69\% increase in page load time for mobile Firefox web browser. User study results indicate that such a small increase in page load time is hardly perceivable.

A variety of advantages from sounds such as measurement and accessibility introduces a new opportunity for mobile applications to offer broad types of interesting, valuable functionalities, supporting a richer user experience. However, in spite of the growing interests on mobile sound applications, few or no works have been done in focusing on managing an audio device effectively. More specifically, their low level of real-time capability for audio resources makes it challenging to satisfy tight timing requirements of mobile sound applications, e.g., a high sensing rate of acoustic sensing applications. To address this problem, this work presents the SounDroid framework, an audio device management framework for real-time audio requests from mobile sound applications. The design of SounDroid is based on the requirement analysis of audio requests as well as an understanding of the audio playback procedure including the audio request scheduling and dispatching on Android. It then incorporates both real-time audio request scheduling algorithms, called EDF-V and AFDS, and dispatching optimization techniques into mobile platforms, and thus improves the quality-of-service of mobile sound applications. Our experimental results with the prototype implementation of SounDroid demonstrate that it is able to enhance scheduling performance for audio requests, compared to traditional mechanisms (by up to 40% of improvement), while allowing deterministic dispatching latency.

The goal of this work is to provide an abstraction of ideal sound environments to a new emerging class of Mobile Multi-speaker Audio (MMA) applications. Typically, it is challenging for MMA applications to implement advanced sound features (e.g., surround sound) accurately in mobile environments, especially due to unknown, irregular loudspeaker configurations. Towards an illusion that MMA applications run over specific loudspeaker configurations (i.e., speaker type, layout), this work proposes AMAC, a new Adaptive Mobile Audio Coordination system that senses the acoustic characteristics of mobile environments and controls individual loud-speakers adaptively and accurately. The prototype of AMAC implemented on commodity smartphones shows that it provides the coordination accuracy in sound arrival time in several tens of microseconds and reduces the variance in sound level substantially.

The last several years have seen a rapid increase in smart phone use. Android offers an open-source software platform on smart phones, that includes a Linux-based kernel, Java applications, and middleware. The Android middleware provides system libraries and services to facilitate the development of performance-sensitive or device-specific functionalities, such as screen display, multimedia, and web browsing. Android keeps track of which applications make use of which system services for some pre-defined functionalities, and which application is running in the foreground attracting the user's attention. Such information is valuable in capturing application characteristics and can be useful for resource management tailored to application requirements. However, the Linux-based Android kernel does not utilize such information for I/O resource management. This paper is the first work, to the best of our knowledge, to attempt to understand application characteristics through Android architecture and to incorporate those characteristics into disk and network I/O management. Our proposed approach, Aciom (Application Characteristics-aware I/O Management), requires no modification to applications and characterizes application I/O requests as time-sensitive, bursty, or plain, depending on which system services are involved and which application receives the user's focus. Aciom then provides differentiated I/O management services for different types of I/O requests, supporting minimum bandwidth reservations for time-sensitive requests and placing maximum bandwidth limits on bursty requests. We present the design of Aciom and a prototype implementation on Android. Our experimental results show that Aciom is quite effective in handling disk and network I/O requests in support of time-sensitive applications in the presence of bursty I/O requests.