The proliferation of mobile and IoT devices has sparked growing interest in multi-device interaction, where a single app operates across multiple devices to leverage their diverse capabilities. However, the current methods of app development and usage remain bound to a single-device paradigm, rendering multi-device apps difficult to implement and deploy. This paper presents COMET , a novel mobile app framework that enables the dynamic distribution of app components across multiple devices at runtime, supporting a wide range of multi-device scenarios, including collaborative applications, smart environments, and interactions across heterogeneous personal devices. COMET enables developers to specify distributable components using lightweight annotations and employs build-time code instrumentation to automate the deployment and execution of selected components onto remote devices. This approach supports multi-device interaction with minimal developer effort and without requiring system-level modifications. To realize this, COMET addresses four key challenges: (i) the static partitioning of app components and extraction of their dependencies at build time, (ii) efficient execution of these components on remote devices, (iii) preservation of intercomponent communication across devices, and (iv) synchronization of distributed components that share a global state. We implemented a prototype of COMET on Android and evaluated it using real-world apps. Our evaluation with eight case-study apps shows that component distribution completes within 251.8 ms. A user study with 15 participants further demonstrates that COMET provides intuitive multi-device interaction with acceptable responsiveness.

Back-of-Device (BoD) interfaces have emerged as a promising solution to free up screen real estate in smartphones by offloading

On-device machine learning applications are increasingly deployed in dynamic and open system environments, where resource availability fluctuates unpredictably. This variability, coupled with limited computing resources, poses significant challenges in achieving high responsiveness. Existing on-device machine learning frameworks typically rely on static and coarse-grained resource allocation, leading to performance degradation under resource contention. To address this, we propose FlexOn, a novel framework that combines fine-grained model segmentation and dynamic resource selection to rapidly adapt to highly dynamic runtime conditions and effectively mitigate unpredictable resource contention. A prototype built on LiteRT demonstrates significant improvements in both average and tail latencies of up to 54% and 58%, respectively, across three different embedded platforms under dynamically varying resource availability. To the best of our knowledge, this is the first work that addresses the resource contention in open embedded systems for better machine learning inference responsiveness.

Dynamic Voltage and Frequency Scaling (DVFS) is a key technology for enhancing power efficiency in computing devices. However, conventional DVFS methods struggle with the unique demands of mobile devices. Recent reinforcement learning (RL)-based approaches address this by tailoring to mobile-specific thermal and workload characteristics. Yet, these solutions make frequency adjustments that ignore device-specific configurations calibrated by vendors, neglect the impact of ambient and non-processor components—such as battery, display, and integrated circuits—that significantly affect processor thermal management, and rely on fixed environment-dependent parameters, limiting adaptability across different environments. To address these limitations, we propose EarDVFS, an environment-adaptable RL-based solution that employs proactive throttling to combine the strengths of traditional and RL-based methods, considers the temperatures of ambient and non-processor components for better thermal management, and features an environment-robust RL parameter design. Extensive experiments across varying ambient temperatures, devices, and workloads demonstrate that EarDVFS consistently enhances power efficiency by an average of 21.6% and up to 49.6% compared to default DVFS while maintaining performance. Furthermore, we conduct comprehensive ablation studies on the action, state, and reward elements of our RL model, confirming that each element significantly contributes to EarDVFS’s adaptability and effectiveness across diverse thermal environments.

Large Foundation Models (LFMs) have unlocked new possibilities in human-computer interaction, particularly with the rise of mobile Graphical User Interface (GUI) Agents capable of interacting with mobile GUIs. These agents allow users to automate complex mobile tasks through simple natural language instructions. However, the inherent probabilistic nature of LFMs, coupled with the ambiguity and context-dependence of mobile tasks, makes LFM-based automation unreliable and prone to errors. To address this critical challenge, we introduce VeriSafe Agent (VSA)1: a formal verification system that serves as a logically grounded safeguard for Mobile GUI Agents. VSA deterministically ensures that an agent's actions strictly align with user intent before executing the action. At its core, VSA introduces a novel autoformalization technique that translates natural language user instructions into a formally verifiable specification. This enables runtime, rule-based verification of agent's actions, detecting erroneous actions even before they take effect. To the best of our knowledge, VSA is the first attempt to bring the rigor of formal verification to GUI agents, bridging the gap between LFM-driven actions and formal software verification. We implement VSA using off-the-shelf LFM services (GPT-4o) and evaluate its performance on 300 user instructions across 18 widely used mobile apps. The results demonstrate that VSA achieves 94.33%–98.33% accuracy in verifying agent actions, outperforming existing LFM-based verification methods by 30.00%–16.33%, and increases the GUI agent's task completion rate by 90%–130%.

The Robot Operating System 2 (ROS 2) is a popular middleware for distributed robotic applications. However, achieving real-time guarantees in ROS 2 is challenging due to unpredictable delays and priority inversions. We reveal that these issues arise from the lack of consistent priority propagation across ROS 2's multi-layered communication architecture, particularly down to the kernel layer. To address this, we present CROS-RT, the first cross-layer scheduler explicitly designed to tackle the unpredictability in ROS 2 inter-process communication caused by multi-layer priority misalignment. CROS-RT ensures consistent, priority-based scheduling across the application, middleware, and kernel layers, introducing mechanisms for priority propagation, kernel-level message prioritization, and dynamic kernel thread adjustment. We have implemented and evaluated CROS-RT on the current stable release of ROS 2. Experiments demonstrate that CROS-RT enhances communication predictability, reducing the worst-case response time by up to 89.3 % over a baseline (vanilla ROS 2). Additionally, we provide an analytical model to derive upper bounds on response times, ensuring reliable realtime performance for safety-critical applications.

PIN authentication is the first line of defense for protecting private data on many smartphone applications, such as lock screens, messengers, and banking apps. However, existing PIN authentication systems have several constraints regarding security, usability, and robustness. To go beyond their limitations, this paper presents TaPIN, a reliable system that authenticates smartphone users with the collaborative use of PINs and tap biometrics. A user is first instructed to enter her PIN by tapping a smartphone screen for authentication. During the PIN entry, the user's fingertip collides with the screen, producing user-specific vibration and sound signals. TaPIN then senses the tap-induced signals and the collision properties, e.g., pressures and sizes, using the smartphone's built-in sensors and leverages them as biometric features. That is, it authenticates the user by verifying not only the entered PIN but also the collected features. Our experiments with 20 real-world users demonstrate that this two-factor authentication system is easy to use, more secure than existing methods, and deployable without dedicated hardware. For example, it accurately authenticates users with an average EER of 1.9% in stationary environments and maintains a reasonable level of security regardless of devices, tap styles, and noise.

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT1, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app---explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 185 tasks across 18 mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.7% accuracy, and is able to adapt them to different contexts with near perfect (98.75%) accuracy while reducing both latency and cost by 62.5% and 68.8%, respectively, compared to the GPT-4 powered baseline.

The increasing complexity and memory demands of Deep Neural Networks (DNNs) for real-time systems pose new significant challenges, one of which is the GPU memory capacity bottleneck, where the limited physical memory inside GPUs impedes the deployment of sophisticated DNN models. This paper presents, to the best of our knowledge, the first study of addressing the GPU memory bottleneck issues, while simultaneously ensuring the timely inference of multiple DNN tasks. We propose RT-Swap, a real-time memory management framework, that enables transparent and efficient swap scheduling of memory objects, employing the relatively larger CPU memory to extend the available GPU memory capacity, without compromising timing guarantees. We have implemented RT-Swap on top of representative machine-learning frameworks, demonstrating its effectiveness in making significantly more DNN task sets schedulable at least 72% over existing approaches even when the task sets demand up to 96.2% more memory than the GPU's physical capacity.

Edge TPU (Tensor Processing Unit) is being widely utilized in various edge computing applications as a high-efficiency, low-power accelerator for deep learning computations. However, temperature rise in Edge TPU can lead to performance degradation, reduced stability, and shortened lifespan, necessitating temperature management through thermal modeling. This paper proposes a task-level thermal modeling technique for predicting Edge TPU temperature. The proposed method estimates power consumption of CPU and Edge TPU based on workloads of various deep learning tasks and predicts the convergence temperature of Edge TPU using a steady temperature model. Through experiments, we confirmed that the proposed method accurately predicts Edge TPU temperature for various workloads. The average prediction error was 0.7°C. This study is expected to serve as a foundation for developing temperature management techniques by presenting an effective temperature prediction model that considers the thermal characteristics of Edge TPU.

Samsung Pay, a widely-used mobile payment service, enables users to pay using just their smartphone thanks to Magnetic Secure Transmission (MST). This technology facilitates communication between smartphones and magnetic card terminals by transmitting payment tokens through magnetic waves. Intriguingly, such magnetic waves inherently produce a distinct sound pattern (calledMST sound) containing payment information, which opens up new opportunities for both potential attackers and payment users. That is, MST sound can serve either as a new side channel for attackers to eavesdrop on MST transactions or as an easily accessible communication channel that enhances the payment experience for users. Inspired by these possibilities, we aim to deeply explore the potential of MST sound across these two dimensions, presenting two frameworks with different objectives: MagSnoop and M2APay. The first is the inference framework, which accurately, robustly, and efficiently infers payment tokens by listening to MST sounds. The second is the payment framework, which helps users establish a secure communication channel between MST-supported smartphones and microphone-equipped smartphones by shielding the vulnerability inherent in MST sound. Our experiments with prototypes of these frameworks achieved high accuracy in token inference and data transmission. Furthermore, both MagSnoop and M2APay are capable of accurately decoding tokens in diverse payment environments, including noisy environments and real-world scenarios.

The growing trend of multi-device ownerships creates opportunities to use applications across devices. However, the current methods of app development/usage remain in the single-device paradigm, which is far below user expectations. For example, it is currently impossible for users to dynamically partition an existing app across different devices to utilize multiple surfaces. We introduce FLUID, a novel multi-device platform that supports simultaneous operation of multiple devices. FLUID aims toi)distribute the user interfaces (UIs) of a single app across multiple devices,ii)support unmodified legacy apps without extra engineering, andiii)support numerous apps with customized UIs. Previous approaches, like screen mirroring and app migration, do not satisfy those goals altogether. However, FLUID is designed to satisfy the goals. It can efficiently deploy UI objects to different devices by identifying only UI states necessary for accurate rendering. And FLUID can execute the distributed UI objects by supporting cross-device method invocations transparently and synchronizing the replicated UIs across devices. Furthermore, FLUID automatically handles unexpected events that may degrade its usability by efficiently maintaining the distributed UIs up to date. Our evaluation using 20 legacy apps shows that FLUID can transparently support numerous apps and is fast enough for interactive use.

Deep neural networks (DNNs) have been deployed in many safety-critical real-time embedded systems. To support DNN tasks in real-time, most previous studies focused on GPU or CPU. However, Edge TPU has not yet been studied for real-time guarantees. This paper presents a real-time DNNs framework for Edge TPU to satisfy multiple DNN inference tasks’ timing requirements. The proposed framework provides 1) SRAM allocation and model partitioning techniques and 2) a MIP-based algorithm that determines the amount of SRAM and the number of segments for each task. The experiment result shows that our framework provides 79% higher schedulability than the existing Edge TPU system.

Mobile apps offer a variety of features that greatly enhance user experience. However, users still often find it difficult to use mobile apps in the way they want. For example, it is not easy to use multiple apps simultaneously on a small screen of a smartphone. In this paper, we present A-Mash, a mobile platform that aims to simplify the way of interacting with multiple apps concurrently to the level of using a single app only. A key feature of A-Mash is that users can mash up the UIs of different existing mobile apps on a single screen according to their preferences. To this end, A-Mash 1) extracts UIs from unmodified existing apps (dynamic UI extraction) and 2) embeds extracted UIs from different apps into a single wrapper app (cross-process UI embedding), while 3) making all these processes hidden from the users (transparent execution environment). To the best of our knowledge, A-Mash is the first work to enable UIs of different unmodified legacy apps to seamlessly integrate and synchronize on a single screen, providing an illusion as if they were developed as a single app. A-Mash offers great potential for a number of useful usage scenarios. For instance, a user can mashup UIs of different IoT administration apps to create an all-in-one IoT device controller or one can mashup today's headlines from different news and magazine apps to craft one's own news headline collection. In addition, A-Mash can be extended to an AR space, in which users can map UI elements of different mobile apps to physical objects inside their AR scenes. Our evaluation of the A-Mash prototype implemented in Android OS demonstrates that A-Mash successfully supports the mashup of various existing mobile apps with little or no performance bottleneck. We also conducted in-depth user studies to assess the effectiveness of the A-Mash in real-world use cases.

Samsung Pay, one of the most representative mobile payment services, allows mobile users to make payment transactions almost anywhere using only their smartphone. This is thanks to MST (Magnetic Secure Transmission) that supports communication between smartphones and payment terminals for magnetic cards by transferring payment tokens via magnetic waves. Several attack methods have targeted this new technology by eavesdropping on magnetic fields to intercept the tokens, but with the use of dedicated hardware. This paper raises new security concerns for mobile payment users in a different, yet more effective way; by introducing MagSnoop, a novel framework that infers payment tokens from listening to MST sounds generated during the activation of MST payment transactions. More specifically, we first explore the principle, causing the generation of MST sounds, and the fundamental characteristics of these sounds. We then use these observations to infer payment tokens with a high degree of accuracy, robustness, applicability, and data efficiency. Our experiments with a prototype of MagSnoop demonstrate that it can support high accuracy in token inference (more than 77.8%). In addition, MagSnoop can maintain a reasonable level of accuracy regardless of the payment environments (e.g., 69.2% with a noise level of 50 dBA) and even in the real world (an inference success rate of 68.0% with 15 real-world users).

Being able to use a single app across multiple devices can bring novel experiences to the users in various domains including entertainment and productivity. For instance, a user of a video editing app would be able to use a smart pad as a canvas and a smartphone as a remote toolbox so that the toolbox does not occlude the canvas during editing. However, existing approaches do not properly support the single-app multi-device execution due to several limitations, including high development cost, device heterogeneity, and high performance requirement. In this paper, we introduce FLUID-XP, a novel cross-platform multi-device system that enables UIs of a single app to be executed across heterogeneous platforms, while overcoming the limitations of previous approaches. FLUID-XP provides flexible, efficient, and seamless interactions by addressing three main challenges: i) how to transparently enable a single-display app to use multiple displays, ii) how to distribute UIs across heterogeneous devices with minimal network traffic, and iii) how to optimize the UI distribution process when multiple UIs have different distribution requirements. Our experiments with a working prototype of FLUID-XP on Android confirm that FLUID-XP successfully supports a variety of unmodified real-world apps across heterogeneous platforms (Android, iOS, and Linux). We also conduct a lab study with 25 participants to demonstrate the effectiveness of FLUID-XP with real users.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. For example, it is currently not possible for a user to dynamically partition an existing live streaming app with chatting capabilities across different devices, such that she watches her favorite broadcast on her smart TV while real-time chatting on her smartphone. In this paper, we present FLUID, a new Android-based multi-device platform that enables innovative ways of using multiple devices. FLUID aims to i) allow users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices (high flexibility), ii) require no additional development effort to support unmodified, legacy applications (ease of development), and iii) support a wide range of apps that follow the trend of using custom-made UIs (wide applicability). Previous approaches, such as screen mirroring, app migration, and customized apps utilizing multiple devices, do not satisfy those goals altogether. FLUID, on the other hand, meets the goals by carefully analyzing which UI states are necessary to correctly render UI objects, deploying only those states on different devices, supporting cross-device function calls transparently, and synchronizing the UI states of replicated UI objects across multiple devices. Our evaluation with 20 unmodified, real-world Android apps shows that FLUID can transparently support a wide range of apps and is fast enough for interactive use.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. We present FLUID, a new multi-device platform that allows users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices. In addition, FLUID aims to require no extra development effort to support a wide range of legacy apps that follow the trend of using custom-made UIs. To this end, FLUID analyzes which UI states are necessary to correctly render UI objects, deploys only those states on different devices, and supports cross-device function calls transparently. In this demo, we demonstrate several interesting use cases supported by our Android-based FLUID prototype.

In recent years, the explosion of diverse smart devices such as mobile phones, TVs, watches, and even cars, has completely changed our lives. We communicate with friends through social network services (SNSs) whenever we want, buy stuff without visiting shops, and enjoy multimedia wherever we are, thanks to these devices. However, these smart devices cannot simply interact with each other even though they are right next to each other. For example, when you want to read a PDF stored on a smartphone on a larger TV screen, you need to do complicated work or plug in a bunch of cables. In this paper, we introduce M+, an extension of Android that supports cross-device functionality sharing in a transparent manner. As a platform-level solution, M+ enables unmodified Android applications to utilize not only application functionalities but also system functionalities across devices, as if they were to utilize them inside the same device. In addition to secure connection setup, M+ also allows performing of permission checks for remote applications in the same way as for local. Our experimental results show that M+ enables transparent cross-device sharing for various functionalities and achieves performance close to that of within-device sharing unless a large amount of data is transferred.

Modern mobile devices are equipped with multiple network interfaces, including 3G/LTE and WiFi. Bandwidth aggregation over LTE and WiFi links offers an attractive opportunity of supporting bandwidth-intensive services, such as high-quality video streaming, on mobile devices. However, achieving effective bandwidth aggregation in mobile environments raises several challenges related to deployment, link heterogeneity, network fluctuation, and energy consumption. We present GreenBag, an energy-efficient bandwidth aggregation middleware that supports real-time data-streaming services over asymmetric wireless links, requiring no modifications to the existing Internet infrastructure and servers. GreenBag employs several techniques, including medium load balancing, efficient segment management, and energy-aware mode control, to resolve such challenges. We implement a prototype of GreenBag on Android-based mobile devices which hosts, to the best knowledge of the authors, the first LTE-enabled bandwidth aggregation prototype for energy-efficient real-time video streaming. Our experiment results in both emulated and real-world environments show that GreenBag not only achieves good bandwidth aggregation to provide QoS in bandwidth-scarce environments but also efficiently saves energy on mobile devices. Moreover, energy-aware GreenBag can minimize video interruption while consuming 14-25% less energy than the non-energy-aware counterpart in real-world experiments.