Large Foundation Models (LFMs) have unlocked new possibilities in human-computer interaction, particularly with the rise of mobile Graphical User Interface (GUI) Agents capable of interacting with mobile GUIs. These agents allow users to automate complex mobile tasks through simple natural language instructions. However, the inherent probabilistic nature of LFMs, coupled with the ambiguity and context-dependence of mobile tasks, makes LFM-based automation unreliable and prone to errors. To address this critical challenge, we introduce VeriSafe Agent (VSA)1: a formal verification system that serves as a logically grounded safeguard for Mobile GUI Agents. VSA deterministically ensures that an agent's actions strictly align with user intent before executing the action. At its core, VSA introduces a novel autoformalization technique that translates natural language user instructions into a formally verifiable specification. This enables runtime, rule-based verification of agent's actions, detecting erroneous actions even before they take effect. To the best of our knowledge, VSA is the first attempt to bring the rigor of formal verification to GUI agents, bridging the gap between LFM-driven actions and formal software verification. We implement VSA using off-the-shelf LFM services (GPT-4o) and evaluate its performance on 300 user instructions across 18 widely used mobile apps. The results demonstrate that VSA achieves 94.33%–98.33% accuracy in verifying agent actions, outperforming existing LFM-based verification methods by 30.00%–16.33%, and increases the GUI agent's task completion rate by 90%–130%.

SUMMARY: Accurate metagenomic taxonomic profiling is critical for understanding microbial communities. However, computational analysis often requires command-line proficiency and high-performance computing resources. To lower these barriers, we developed Metabuli App, an all-in-one desktop application that efficiently runs taxonomic profiling locally on a consumer-grade computer. It features user-friendly graphical interfaces for custom database curation, raw read quality control (QC), taxonomic profiling, and interactive result visualization. AVAILABILITY AND IMPLEMENTATION: GPLv3-licensed source code and prebuilt apps for Windows, macOS, and Linux are available at https://github.com/steineggerlab/Metabuli-App and are archived at https://doi.org/10.5281/zenodo.15876171. Analysis scripts are available at https://github.com/jaebeom-kim/metabuli-app-analysis. The Sankey-based taxonomy visualization component is available at https://github.com/steineggerlab/taxoview for easy integration into other web projects.

Even with advances in single-cell batteries, mobile users still experience low battery anxiety. By analyzing 19,855 hours of user behavior, we proposeMixMax, a heterogeneous battery system consisting of three complementary battery types tailored to minimizing low battery time. While the heterogeneous battery system offers an opportunity to simultaneously improve capacity and charging speed, one must face non-trivial challenges to design charge/discharge policies during runtime and determine the ratio of enclosed batteries. They are highly dependent on each other, which entails almost infinite candidates for the choice.MixMaxsimplifies this by reformulating the problem as an optimization problem, breaking it down into manageable sub-problems. However,MixMaxstill faces the challenge of catering to all users due to their diverse battery usage patterns. To address this, we introduce a customizedMixMaxthat groups users based on their usage patterns and provides tailored battery solutions. In evaluatingMixMax, we fabricate coin-cell batteries, develop a precise battery emulator using the fabricated batteries, and prototypeMixMaxon a real-world smartphone. Our evaluation shows thatMixMaxreduces low battery time by up to 24.6% without compromising capacity, volume, weight, or user behavior, and its customized version can further reduce it by up to 46.2%.

The rapid growth of the Internet of Things (IoT) in shared spaces has led to an increasing demand for sharing IoT devices among multiple users. Yet, existing IoT platforms often fall short by offering an all-or-nothing approach to access control, not only posing security risks but also inhibiting the growth of the shared IoT ecosystem. This paper introduces FLUID-IoT, a framework that enables flexible and granular multi-user access control, even down to the User Interface (UI) component level. Leveraging a multi-user UI distribution technique, FLUID-IoT transforms existing IoT apps into centralized hubs that selectively distribute UI components to users based on their permission levels. Our performance evaluation, encompassing coverage, latency, and memory consumption, affirm that FLUID-IoT can be seamlessly integrated with existing IoT platforms and offers adequate performance for daily IoT scenarios. An in-lab user study further supports that the framework is intuitive and user-friendly, requiring minimal training for efficient utilization.

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT1, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app---explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 185 tasks across 18 mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.7% accuracy, and is able to adapt them to different contexts with near perfect (98.75%) accuracy while reducing both latency and cost by 62.5% and 68.8%, respectively, compared to the GPT-4 powered baseline.

The normalization of RNA sequencing data is a primary step for downstream analysis. The most popular method used for the normalization is the trimmed mean of M values (TMM) and DESeq. The TMM tries to trim away extreme log fold changes of the data to normalize the raw read counts based on the remaining non-deferentially expressed genes. However, the major problem with the TMM is that the values of trimming factor M are heuristic. This paper tries to estimate the adaptive value of M in TMM based on Jaeckel's Estimator, and each sample acts as a reference to find the scale factor of each sample. The presented approach is validated on SEQC, MAQC2, MAQC3, PICKRELL and two simulated datasets with two-group and three-group conditions by varying the percentage of differential expression and the number of replicates. The performance of the present approach is compared with various state-of-the-art methods, and it is better in terms of area under the receiver operating characteristic curve and differential expression.

The growing trend of multi-device ownerships creates opportunities to use applications across devices. However, the current methods of app development/usage remain in the single-device paradigm, which is far below user expectations. For example, it is currently impossible for users to dynamically partition an existing app across different devices to utilize multiple surfaces. We introduce FLUID, a novel multi-device platform that supports simultaneous operation of multiple devices. FLUID aims toi)distribute the user interfaces (UIs) of a single app across multiple devices,ii)support unmodified legacy apps without extra engineering, andiii)support numerous apps with customized UIs. Previous approaches, like screen mirroring and app migration, do not satisfy those goals altogether. However, FLUID is designed to satisfy the goals. It can efficiently deploy UI objects to different devices by identifying only UI states necessary for accurate rendering. And FLUID can execute the distributed UI objects by supporting cross-device method invocations transparently and synchronizing the replicated UIs across devices. Furthermore, FLUID automatically handles unexpected events that may degrade its usability by efficiently maintaining the distributed UIs up to date. Our evaluation using 20 legacy apps shows that FLUID can transparently support numerous apps and is fast enough for interactive use.

Online meetings are indispensable in collaborative remote work environments, but they are vulnerable to distractions due to their distributed and location-agnostic nature. While distraction often leads to a decrease in online meeting quality due to loss of engagement and context, natural multitasking has positive tradeoff effects, such as increased productivity within a given time unit. In this study, we investigate the impact of real-time transcriptions (i.e., full-transcripts, summaries, and keywords) as a solution to help facilitate online meetings during distracting moments while still preserving multitasking behaviors. Through two rounds of controlled user studies, we qualitatively and quantitatively show that people can better catch up with the meeting flow and feel less interfered with when using real-time transcriptions. The benefits of real-time transcriptions were more pronounced after distracting activities. Furthermore, we reveal additional impacts of real-time transcriptions (e.g., supporting recalling contents) and suggest design implications for future online meeting platforms where these could be adaptively provided to users with different purposes.

Despite the physical advance of an existing single-cell battery system, mobile users are still suffering from low battery anxiety. With a careful analysis of users' battery usage behavior collected for 19,855 hours, we propose a heterogeneous battery system, MixMax, consisting of three complementary battery types tailored to minimizing the low battery time. While composing a heterogeneous battery system opens up a chance to simultaneously improve the capacity and the charging speed, one must face non-trivial challenges to determine the ratio of enclosed batteries and charge/discharge policies during the run-time. They are highly dependent on each other, which entails almost infinite candidates for the choice. MixMax gracefully unwinds the dependencies as it formulates the decision-making problem into an optimization problem and decomposes it into multiple sub-problems instead. To evaluate MixMax, we fabricate coin-cell batteries and experiment with them to model an accurate battery emulator which sophisticatedly reproduces the dynamics of battery systems. Our experimental results demonstrate that MixMax can reduce the low battery time by up to 24.6% without compromising capacity, volume, weight, and more importantly, users' battery usage behavior. In addition, we prototype MixMax on a smartphone, presenting the practicality of MixMax on mobile systems.

Mobile apps offer a variety of features that greatly enhance user experience. However, users still often find it difficult to use mobile apps in the way they want. For example, it is not easy to use multiple apps simultaneously on a small screen of a smartphone. In this paper, we present A-Mash, a mobile platform that aims to simplify the way of interacting with multiple apps concurrently to the level of using a single app only. A key feature of A-Mash is that users can mash up the UIs of different existing mobile apps on a single screen according to their preferences. To this end, A-Mash 1) extracts UIs from unmodified existing apps (dynamic UI extraction) and 2) embeds extracted UIs from different apps into a single wrapper app (cross-process UI embedding), while 3) making all these processes hidden from the users (transparent execution environment). To the best of our knowledge, A-Mash is the first work to enable UIs of different unmodified legacy apps to seamlessly integrate and synchronize on a single screen, providing an illusion as if they were developed as a single app. A-Mash offers great potential for a number of useful usage scenarios. For instance, a user can mashup UIs of different IoT administration apps to create an all-in-one IoT device controller or one can mashup today's headlines from different news and magazine apps to craft one's own news headline collection. In addition, A-Mash can be extended to an AR space, in which users can map UI elements of different mobile apps to physical objects inside their AR scenes. Our evaluation of the A-Mash prototype implemented in Android OS demonstrates that A-Mash successfully supports the mashup of various existing mobile apps with little or no performance bottleneck. We also conducted in-depth user studies to assess the effectiveness of the A-Mash in real-world use cases.

Being able to use a single app across multiple devices can bring novel experiences to the users in various domains including entertainment and productivity. For instance, a user of a video editing app would be able to use a smart pad as a canvas and a smartphone as a remote toolbox so that the toolbox does not occlude the canvas during editing. However, existing approaches do not properly support the single-app multi-device execution due to several limitations, including high development cost, device heterogeneity, and high performance requirement. In this paper, we introduce FLUID-XP, a novel cross-platform multi-device system that enables UIs of a single app to be executed across heterogeneous platforms, while overcoming the limitations of previous approaches. FLUID-XP provides flexible, efficient, and seamless interactions by addressing three main challenges: i) how to transparently enable a single-display app to use multiple displays, ii) how to distribute UIs across heterogeneous devices with minimal network traffic, and iii) how to optimize the UI distribution process when multiple UIs have different distribution requirements. Our experiments with a working prototype of FLUID-XP on Android confirm that FLUID-XP successfully supports a variety of unmodified real-world apps across heterogeneous platforms (Android, iOS, and Linux). We also conduct a lab study with 25 participants to demonstrate the effectiveness of FLUID-XP with real users.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. For example, it is currently not possible for a user to dynamically partition an existing live streaming app with chatting capabilities across different devices, such that she watches her favorite broadcast on her smart TV while real-time chatting on her smartphone. In this paper, we present FLUID, a new Android-based multi-device platform that enables innovative ways of using multiple devices. FLUID aims to i) allow users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices (high flexibility), ii) require no additional development effort to support unmodified, legacy applications (ease of development), and iii) support a wide range of apps that follow the trend of using custom-made UIs (wide applicability). Previous approaches, such as screen mirroring, app migration, and customized apps utilizing multiple devices, do not satisfy those goals altogether. FLUID, on the other hand, meets the goals by carefully analyzing which UI states are necessary to correctly render UI objects, deploying only those states on different devices, supporting cross-device function calls transparently, and synchronizing the UI states of replicated UI objects across multiple devices. Our evaluation with 20 unmodified, real-world Android apps shows that FLUID can transparently support a wide range of apps and is fast enough for interactive use.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. We present FLUID, a new multi-device platform that allows users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices. In addition, FLUID aims to require no extra development effort to support a wide range of legacy apps that follow the trend of using custom-made UIs. To this end, FLUID analyzes which UI states are necessary to correctly render UI objects, deploys only those states on different devices, and supports cross-device function calls transparently. In this demo, we demonstrate several interesting use cases supported by our Android-based FLUID prototype.

With the widespread use of smartphones, users tend to use their smartphone for a long period of time in unhealthy postures; bending forward the neck and watching the relatively small screen closely with concentration. If users keep such unhealthy postures for a long time, they are susceptible to musculoskeletal disorders and eye problems such as cervical disc and myopia, respectively. To prevent users from having these diseases, we propose a new methodology to monitor the posture of smartphone users with built-in sensors. The proposed mechanism estimates various values representing user postures like the tilt angle of the neck, viewing distance, and gaze condition of the user, by analyzing sensor data from a front-faced camera, 3-axis accelerometer, orientation sensor, or any combination thereof, and warns the user if estimated values are maintained within the abnormal range over the pre-defined time. As a proof of concept, we developed an Android application named Smart Pose which estimates the user's neck tilt angle from analysis of the facial image, shakiness and tilt angle of the smartphone, and then notifies the user when her/his neck tilt angle is maintained in an unusual range during the smartphone operation. Also, we validated the result of our system by the comparison with measurements from 3D posture imaging equipment in our research facility. Via the proposed mechanism, a participant was able to be aware of his unhealthy postures, and then try to correct them.

With the widespread use of smartphones, users tend to use their smartphone for a long period of time in unhealthy postures; bending forward the neck and watching the relatively small screen closely with concentration. If users keep such unhealthy postures for a long time, they are susceptible to musculoskeletal disorders and eye problems such as cervical disc and myopia, respectively. To prevent users from having these diseases, we propose a new methodology to monitor the posture of smartphone users with built-in sensors. The proposed mechanism estimates various values representing user postures like the tilt angle of the neck, viewing distance, and gaze condition of the user, by analyzing sensor data from a front-faced camera, 3-axis accelerometer, orientation sensor, or any combination thereof, and warns the user if estimated values are maintained within the abnormal range over the pre-defined time. As a proof of concept, we developed an Android application named Smart Pose which estimates the user's neck tilt angle from analysis of the facial image, shakiness and tilt angle of the smartphone, and then notifies the user when her/his neck tilt angle is maintained in an unusual range during the smartphone operation. Also, we validated the result of our system by the comparison with measurements from 3D posture imaging equipment in our research facility. Via the proposed mechanism, a participant was able to be aware of his unhealthy postures, and then try to correct them.

With the widespread use of smartphones, users tend to use their smartphone for a long period of time in unhealthy postures; bending forward the neck and watching the relatively small screen closely with concentration. If users keep such unhealthy postures for a long time, they are susceptible to musculoskeletal disorders and eye problems such as cervical disc and myopia, respectively. To prevent users from having these diseases, we propose a new methodology to monitor the posture of smartphone users with built-in sensors. The proposed mechanism estimates various values representing user postures like the tilt angle of the neck, viewing distance, and gaze condition of the user, by analyzing sensor data from a front-faced camera, 3-axis accelerometer, orientation sensor, or any combination thereof, and warns the user if estimated values are maintained within the abnormal range over the pre-defined time. As a proof of concept, we developed an Android application named Smart Pose which estimates the user's neck tilt angle from analysis of the facial image, shakiness and tilt angle of the smartphone, and then notifies the user when her/his neck tilt angle is maintained in an unusual range during the smartphone operation. Also, we validated the result of our system by the comparison with measurements from 3D posture imaging equipment in our research facility. Via the proposed mechanism, a participant was able to be aware of his unhealthy postures, and then try to correct them.

Awareness of the emotion of those who communicate with others is a fundamental challenge in building affective intelligent systems. Emotion is a complex state of the mind influenced by external events, physiological changes, or relationships with others. Because emotions can represent a user's internal context or intention, researchers suggested various methods to measure the user's emotions from analysis of physiological signals, facial expressions, or voice. However, existing methods have practical limitations to be used with consumer devices, such as smartphones; they may cause inconvenience to users and require special equipment such as a skin conductance sensor. Our approach is to recognize emotions of the user by inconspicuously collecting and analyzing user-generated data from different types of sensors on the smartphone. To achieve this, we adopted a machine learning approach to gather, analyze and classify device usage patterns, and developed a social network service client for Android smartphones which unobtrusively find various behavioral patterns and the current context of users. Also, we conducted a pilot study to gather real-world data which imply various behaviors and situations of a participant in her/his everyday life. From these data, we extracted 10 features and applied them to build a Bayesian Network classifier for emotion recognition. Experimental results show that our system can classify user emotions into 7 classes such as happiness, surprise, anger, disgust, sadness, fear, and neutral with a surprisingly high accuracy. The proposed system applied to a smartphone demonstrated the feasibility of an unobtrusive emotion recognition approach and a user scenario for emotion-oriented social communication between users.

With the widespread use of a smartphone, users tend to use their smartphone for a long period of time in unhealthy postures; bending forward the neck and watching the relatively small screen closely with concentration. If users keep such unhealthy postures for a long time, they are susceptible to musculoskeletal disorders and eye problems such as cervical disc and myopia, respectively. To prevent users from having these diseases, we propose a new methodology to monitor the posture of smartphone users with built-in sensors. The proposed mechanism estimates various values representing user postures like the tilt angle of the neck, viewing distance, and gaze condition of the user, by analyzing sensor data from a front-faced camera, 3-axis accelerometer, orientation sensor, or any combination thereof, and warns the user if estimated values are maintained within the abnormal range over the allowed time. Via the proposed mechanism, users are able to be aware of their unhealthy postures, and then try to correct their postures.

We present an ontology-based reasoning approach for saving energy in a smart home setting where a mobile phone can serve as a generic sensor which can collect the inhabitant's contextual data. The paper details an ontology that describes the smart home domain and a prototype to test the system. Finally, we conclude with lessons learned from our work in developing an energy-aware smart home prototype and suggestions for future work.