Modern spoken language understanding (SLU) approaches optimize the system in an end-to-end (E2E) manner. This approach offers two key advantages. Firstly, it helps mitigate error propagation from upstream systems. Secondly, combining various information types and optimizing them towards the same objective is straightforward. In this study, we attempt to build an SLU system by integrating information from two modalities, i.e., speech and text, and concurrently optimizing the associated tasks. We leverage a pre-trained model built with speech and text data and fine-tune it for the E2E SLU tasks. The SLU model is jointly optimized with automatic speech recognition (ASR) and SLU tasks under single-mode and dual-mode schemes. In the single-mode model, ASR and SLU results are predicted sequentially, whereas the dualmode model predicts either ASR or SLU outputs based on the task tag. Our proposed method demonstrates its superiority through benchmarking against FSC, SLURP, and in-house datasets, exhibiting improved intent accuracy, SLU-F1, and Word Error Rate (WER).

This paper proposes Mimic, an automated UI compatibility testing system for Android apps. Mimic is designed specifically for comparing the UI behavior of an app across different devices, different Android versions, and different app versions. This design choice stems from a common problem that Android developers and researchers face-how to test whether or not an app behaves consistently across different environments or internal changes. Mimic allows Android app developers to easily perform backward and forward compatibility testing for their apps. It also enables a clear comparison between a stable version of app and a newer version of app. In doing so, Mimic allows multiple testing strategies to be used, such as randomized or sequential testing. Finally, Mimic programming model allows such tests to be scripted with much less developer effort than other comparable systems. Additionally, Mimic allows parallel testing with multiple testing devices and thereby speeds up testing time. To demonstrate these capabilities, we perform extensive tests for each of the scenarios described above. Our results show that Mimic is effective in detecting forward and backward compatibility issues, and verify runtime behavior of apps. Our evaluation also shows that Mimic significantly reduces the development burden for developers.

Gesto is a system that enables task automation for Android apps using gestures and voice commands. Using Gesto, a user can record a UI action sequence for an app, choose a gesture or a voice command to activate the UI action sequence, and later trigger the UI action sequence by the corresponding gesture/voice command. Gesto enables this for existing Android apps without requiring their source code or any help from their developers. In order to make such capability possible, Gesto combines bytecode instrumentation and UI action record-and-replay. To show the applicability of Gesto, we develop four use cases using real apps downloaded from Google Play-Bing, Yelp, AVG Cleaner, and Spotify. For each of these apps, we map a gesture or a voice command to a sequence of UI actions. According to our measurement, Gesto incurs modest overhead for these apps in terms of memory usage, energy usage, and code size increase. We evaluate our instrumentation capability and overhead using 1,000 popular apps downloaded from Google Play. Our result shows that Gesto is able to instrument 94.9% of the apps without any significant overhead. In addition, since our prototype currently supports 6 main UI elements of Android, we evaluate our coverage and measure what percentage of UI element uses we can cover. Our result shows that our 6 UI elements can cover 96.4% of all statically-declared UI element uses in the 1,000 Google Play apps.

Mobile systems face privacy challenges including coarsegrained privacy control and the inability to distinguish private and public files. We propose System-E, a novel system which can enhance the user privacy on mobile systems (e.g., Android) by (1) enabling users to set finer grained permissions for apps accessing data, and (2) enabling automatic classification of data (e.g., photos) at the storage layer (e.g., by using deep learning) to prevent potentially sensitive data from being stored/accessed with open permissions.

In this paper, we design a pluggable data management solution for modern mobile platforms (e.g., Android). Our goal is to allow data management mechanisms and policies to be implemented independently of core app logic. Our design allows a user to install data management solutions as apps, install multiple such solutions on a single device, and choose a suitable solution each for one or more apps. It allows app developers to focus their effort on app logic and helps the developers of data management solutions to achieve wider deployability. It also gives increased control of data management to end users and allows them to use different solutions for different apps. We present a prototype implementation of our design called BlueMountain, and implement several data management solutions for file and database management to demonstrate the utility and ease of using our design. We perform detailed microbenchmarks as well as end-to-end measurements for files and databases to demonstrate the performance overhead incurred by our implementation.

BlueMountain is a system that enables building pluggable data management solutions which can be linked with any Android app at runtime, without requiring any modifications to the Android platform. BlueMountain simplifies the app development, provides flexibility to end users, and works with existing apps.

This paper proposes a new technique that enables open innovation in mobile platforms. Our technique allows third-party developers to modify, instrument, or extend platform API calls and deploy their modifications seamlessly. The uniqueness of our technique is that it enables modifications completely at the app layer without requiring any platform-level changes. This allows practical openness---third parties can easily distribute their modifications for a platform without the need to update the entire platform. To demonstrate the benefits of our technique, we have developed a prototype on Android called Reptor and used it to instrument real-world apps with novel functionality. Our evaluation in realistic scenarios shows that Reptor has little overhead in performance and energy, and only modest overhead in memory usage that ranges from 0.6% to 10% for the observed worst cases.

We demonstrate AutoClicker, a fully automated UI testing system for large-scale Android apps using multiple devices. It provides a way to quickly and easily verify that a large number of Android apps behave correctly at runtime in a repeatable manner.

We demonstrate Reptor, a bytecode instrumentation tool enabling API virtualization on Android. It provides a general way to alter functionality of platform APIs on Android. With Reptor, third-party developers can modify the behavior of platform APIs according to their needs. All modifications are completely at the app layer without modifying the underlying platform. This allows practical openness---third-party developers can easily distribute their modifications for a platform without the need to update the entire platform.

We demonstrate Gesto, a dynamic gesture mapping tool. It provides users to map any gesture to a certain UI event that the users need. Also, the mapping can be easily changed by users.

Searching for parking spots generates frustration and pollution. To address these parking problems, we present PocketParker, a crowdsourcing system using smartphones to predict parking lot availability. PocketParker is an example of a subset of crowdsourcing we call pocketsourcing. Pocketsourcing applications require no explicit user input or additional infrastructure, running effectively without the phone leaving the user's pocket. PocketParker detects arrivals and departures by leveraging existing activity recognition algorithms. Detected events are used to maintain per-lot availability models and respond to queries. By estimating the number of drivers not using PocketParker, a small fraction of drivers can generate accurate predictions. Our evaluation shows that PocketParker quickly and correctly detects parking events and is robust to the presence of hidden drivers. Camera monitoring of several parking lots as 105 PocketParker users generated 10;827 events over 45 days shows that PocketParker was able to correctly predict lot availability 94% of the time.

Today's mobile applications operate in a diverse set of environments, where it is difficult for a developer to know beforehand what conditions his or her application will be put under. For example, once deployed on an online application store, an application can be downloaded on different types of hardware, ranging from budget smartphones to high-end tablets. In addition, network conditions can vary widely from Wi-Fi to 3G to 4G. Mobile applications also need to co-exist with other applications that compete for resources at different times.