We study the patch selection problem for efficient transformer-based visual understanding wherein a sampler can be used to drop less informative patches at inference in order to speed up model execution on resource-constrained devices. As no labels are available on the saliency of the patches, existing works either try to solve an auxiliary task of locating distinctive patches or learn a policy network through the global image/video-level supervision. The former approach could drop redundant but important patches while the latter suffers from the weak supervision of a single class label per image/video. In this work, we observe that the attention weights in trained transformer-based models clearly highlight the salient regions in images and videos. Therefore, we propose a learned patch sampling framework called SAMPL that utilizes the attention weights as fine-grained patch-level supervision to learn a lightweight policy network for patch selection. To train SAMPL end-to-end with the transformer-based models, we introduce a new loss function based on the REINFORCE algorithm to match the distribution of patch selection probabilities and the attention scores. Experimental results on ImageNet, UCF101, Something-Something v2 and Kinetics-400 show that SAMPL can effectively increase the throughput by at least 1.5× while achieving competitive classification accuracy.

Advances in deep learning have enabled accurate language-based search and retrieval (e.g., over user photos) in the cloud. Many users prefer to store their photos in the home due to privacy concerns. As such, a need arises for models that can perform cross-modal search on resource-limited devices. State-of-the-art (SOTA) cross-modal retrieval models achieve high accuracy through learning entangled representations that enable fine-grained similarity calculation between a language query and an image, but at the expense of having a prohibitively high retrieval latency. Alternatively, there is a new class of methods that exhibits good performance with low latency but requires a lot more computational resources and an order of magnitude more training data (i.e., large web-scraped datasets consisting of millions of image–caption pairs), making them infeasible to use in a commercial context. From a pragmatic perspective, none of the existing methods are suitable for developing commercial applications for low-latency cross-modal retrieval on low-resource devices. We propose CrispSearch, a cascaded approach that greatly reduces the retrieval latency with minimal loss in ranking accuracy for on-device language-based image retrieval. The idea behind our approach is to combine a light-weight and runtime-efficient coarse model with a fine re-ranking stage. Given a language query, the coarse model effectively filters out many of the irrelevant image candidates. After this filtering, only a handful of strong candidates will be selected and sent to a fine model for re-ranking. Extensive experimental results with two SOTA models for the fine re-ranking stage on standard benchmark datasets show that CrispSearch results in a speedup of up to 38 times over the SOTA fine methods with negligible performance degradation. Moreover, our method does not require millions of training instances, making it a pragmatic solution to on-device search and retrieval.

Incorporating large image-text foundation models such as CLIP has substantially improved the performance of the multimodal video retrieval task. However, how to practically sample the frames from a video and aggregate the frame features into a video representation is still an open research question. In particular, real-world deployment scenarios, such as embodiment within consumer electronics or cloud-based inference pipelines, require two key facets of retrieval (representation building and search) to be computationally light and fast. In this paper, we propose AdaCLIP, a computation- and latency-aware system for pragmatic multimodal video retrieval. AdaCLIP consists of a learning-based frame selection module to select informative frames and a query-independent frame aggregation module to obtain strong video representations from the frame features. Specifically, in the frame selection module, we introduce a differentiable Hard-Top-k algorithm to sample a subset of the frames while optimizing the performance of the video retrieval task in an end-to-end manner. Moreover, to be latency-aware, we also propose a query-independent lightweight approach, MLP-Score, to aggregate the frame features into the video representation, which offers up to 142x speedup on GPU and 822x speedup on CPU in similarity search time compared to query-dependent matching methods. Experimental results on several popular video retrieval datasets confirm the effectiveness of AdaCLIP.

Advances in deep learning have enabled accurate language-based search and retrieval, e.g., over user photos, in the cloud. Many users prefer to store their photos in the home due to privacy concerns. As such, a need arises for models that can perform cross-modal search on resource-limited devices. State-of-the-art cross-modal retrieval models achieve high accuracy through learning entangled representations that enable fine-grained similarity calculation between a language query and an image, but at the expense of having a prohibitively high retrieval latency. Alternatively, there is a new class of methods that exhibits good performance with low latency, but requires a lot more computational resources, and an order of magnitude more training data (i.e. large web-scraped datasets consisting of millions of image-caption pairs) making them infeasible to use in a commercial context. From a pragmatic perspective, none of the existing methods are suitable for developing commercial applications for low-latency cross-modal retrieval on low-resource devices. We propose CrispSearch, a cascaded approach that greatly reduces the retrieval latency with minimal loss in ranking accuracy for on-device language-based image retrieval. The idea behind our approach is to combine a light-weight and runtime-efficient coarse model with a fine re-ranking stage. Given a language query, the coarse model effectively filters out many of the irrelevant image candidates. After this filtering, only a handful of strong candidates will be selected and sent to a fine model for re-ranking. Extensive experimental results with two SOTA models for the fine re-ranking stage, on two standard benchmark datasets (namely, MSCOCO and Flickr30k) show that CrispSearch results in a speedup of up to 38 times over the SOTA fine methods with negligible performance degradation. Moreover, our method does not require millions of training instances, making it a pragmatic solution to on-device search and retrieval.

Current video retrieval systems on mobile devices cannot process complex natural language queries, especially if they contain personalized concepts, such as proper names. To address these shortcomings, we propose an efficient and privacy-preserving video retrieval system that works well with personalized queries containing proper names, without re-training using personalized labelled data from users. Our system first computes an initial ranking of a video collection by using a generic attention-based video-text matching model (i.e., a model designed for non-personalized queries), and then uses a face detector to conduct personalized adjustments to these initial rankings. These adjustments are done by reasoning over the face information from the detector and the attention information provided by the generic model. We show that our system significantly outperforms existing keyword-based retrieval systems, and achieves comparable performance to the generic matching model fine-tuned on plenty of labelled data. Our results suggest that the proposed system can effectively capture both semantic context and personalized information in queries.

We present VASTA, a novel vision and language-assisted Programming By Demonstration (PBD) system for smartphone task automation. Development of a robust PBD automation system requires overcoming three key challenges: first, how to make a particular demonstration robust to positional and visual changes in the user interface (UI) elements; secondly, how to recognize changes in the automation parameters to make the demonstration as generalizable as possible; and thirdly, how to recognize from the user utterance what automation the user wishes to carry out. To address the first challenge, VASTA leverages state-of-the-art computer vision techniques, including object detection and optical character recognition, to accurately label interactions demonstrated by a user, without relying on the underlying UI structures. To address the second and third challenges, VASTA takes advantage of advanced natural language understanding algorithms for analyzing the user utterance to trigger the VASTA automation scripts, and to determine the automation parameters for generalization. We run an initial user study that demonstrates the effectiveness of VASTA at clustering user utterances, understanding changes in the automation parameters, detecting desired UI elements, and, most importantly, automating various tasks. A demo video of the system is available here: http://y2u.be/kr2xE-FixjI.

Micro services architecture has started a new trend for application development for a number of reasons: (1) to reduce complexity by using tiny services, (2) to scale, remove and deploy parts of the system easily, (3) to improve flexibility to use different frameworks and tools, (4) to increase the overall scalability, and (5) to improve the resilience of the system. Containers have empowered the usage of micro services architectures by being lightweight, providing fast start-up times, and having a low overhead. Containers can be used to develop applications based on monolithic architectures where the whole system runs inside a single container or inside a micro services architecture where one or few processes run inside the containers. Two models can be used to implement a micro services architecture using containers: master-slave, or nested-container. The goal of this work is to compare the performance of CPU and network running benchmarks in the two aforementioned models of micro services architecture hence provide a benchmark analysis guidance for system designers.

We demonstrate a prototype system called COLD that we are developing at IBM Research which provides optimized deployment of workload in the cloud. A workload refers to an application, consisting of virtual entities (e.g. VM, volume), to be deployed in a cloud infrastructure, consisting of physical entities (e.g. PM, storage). The resource requirements of the virtual entities, as well as metadata describing relations among virtual entities (e.g. location proximity requirement), are described using a declarative workload definition language, namely using a HOT template. COLD provides a clean separation between the underlying mechanisms offered by a given cloud environment for lifecycle management of virtual resources and policies that influence optimal placement. The current implementation of COLD is tied to Open Stack, though we have plans in the future to make it cloud agnostic. It is designed to be fault-tolerant. In a software defined environment of an Open Stack cloud with various hyper visors, we show the operation of COLD to place various Heat stacks that contain specific policies such as rack-level antic location.

Cell phones connected to high-speed 3G networks constitute an increasingly important class of clients on the Internet. From the viewpoint of the servers they connect to, such devices are virtually indistinguishable from conventional end-hosts. In this study, we examine the IP addresses seen by Internet servers for cell phone clients and make two observations. First, individual cell phones can expose different IP addresses to servers within time spans of a few minutes, rendering IP-based user identification and blocking inadequate. Second, cell phone IP addresses do not embed geographical information at reasonable fidelity, reducing the effectiveness of commercial geolocation tools used by websites for fraud detection, server selection and content customization. In addition to these two observations, we show that application-level latencies between cell phones and Internet servers can differ greatly depending on the location of the cell phone, but do not vary much at a given location over short time spans; as a result, they provide fine-grained location information that IPs do not.

Mobile devices are increasingly equipped with hardware and software services allowing them to determine their locations, but support for building location-aware applications remains rudimentary. This paper proposes tracks of location coordinates as a high-level abstraction for a new class of mobile applications including ride sharing, location-based collaboration, and health monitoring. Each track is a sequence of entries recording a person's time, location, and application-specific data. StarTrack provides applications with a comprehensive set of operations for recording, comparing, clustering and querying tracks. StarTrack can efficiently operate on thousands of tracks.

A promising architecture for remote healthcare monitoring involves the use of a pervasive device (such as a cellular phone), which aggregates data from multiple body-worn medical sensors and transmits the data to the backend. Unfortunately, the volume of data generated by increasingly sophisticated continuously- active sensors can overwhelm the resources on the mobile device. We propose imbuing the mobile device with the intelligence to perform context-aware filtering of sensor data streams in order to reduce transmissions in cases where the observed data corresponds to the norm expected by the system in a given context. To investigate the efficacy of this technique, we implemented the HARMONI middleware on a mobile device, and used it to collect real sensor data from users. Our experiments demonstrate that context-aware filtering can reduce the uplink bandwidth requirements of the system by up to 72%.

A particularly compelling vision of long-term remote health monitoring advocates the use of a personal pervasive device (such as a cellphone) as an intermediate relay, which transports data streams from multiple body-worn sensors to a backend analytics infrastructure. Unfortunately, a pure relay-based functionality on the cellphone is inadequate in the longer term, as increasingly sophisticated medical sensors impose unnacceptably high uplink traffic and energy consumption costs on the mobile device. To address this challenge, we are building an event-processing middleware, called HARMONI, which enables the pervasive device to perform context-aware processing and event filtering on the sensor data streams and locally extract higher-level features of interest, thereby reducing the volume of transmitted data. This paper presents the design and architectural components of HARMONI, with special emphasis on its implementation of context-aware event processing. This paper then demonstrates that the mobile device can extract localized context from the incoming sensor stream with sufficient accuracy to achieve satisfactory context-aware filtering. Our results also establish the need for personalizing such context extraction, as they show that similar sensor data patterns obtained from different individuals can imply significantly different activity contexts.

Most pages on the Web are designed for the desktop environment and render poorly on the small screens available on handheld devices. We introduce Reusable End-User Customization (REUC), a technique that lets end users adapt the layout of Web pages by removing, resizing and moving page elements. REUC records the user's customizations and automatically reapplies them on subsequent visits to the same page or to other, similar pages, on the same Web site. We present PageTailor, a REUC prototype based on the Minimo Web browser that runs on Windows Mobile PDAs. We show that users can utilize PageTailor to adapt sophisticated Web sites, such as Amazon, BBC and MSN, for browsing on a PDA. Moreover, the customizations remain effective for up to a year, even as the content of pages is updated, and can be reused across similar pages, limiting the customization effort required to browse a site.

Usage-awaRe Interactive Content Adaptation (URICA) is an automatic technique that adapts content for display on mobile devices based on usage semantics. URICA allows users who are unsatisfied with the system's adaptation decision to take control of the adaptation process and make changes until the content is suitably adapted for their purposes. The successful adaptation is recorded and used in making future adaptation decisions. To validate URICA, we implemented a prototype system called Chameleon that performs fidelity adaptation on web images. We conducted a user study in which participants used Chameleon to browse image-rich web pages on bandwidth-limited cellular links and used the collected traces to evaluate our system. We show that Chameleon reduces the latency for browsing web content by up to 65% and reduces bandwidth consumption by up to 80%. Chameleon also allows users to exchange bandwidth consumption for user interaction based on their personal preferences.

Automatic adaptation of content for mobile devices is a challenging problem because optimal adaptation often depends on the usage semantics of content, as well as the context of users (e.g., screen size of device being used, network connectivity, location, etc.). UsageawaRe Interactive Content Adaptation (URICA) is an automatic technique that adapts content for mobile devices based on usage semantics. URICA allows a user who is unsatisfied with the system’s current adaptation prediction to take control of the adaptation process and make changes until the content is suitably adapted for her purposes. The adaptation system learns from the user’s modifications and adjusts its prediction for future accesses by other users. This paper shows that it is possible to exploit user interaction to learn how to adapt content based on context. We introduce Feedback-driven Context Selection (FCS), an automatic technique that leverages user interaction to identify the context that has the most impact on adaptation requirements. We added contextawareness to URICA so that it makes adaptation predictions for a user based only on the history of the community of users that share the context identified by FCS. The result is an automatic adaptation system that provides fine grain adaptations that reflect both the user’s context and the content’s usage semantics. This level of fine grain adaptation was previously available only in content that was customized manually. Experiments with two context-aware URICA prototypes show that FCS correctly identifies the contextual characteristics that impact adaptation requirements, and that grouping users into communities based on context improves the performance of the adaptation system by up to 79%.