The emergence of Vision Transformer (ViT) models and their variants (e.g., Swin Transformers) are prevalent in recent years due to their higher accuracy in vision AI applications. However, their efficient execution in edge computing environments (e.g., mobile phones/embedded platforms) remains a challenge due to the heavy computational demands (both GPU cycles and GPU memory) of these large-sized models. To serve these models efficiently at the edge, we introduce a novel approach combining model slicing and smart batching to distribute workloads between resource-constrained client devices and powerful edge servers. Model slicing allows breaking a large model into smaller segments, called slices, and let the client execute a few initial but a variable number of slices (head slices) and a nearby edge server runs the rest of the slices (tail slices), smart batching enables the server to queue several requests from multiple clients batch together for inference leading to better GPU resource utilization. We propose two batching strategies at the server: one runs faster but requires higher GPU memory and the other one demands less memory with slight overhead of internal data movement. Experimental results show that our approach achieves inference time reductions of up to 67 % while maintaining high GPU utilization and demonstrates significant improvements in inference speed, showcasing the viability of this approach for distributed AI systems.

Retrieval-augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for understanding of videos is appealing but there are two critical limitations. One-time, upfront conversion of all content in large corpus of videos into text descriptions entails high processing times. Also, not all information in the rich video data is typically captured in the text descriptions. Since user queries are not known apriori, developing a system for video to text conversion and interactive querying of video data is challenging.

Training a neural network model from scratch is a computationally intensive operation. To alleviate this issue, researchers often employ "transfer learning" that transfers knowledge from a source data distribution to a target data distribution, instead of training the whole model from scratch. Typically, the last few layers of a pretrained convolutional neural network (CNN) are chosen for many transfer learning tasks where the outputs of those selected layers are combined to construct a feature space based on which a task-specific classification n etwork i s t rained o r fi ne-tuned. Th is arbitrary way of selecting layers, however, often fails to achieve the desired accuracy for the target task. What we need is an intelligent way of selecting layers from a pretrained model for a given task so that the additional overhead of successive training remains low. To this end, we propose a novel method, called TransJury, to find t he m ost s ignificant la yers from a pretrained mo del for transfer learning along with preserving the knowledge for the source domain. Through extensive experimentation on several target domain datasets, we show the supremacy of our approach in terms of lower training overhead and improved accuracy. By deploying MobileNet-v2, a lightweight CNN model pretrained on the ImageNet dataset on an edge device, we also discuss the future direction of this research.

In this paper, we address the challenges in supporting reliability and scalability in societal-scale notification systems that aim to reach large populations with customized alerts. We explore fault tolerance (FT) techniques in the context of Big Data Publish-Subscribe systems (BDPS), a scalable hierarchical architecture, that meshes big-data platforms (to store and operate on large volumes of data) with a distributed pub/sub broker network (to manage and communicate with a large number of end subscribers). The role of brokers in this architecture is critical since they serve to mediate interactions between subscribers and the backend big data system. We propose the REAPS (REliable Active Publish Subscribe) framework that can handle different classes of broker failures including randomized failures and geographically-correlated failures (as in a natural disaster). REAPS implements a low overhead fault tolerance service using a primary-backup approach; key features include the ability to exploit subscription similarity among brokers and techniques for quasi-active state replication to support fast recovery and delivery guarantees of notification services. We implement REAPS and conduct measurement studies on a prototype BDPS platform using real world usecases. We further evaluate REAPS under various failure scenarios to explore the scalability and performance of our proposed FT mechanisms via simulation studies.

Nearly all of today's Big Data systems are passive in nature. We demonstrate our Big Active Data ("BAD") system, a scalable system that continuously and reliably captures Big Data and facilitates the timely and automatic delivery of new information to a large population of interested users as well as supporting analyses of historical information. We built our BAD project by extending an existing scalable, open-source BDMS (AsterixDB [1]) in this active direction. In this demonstration, we allow our audience to participate in an emergency notification application built on top of our BAD platform, and highlight its capabilities.