Few-shot recognition (FSR) aims to train a classification model with only a few labeled examples of each concept concerned by a downstream task, where data annotation cost can be prohibitively high. We develop methods to solve FSR by leveraging a pretrained Vision-Language Model (VLM). We particularly explore retrieval-augmented learning (RAL), which retrieves open data, e.g., the VLM’s pre-training dataset, to learn models for better serving downstream tasks. RAL has been studied in zero-shot recognition but remains under-explored in FSR. Although applying RAL to FSR may seem straightforward, we observe interesting and novel challenges and opportunities. First, somewhat surprisingly, finetuning a VLM on a large amount of retrieved data underperforms state-of-the-art zero-shot methods. This is due to the imbalanced distribution of retrieved data and its domain gaps with the few-shot examples in the downstream task. Second, more surprisingly, we find that simply finetuning a VLM solely on few-shot examples significantly outperforms previous FSR methods, and finetuning on the mix of retrieved and few-shot data yields even better results. Third, to mitigate the imbalanced distribution and domain gap issues, we propose Stage-Wise retrieval-Augmented fineTuning (SWAT), which involves end-to-end finetuning on mixed data in the first stage and retraining the classifier on the few-shot data in the second stage. Extensive experiments on nine popular benchmarks demonstrate that SWAT significantly outperforms previous methods by >6% accuracy.

Localizing unusual activities in videos, such as abnormal behaviors or traffic incidents, holds practical significance. However, pretrained foundation models struggle with localizing diverse unusual events likely because of their insufficient representation in the models' pretraining datasets. To explore foundation models' capability in localizing unusual activities, we introduce UALBench, a comprehensive benchmark for unusual activity localization, featuring three video datasets (UAG-OOPS, UAG-SSBD, and UAG-FunQA), and an instruction-tuning dataset (OOPS-UAG-Instruct), to improve model capabilities. We also introduce a new metric, R@1, TD ≤ p, as an auxiliary metric to reasonably consider detections as true positive if their starting and ending timestamps are within a threshold. On UAL-Bench, we evaluate three approaches: Video-Language Models (VidLLMs), instruction-tuned Vid-LLMs, and a novel integration of Vision-Language Models and Large Language Models (VLM-LLM). Our results show the VLM-LLM approach excels in localizing short-span unusual events and predicting their onset (start time) more accurately than VidLLMs. Our findings highlight the challenges posed by longduration videos, particularly in autism diagnosis scenarios, and the need for further advancements in localization techniques. Our work not only provides a benchmark for unusual activity localization but also outlines the key challenges for existing foundation models, suggesting future research directions on this important task.

Vision-language models (VLMs) excel in zero-shot recognition but their performance varies greatly across different visual concepts. For example, although CLIP achieves impressive accuracy on ImageNet (60-80%), its performance drops below 10% for more than ten concepts like night snake, presumably due to their limited presence in the pretraining data. However, measuring the frequency of concepts in VLMs' large-scale datasets is challenging. We address this by using large language models (LLMs) to count the number of pretraining texts that con-tain synonyms of these concepts. Our analysis confirms that popular datasets, such as LAION, exhibit a long-tailed concept distribution, yielding biased performance in VLMs. We also find that downstream applications of VLMs, including visual chatbots (e.g., GPT-4V) and text-to-image models (e.g., Stable Diffusion), often fail to recognize or generate images of rare concepts identified by our method. To mit-igate the imbalanced performance of zero-shot VLMs, we propose REtrieval-Augmented Learning (REAL). First, in-stead of prompting VLMs using the original class names, REAL uses their most frequent synonyms found in pretraining texts. This simple change already outperforms costly human-engineered and LLM-enriched prompts over nine benchmark datasets. Second, REAL trains a linear classifier on a small yet balanced set of pretraining data re-trieved using concept synonyms. REAL surpasses the previous zero-shot SOTA, using 400× less storage and 10,000× less training time!

Accurate and prompt delivery of Emergency Medical Services (EMS) is critical in emergency incidents, e.g., man-made or natural disaster areas. However, quickly selecting the correct EMS protocol(s) (which dictate the medical procedures to be administered to patients) in complex medical scenarios, remains a key, demanding task for Emergency Medical Technicians (EMT). In this paper, we present EMSAssist, the first end-to-end mobile voice assistant at the edge for EMS. EMSAssist consists of three major components that address technical challenges present in state-of-the-art solutions: 1) For the first time, EMSAssist proposes and applies a few-sample fine-tuning technique in medical speech recognition task, that achieves a faster and more accurate speech transcription on our EMS audio dataset, when compared to Google Cloud Speech-to-Text; 2) A WordPiece tokenizer helps boosting the end-to-end EMS protocol selection accuracy by retrieving useful information from incorrect transcriptions; 3) A novel data customization framework that enables our data-driven EMSMobileBERT model to become the new state-of-the-art for EMS protocol selection. Extensive end-to-end evaluation results at the edge show EMSAssist can more accurately select EMS protocols (Top-5 accuracy above 96%) for EMTs, with end-to-end latencies of around 4.2 seconds.

We present EMSAssist, the first end-to-end mobile voice assistant for emergency medical services (EMS). EMSAssist allows Emergency Medical Technicians (EMT) to verbally describe patients' signs and symptoms and uses EMTs' voice input to recommend top-5 EMS protocols. Through this demo, we allow the attendees to evaluate EMSAssist through a pair of Google Glass and a mobile phone. Both mobile devices can collect users' voices as input and output top-5 recommended protocols. A companion youtube video of using EMSAssist on the Google Glass is provided: https://www.youtube.com/watch?v=bj7aQJKf4aE