Classification tasks play a fundamental role in various applications, spanning domains such as healthcare, natural language processing and computer vision. With the growing popularity and capacity of machine learning models, people can easily access trained classifiers as a service online or offline. However, model use comes with a cost and classifiers of higher capacity (such as large foundation models) usually incur higher inference costs. To harness the respective strengths of different classifiers, we propose a principled approach, OCCAM, to compute the best classifier assignment strategy over classification queries (termed as the optimal model portfolio) so that the aggregated accuracy is maximized, under user-specified cost budgets. Our approach uses an unbiased and low-variance accuracy estimator and effectively computes the optimal solution by solving an integer linear programming problem. On a variety of real-world datasets, OCCAM achieves 40% cost reduction with little to no accuracy drop.

Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.

Knowledge Base Question Answering (KBQA) aims to find correct answers to natural language questions by reasoning over large-scale knowledge bases.The main challenge is multi-hop reasoning, which requires inferring answers through multiple edges and nodes.Due to high data annotation costs, most datasets only provide final answer annotations, leaving the reasoning process unknown.The model cannot provide the reasoning path along with the answer, making the answer uninterpretable.To address this, we propose a generation-retrieval multi-hop KBQA method combining large language models (LLMs) and the logic programming language Prolog.In the generation phase, we fine-tune the open-source LLMs to generate the logical form of the corresponding prolog representation for the natural language question.In the retrieval phase, the model uses prolog query to reasoning over the KB to get the final answer and the reasoning path.A transparent reasoning path also helps identify and correct errors, enhancing the model's reliability and practicality.Experimental results on two standard KBQA datasets, MetaQA and WebQSP, demonstrate that our method outperforms existing models in both performance and interpretability.

The goal of scene graph generation is to predict a graph from an input image, where nodes correspond to identified and localized objects and edges to their corresponding interaction predicates. Existing methods are trained in a fully supervised manner and focus on message passing mechanisms, loss functions, and/or bias mitigation. In this work we introduce a simple-yet-effective self-supervised relational alignment regularization designed to improve the scene graph generation performance. The proposed alignment is general and can be combined with any existing scene graph generation framework, where it is trained alongside the original model’s objective. The alignment is achieved through distillation, where an auxiliary relation prediction branch, that mirrors and shares parameters with the supervised counterpart, is designed. In the auxiliary branch, relational input features are partially masked prior to message passing and predicate prediction. The predictions for masked relations are then aligned with the supervised counterparts after the message passing. We illustrate the effectiveness of this self-supervised relational alignment in conjunction with two scene graph generation architectures, SGTR [25] and Neural Motifs [53], and show that in both cases we achieve significantly improved performance.

Multi-modal learning, particularly among imaging and linguistic modalities, has made amazing strides in many high-level fundamental visual understanding problems, ranging from language grounding to dense event captioning. However, much of the research has been limited to approaches that either do not take audio corresponding to video into account at all, or those that model the audio-visual correlations in service of sound or sound source localization. In this paper, we present the evidence, that audio signals can carry surprising amount of information when it comes to high-level visual-lingual tasks. Specifically, we focus on the problem of weakly-supervised dense event captioning in videos and show that audio on its own can nearly rival performance of a state-of-the-art visual model and, combined with video, can improve on the state-of-the-art performance. Extensive experiments on the ActivityNet Captions dataset show that our proposed multi-modal approach outperforms state-of-the-art unimodal methods, as well as validate specific feature representation and architecture design choices.

Offering the services of real-time tracking and arrival time prediction is a common welfare for bus riders and transit agencies, especially in urban environments. On the down side, the traditional GPS-based solutions work poorly in urban areas due to urban canyons, while the location systems based on cellular signal also suffer from inherent limitations. In this paper, we present a powerful tool named Signal Voronoi Diagram (SVD) to partition the radio-frequency (RF) signal space of WiFi Access Points (APs), distributed where a bus travels, into Signal Cells, and then into fine-grained Signal Tiles, tackling the problem of noisy received signal strength (RSS) readings and possible AP dynamics. On top of SVD, we present a novel framework so-called WiLocator, to track and predict the arrival time of an urban bus based on the surrounding WiFi information collected by the commodity off-the-shelf (COTS) smartphones of bus riders, the mobility constraint of a bus and the temporal consistency of travel time of buses on the overlapped road segments. We also show the WiLocator's power of generating an accurate and real-time traffic map with the predicted travel time on each road segment. We implement the prototype of WiLocator and conduct the in-situ experiment to demonstrate its accuracy.