Deep Learning and Computer Vision advances enabled sophisticated information extraction out of images and videos. Recent research aims to make objects, their types and relative locations, first class citizens for query processing purposes. We initiate research to explore declarative queries for real time video streams involving objects and their interactions. We seek to efficiently identify frames in which an object is interacting with another in a specific way. We propose progressive filters (PF) algorithm which deploys a sequence of inexpensive and less accurate filters to detect the presence of query specified objects on frames. We demonstrate that PF derives a least cost sequence of filters given the query objects' current selectivities. Since selectivities may vary as the video evolves, we present a statistical test to determine when to trigger filters' re-optimization. Finally, we present Interaction Sheave, a filtering approach that uses learned spatial information about objects and interactions to prune frames that are unlikely to involve the query specified action between them, thus improving the frame processing rate. We present the results of a thorough experimental evaluation involving real datasets. We experimentally demonstrate that our techniques can improve query performance (up to an order of magnitude) while maintaining competitive F1-score.

Recent advances in video processing utilizing deep learning primitives achieved breakthroughs in fundamental problems in video analysis such as frame classification and object detection enabling an array of new applications. In this paper we study the problem of interactive declarative query processing on video streams. In particular we introduce a set of approximate filters to speed up queries that involve objects of specific type (e.g., cars, trucks, etc.) on video frames with associated spatial relationships among them (e.g., car left of truck). The resulting filters are able to assess quickly if the query predicates are true to proceed with further analysis of the frame or otherwise not consider the frame further avoiding costly object detection operations. We propose two classes of filters IC and OD, that adapt principles from deep image classification and object detection. The filters utilize extensible deep neural architectures and are easy to deploy and utilize. In addition, we propose statistical query processing techniques to process aggregate queries involving objects with spatial constraints on video streams and demonstrate experimentally the resulting increased accuracy on the resulting aggregate estimation. Combined these techniques constitute a robust set of video monitoring query processing techniques. We demonstrate that the application of the techniques proposed in conjunction with declarative queries on video streams can dramatically increase the frame processing rate and speed up query processing by at least two orders of magnitude. We present the results of a thorough experimental study utilizing benchmark video data sets at scale demonstrating the performance benefits and the practical relevance of our proposals.

Deep neural nets enabled sophisticated information extraction out of images, including video frames. Recently, there has been interest in techniques and algorithms to enable interactive declarative query processing of objects appearing on video frames and their associated interactions on the video feed. SVQ++ is a system for declarative querying on real-time video streams involving objects and their interactions. The system utilizes a sequence of inexpensive and less accurate models (filters), called Progressive Filters (PF), to detect the presence of the query specified objects on frames, and a filtering approach, called Interaction Sheave (IS), to effectively prune frames that are not likely to contain interactions. We demonstrate that this system can efficiently identify frames in a streaming video in which an object is interacting with another in a specific way, increasing the frame processing rate dramatically and speed up query processing by at least two orders of magnitude depending on the query.

Recent advances in video processing utilizing deep learning primitives achieved breakthroughs in fundamental problems in video analysis such as frame classification and object detection enabling an array of new applications.