Video Database Management Systems (VDBMS) leverage advancements in computer vision and deep learning for efficient video data analysis and retrieval. This paper introduces the concept of user-specified Clues, allowing users to incorporate domain-specific knowledge, referred to as Clues, into query optimization. Clues are expressed as Clue types, each associated with optimization rules, and applied to queries through Clue instances. The extensible ClueVQS system we present to incorporate these ideas, optimizes queries automatically, utilizing Clues to improve processing efficiency. We also introduce algorithms to optimize queries using Clues allowing for trade-offs between speed and query accuracy. Our proposals and system address challenges such as data-dependent Clue effectiveness, limiting search space, and accuracy-efficiency trade-offs. Detailed experimental results demonstrate query speedups of up to two orders of magnitude compared to other applicable approaches, and a reduction of the query optimizer time by up to 95% while respecting user-specified accuracy constraints, showcasing the effectiveness of the proposed framework.

Video analysis frameworks supporting declarative queries are actively researched in recent years. A major prerequisite in executing such queries is the ability to accurately extract metadata at the frame level utilizing various computer vision algorithms, including object tracking models. Tracking models are of profound importance as they establish unique identifiers for the objects across frames.Despite the maturity of tracking algorithms, they still face challenges (such as occlusions, object glaze etc.) which diminish their quality and accuracy. This gives rise to the track fragmentation problem in which a single track is fragmented into multiple smaller tracks. This impacts downstream temporal querying applications degrading query accuracy.In this paper, we propose an algorithm, TMerge for identifying and merging fragmented tracks that constitutes a pre-processing step during data ingestion for video query processing. The algorithm exploits the properties of the problem and utilizes a sampling methodology that significantly reduces the time required to pre-process and ingest the video sequence.We comprehensively describe and analyze our proposals utilizing real data sets and also present the results of a detailed experimental evaluation varying parameters of interest. We demonstrate performance savings of up to two orders of magnitude without loss in accuracy.

We present SVQ-ACT, a system capable of evaluating declarative action and object queries over input videos. Our approach is independent of the underlying object and action detection models utilized. Users may issue queries involving action and specific objects (e.g., a human riding a bicycle, close to a traffic light and a car left of the bicycle) and identify video clips that satisfy query constraints. Our system is capable of operating in two main settings, namely online and offline. In the online setting, the user specifies a video source (e.g., a surveillance video) and a declarative query containing an action and object predicates. Our system will identify and label in real-time all frame sequences that match the query. In the offline mode, the system accepts a video repository as input, preprocesses all the video in an offline manner and extracts suitable metadata. Following this step, users can execute any query they wish interactively on the video repository (containing actions and objects supported by the underlying detection models) to identify sequences of frames from videos that satisfy the query. In this case, to limit the number of results produced, we introduce novel result ranking algorithms that can produce the k most relevant results efficiently.We demonstrate that SVQ-ACT can correctly capture the desired query semantics and execute queries efficiently and correctly, delivering a high degree of accuracy.

Numerous cloud platforms are available to deploy and train deep models as well as process data, such as Amazon Rekognition and Azure custom Vision Service, which have made it easy for companies to adopt deep learning technologies in their operations. Commonly such services price usage per image or frame in typical applications that consume video streams and as a result the costs rapidly accumulate. In this paper we introduce a model, named EventHit, that is able to marshal model inference requests in such services by making predictions over the video stream about events of interest. As such only relevant video segments are sent for analysis to the cloud infrastructure and irrelevant parts are filtered from further processing. We introduce the architecture and fully describe its components. We present two novel optimizations in this context that aim to provide control over the trade-off between prediction accuracy (especially regarding the probability of missing an event of interest) and processing cost at the cloud infrastructure. We fully describe and analyze our proposals in the context of real datasets. We also present the results of a detailed experimental evaluation varying parameters of interest and demonstrate the practical utility of our proposals.

Deep neural nets enabled sophisticated information extraction out of images, including video frames. Recently, there has been interest in techniques and algorithms to enable interactive declarative query processing of objects appearing on video frames and their associated interactions on the video feed. SVQ++ is a system for declarative querying on real-time video streams involving objects and their interactions. The system utilizes a sequence of inexpensive and less accurate models (filters), called Progressive Filters (PF), to detect the presence of the query specified objects on frames, and a filtering approach, called Interaction Sheave (IS), to effectively prune frames that are not likely to contain interactions. We demonstrate that this system can efficiently identify frames in a streaming video in which an object is interacting with another in a specific way, increasing the frame processing rate dramatically and speed up query processing by at least two orders of magnitude depending on the query.

Short text stream clustering has become an increasingly important problem due to the explosive growth of short text in diverse social medias. In this paper, we propose a model-based short text stream clustering algorithm (MStream) which can deal with the concept drift problem and sparsity problem naturally. The MStream algorithm can achieve state-of-the-art performance with only one pass of the stream, and can have even better performance when we allow multiple iterations of each batch. We further propose an improved algorithm of MStream with forgetting rules called MStreamF, which can efficiently delete outdated documents by deleting clusters of outdated batches. Our extensive experimental study shows that MStream and MStreamF can achieve better performance than three baselines on several real datasets.