In this paper, we address the dual problems of retrieval and part (Region Of Interest (ROI)) matching, especially in large-scale image datasets. At the heart of our solution lies a patch-based unsupervised deep image representational model, termed a Bag of Variational Deep Embedded Visual Words (BoVDEVW). This model has three main components. Firstly, we extract a set of informative patches based on their entropy values. Secondly, the Variational AutoEncoder (VAE) is used to represent these informative patches in a robust low-dimensional probabilistic latent space. Third, deep embedded clustering is applied to group them. We introduce a composite loss function in this connection, which consists of reconstruction, regularization, and clustering losses, and explicitly show that it is convex. The cluster centers are termed as Variational Deep Embedded Visual Words. Image retrieval is performed by computing the Euclidean distance between a query image and the database images, each being represented as a Bag of Variational Deep Embedded Visual Words (BoVDEVW). The problem of finding ROI(s) in the retrieved images corresponding to ROI(s) in a query image is modeled as a matching problem in a bipartite graph constructed in the probabilistic latent space of the VAE. Comprehensive experimentation on large image datasets, NUS-WIDE and Google Landmarks Dataset (GLD) v2, and medium image datasets, Oxford-5K and Paris-6K clearly shows the efficacy of our proposed solution.

Analysis of egocentric video has recently drawn attention of researchers in the computer vision as well as multimedia communities. In this paper, we propose a weakly supervised superpixel level joint framework for localization, recognition and summarization of actions in an egocentric video. We first recognize and localize single as well as multiple action(s) in each frame of an egocentric video and then construct a summary of these detected actions. The superpixel level solution helps in precise localization of actions in addition to improving the recognition accuracy. Superpixels are extracted within the central regions of the egocentric video frames; these central regions being determined through a previously developed center-surround model. A sparse spatio-temporal video representation graph is constructed in the deep feature space with the superpixels as nodes. A weakly supervised solution using random walks yields action labels for each superpixel. After determining action label(s) for each frame from its constituent superpixels, we apply a fractional knapsack type formulation for obtaining a summary (of actions). Experimental comparisons on publicly available ADL, GTEA, EGTEA Gaze+, EgoGesture, and EPIC-Kitchens datasets show the effectiveness of the proposed solution.

In this paper, we have addressed the problem of automated detection of safe zone(s) for helicopter landing in hazardous environments from videos captured by an Unmanned Aerial Vehicle (UAV). The unconstrained motion of the video capturing drone (the UAV in our case) makes the problem further difficult. The solution pipeline consists of natural landmark detection and tracking, stereo-pair generation using constrained graph clustering, digital terrain map construction and safe landing zone detection. The main methodological contribution lies in mathematically formulating epipolar constraint and then using it in a Minimum Spanning Tree (MST) based graph clustering approach. We have also made publicly available AHL (Autonomous Helicopter Landing) dataset, a new aerial video dataset captured by a drone, with annotated ground-truths. Experimental comparisons with other competing clustering methods i) in terms of Dunn Index and Davies Bouldin Index as well as ii) for frame-level safe zone detection in terms of F-measure and confusion matrix clearly demonstrate the effectiveness of the proposed formulation.

Video co-summarization has emerged as an important problem in the areas of computer vision and multimedia communities. In this paper, we present a novel approach of co-summarizing egocentric videos at shot level. Our solution pipeline consists of three major components. We develop a new way of characterizing egocentric video frames by computing the differences in contrast, entropy and optic flow values between a central region and the surrounding region in a frame. This is termed as the center-surround model. Visual similarity between a test video shot and a database video shot is next derived using a game-theoretic framework. Each video shot is modelled as a player and the expected pay-off difference between any two such players at mixed Nash equilibrium is deemed as the similarity between them. A weighted bipartite graph is constructed next between the shots in a test and in a database video. Game-theoretic similarity values are deemed as the weights. Maximum Cardinality Minimum Weight matching in the bipartite graph yields non-greedy shot correspondences. Best matched shots from the test video are used to form the summary. Experimental comparisons on standard datasets clearly indicate the advantage of our solution.