In this paper, we address the dual problems of retrieval and part (Region Of Interest (ROI)) matching, especially in large-scale image datasets. At the heart of our solution lies a patch-based unsupervised deep image representational model, termed a Bag of Variational Deep Embedded Visual Words (BoVDEVW). This model has three main components. Firstly, we extract a set of informative patches based on their entropy values. Secondly, the Variational AutoEncoder (VAE) is used to represent these informative patches in a robust low-dimensional probabilistic latent space. Third, deep embedded clustering is applied to group them. We introduce a composite loss function in this connection, which consists of reconstruction, regularization, and clustering losses, and explicitly show that it is convex. The cluster centers are termed as Variational Deep Embedded Visual Words. Image retrieval is performed by computing the Euclidean distance between a query image and the database images, each being represented as a Bag of Variational Deep Embedded Visual Words (BoVDEVW). The problem of finding ROI(s) in the retrieved images corresponding to ROI(s) in a query image is modeled as a matching problem in a bipartite graph constructed in the probabilistic latent space of the VAE. Comprehensive experimentation on large image datasets, NUS-WIDE and Google Landmarks Dataset (GLD) v2, and medium image datasets, Oxford-5K and Paris-6K clearly shows the efficacy of our proposed solution.

Brain cancer has a very high mortality rate. Gliomas are the most common malignant brain tumors, causing this severe fatality. Recent biological investigations revealed that a holistic study of biomarkers, responsible for causing genetic mutations in gliomas, can ensure a comprehensive prognosis and treatment plan for the patients. In this paper, we simultaneously predict five such important genetic markers, namely, IDH, 1p/19q codeletion status, ATRX, MGMT, and TERT from Whole Slide Images using deep learning. At the heart of our deep learning based solution lies a novel composite loss function, which uniquely combines individual, pairwise and groupwise traits of the above bio-markers. While multi-label weighted cross-entropy loss captures individual characteristics, a conditional probability loss was developed for pairwise behavior, and, a spectral graph loss was formulated for modeling group properties. Comprehensive experimentation, along with an ablation study, clearly demonstrates the effectiveness of our solution, by achieving state-of-the-art prediction.

Lung cancer is the leading cause of cancer-related deaths worldwide. According to the American Cancer Society, early diagnosis of pulmonary nodules in computed tomography (CT) scans can improve the five-year survival rate up to 70% with proper treatment planning. In this article, we propose an attribute-driven Generative Adversarial Network (ADGAN) for synthesis and multiclass classification of Pulmonary Nodules. A self-attention U-Net (SaUN) architecture is proposed to improve the generation mechanism of the network. The generator is designed with two modules, namely, self-attention attribute module (SaAM) and a self-attention spatial module (SaSM). SaAM generates a nodule image based on given attributes whereas SaSM specifies the nodule region of the input image to be altered. A reconstruction loss along with an attention localization loss (AL) is used to produce an attention map prioritizing the nodule regions. To avoid resemblance between a generated image and a real image, we further introduce an adversarial loss containing a regularization term based on KL divergence. The discriminator part of the proposed model is designed to achieve the multiclass nodule classification task. Our proposed approach is validated over two challenging publicly available datasets, namely LIDC-IDRI and LUNGX. Exhaustive experimentation on these two datasets clearly indicate that we have achieved promising classification accuracy as compared to other state-of-the-art methods.

Hippocampus (HC) segmentation plays a key role in diagnosis of predominant neuro-degenerative diseases like Alzheimer's, Parkinson's and common neurological disorders like Epilepsy. In this paper, we propose a solution to the 3D HC segmentation problem from the MRI data using shape driven loss function and attention Unet. In particular, a Histogram of Oriented Gradients (HOG) based formulation is developed to extract shape features. We suggest a pooling technique as a substitute to the histogram calculation for HOG. This is to address the problem that histogram is not derivable thereby making the error in a loss function from histogram unsuitable for back propagation in deep learning models. The performance of our proposed model is validated on two publicly available datasets, namely, HarP and Kulaga-Yoskovitz (KY). Our segmentation accuracy with a dice similarity score of 0.947 and 0.923 in HarP and KY respectively is found to outperform the attention UNet model with only Dice loss, and, a number of state-of-the-art approaches.

In this paper, we address the problem of brain tumor classification from radiology and histopathology data. A coarse-to-fine classification approach is adopted using a combination of deep features and Graph Convolution Network (GCN). As a first coarse step, we use 3D CNN to detect Glioblastoma from MRI images. In order to infer about Astrocytoma and Oligodendroglioma, Whole Slide Images (WSI) are employed in the second stage. During this fine classification stage, 2D CNN features are extracted at two different (global and local) magnification levels. A graph is constructed with nodes in the space of concatenated global and local features. Edges are constructed from feature similarity and graph topology. Finally, GCN is used with normalized graph Laplacian to ensure better relation-aware-representation leading to more accurate classification. Experimental comparisons on the CPM-RadPath2020 challenge dataset clearly demonstrate the state-of-the-art performance of our proposed strategy. The code implementation is available at https://github.com/arijitde92/BrainTumorClassification.

Analysis of egocentric video has recently drawn attention of researchers in the computer vision as well as multimedia communities. In this paper, we propose a weakly supervised superpixel level joint framework for localization, recognition and summarization of actions in an egocentric video. We first recognize and localize single as well as multiple action(s) in each frame of an egocentric video and then construct a summary of these detected actions. The superpixel level solution helps in precise localization of actions in addition to improving the recognition accuracy. Superpixels are extracted within the central regions of the egocentric video frames; these central regions being determined through a previously developed center-surround model. A sparse spatio-temporal video representation graph is constructed in the deep feature space with the superpixels as nodes. A weakly supervised solution using random walks yields action labels for each superpixel. After determining action label(s) for each frame from its constituent superpixels, we apply a fractional knapsack type formulation for obtaining a summary (of actions). Experimental comparisons on publicly available ADL, GTEA, EGTEA Gaze+, EgoGesture, and EPIC-Kitchens datasets show the effectiveness of the proposed solution.

Person re-identification in the wild needs to simultaneously (frame-wise) detect and re-identify persons and has wide utility in practical scenarios. However, such tasks come with an additional open-set re-ID challenge as all probe persons may not necessarily be present in the (frame-wise) dynamic gallery. Traditional or close-set re-ID systems are not equipped to handle such cases and raise several false alarms as a result. To cope with such challenges open-set metric learning (OSML), based on the concept of Large margin nearest neighbor (LMNN) approach, is proposed. We term our method Open-Set LMNN (OS-LMNN). The goal of separating impostor samples from the genuine samples is achieved through a joint optimization of the Weibull distribution and the Mahalanobis metric learned through this OS-LMNN approach. The rejection is performed based on low probability over distance of imposter pairs. Exhaustive experiments with other metric learning techniques over the publicly available PRW dataset clearly demonstrate the robustness of our approach.

In this paper, we have addressed the problem of automated detection of safe zone(s) for helicopter landing in hazardous environments from videos captured by an Unmanned Aerial Vehicle (UAV). The unconstrained motion of the video capturing drone (the UAV in our case) makes the problem further difficult. The solution pipeline consists of natural landmark detection and tracking, stereo-pair generation using constrained graph clustering, digital terrain map construction and safe landing zone detection. The main methodological contribution lies in mathematically formulating epipolar constraint and then using it in a Minimum Spanning Tree (MST) based graph clustering approach. We have also made publicly available AHL (Autonomous Helicopter Landing) dataset, a new aerial video dataset captured by a drone, with annotated ground-truths. Experimental comparisons with other competing clustering methods i) in terms of Dunn Index and Davies Bouldin Index as well as ii) for frame-level safe zone detection in terms of F-measure and confusion matrix clearly demonstrate the effectiveness of the proposed formulation.

This letter introduces a unified framework for accurate segmentation of five different types of pulmonary nodules, namely, solid, juxtapleural, juxtavascular, part solid and ground glass by designing a contrast-adaptive shape-driven level set algorithm. Most of the existing methods have targeted segmenting few specific types of nodules. Variability of shapes along with poor contrast make pulmonary nodule segmentation an extremely challenging problem. To deal with low contrast, a contrast-adaptive term, based on intensities, is incorporated to guide the evolution of level set. A shape term is further introduced for accurate segmentation of different pulmonary nodules having varying shapes. Experiments on the publicly available LIDC/IDRI dataset clearly reveal that our method achieves promising results as compared to several state-of-the-art competitors.

Superpixel segmentation has emerged as an important research problem in the areas of image processing and computer vision. In this paper, we propose a framework, namely Iterative Spanning Forest (ISF), in which improved sets of connected superpixels (supervoxels in 3D) can be generated by a sequence of image foresting transforms. In this framework, one can choose the most suitable combination of ISF components for a given application-i.e., 1) a seed sampling strategy; 2) a connectivity function; 3) an adjacency relation; and 4) a seed pixel recomputation procedure. The superpixels in ISF structurally correspond to spanning trees rooted at those seeds. We present five ISF-based methods to illustrate different choices for those components. These methods are compared with a number of state-of-the-art approaches with respect to effectiveness and efficiency. Experiments are carried out on several datasets containing 2D and 3D objects with distinct texture and shape properties, including a high-level application, named sky image segmentation. The theoretical properties of ISF are demonstrated in the supplementary material and the results show ISF-based methods rank consistently among the best for all datasets.

Video co-summarization has emerged as an important problem in the areas of computer vision and multimedia communities. In this paper, we present a novel approach of co-summarizing egocentric videos at shot level. Our solution pipeline consists of three major components. We develop a new way of characterizing egocentric video frames by computing the differences in contrast, entropy and optic flow values between a central region and the surrounding region in a frame. This is termed as the center-surround model. Visual similarity between a test video shot and a database video shot is next derived using a game-theoretic framework. Each video shot is modelled as a player and the expected pay-off difference between any two such players at mixed Nash equilibrium is deemed as the similarity between them. A weighted bipartite graph is constructed next between the shots in a test and in a database video. Game-theoretic similarity values are deemed as the weights. Maximum Cardinality Minimum Weight matching in the bipartite graph yields non-greedy shot correspondences. Best matched shots from the test video are used to form the summary. Experimental comparisons on standard datasets clearly indicate the advantage of our solution.

Video surveillance systems have become extremely important recently. It has been observed that information extracted from a single spectrum video is often insufficient in adverse conditions like low illumination, shadowing, smoke, dust, unstable background, and camouflage. Real-time video processing systems also need to be very fast where future frames are often unavailable at the time of processing the current frame. In this paper, we propose a superpixel-based causal multisensor video fusion algorithm suitable for real-time surveillance tasks. We develop new superpixel level spatial and temporal saliency models. Novel superpixel level multiple fusion rules are also designed to obtain the fused output. Comprehensive comparisons with several existing works clearly indicate the benefit of our solution.

Movie scene detection has emerged as an important problem in present day multimedia applications. Since a movie typically consists of huge amount of video data with widespread content variations, detecting a movie scene has become extremely challenging. In this paper, we propose a fast yet accurate solution for movie scene detection using Nyström approximated multisimilarity spectral clustering with a temporal integrity constraint. We use multiple similarity matrices to model the wide content variations typically present in any movie dataset. Nyström approximation is employed to reduce the high computational cost of constructing multiple similarity measures. The temporal integrity constraint captures the inherent temporal cohesion of the movie shots. Experiments on five movie datasets from different genres clearly demonstrate the superiority of the proposed solution over the state-of-the-art methods.

Level set-based contour tracking methods have generated recent interest in the computer vision community. In this paper, we propose a novel level set-based algorithm for tracking dynamic implicit contours that utilizes minimal prior information. Our solution consists of two main steps. In the first step, a simple first-order Markov chain model is employed for the coarse localization of a target object. In the second step, we evolve level sets within a narrow band to accurately track the target contour. Narrow band curve evolution is guided through color- and region-based terms in the standard Chan-Vese framework. Comprehensive experimentation on a dataset comprising of several publicly available video sequences clearly demonstrate the advantage of the proposed tracking algorithm.

The task of multi-view video summarization is to efficiently represent the most significant information from a set of videos captured for a certain period of time by multiple cameras. The problem is highly challenging because of the huge size of the data, presence of many unimportant frames with low activity, inter-view dependencies, and significant variations in illumination. In this paper, we propose a graph-theoretic solution to the above problems. Semantic feature in form of visual bag of words and visual features like color, texture, and shape are used to model shot representative frames after temporal segmentation . Gaussian entropy is then applied to filter out frames with low activity. Inter-view dependencies are captured via bipartite graph matching. Finally, the optimum-path forest algorithm is applied for the clustering purpose. Subjective as well as objective evaluations clearly indicate the effectiveness of the proposed approach.

Accuracy of the well-known kNN classifier depends significantly on the suitable choice of k. In this paper, we propose an improved kNN algorithm with a novel non-parametric test point specific k estimation strategy. To estimate k for any test point, we first construct a hypersphere around it to capture the local distribution of the surrounding training points. Class hubness information is then used as a weight on the hypervolume of the above hyper sphere. Experiments on several UCI benchmark datasets clearly demonstrate the supremacy of our improved kNN algorithm over various existing versions such as i) kNN with fixed values of k (k= 1, 3, 5, 7, [Number of training points]) [1-3], ii) kNN with test point specific k [4], and, iii) kNN with hubness information [5].

Scalable video summarization has emerged as an important problem in present day multimedia applications. Effective summaries need to be provided to the users for videos of any duration at low computational cost. In this paper, we propose a framework which is scalable during both the analysis and the generation stages of video summarization. The problem of scalable video summarization is modeled as a problem of scalable graph clustering and is solved using skeleton graph and random walks in the analysis stage. A cluster significance factor-based ranking procedure is adopted in the generation stage. Experiments on videos of different genres and durations clearly indicate the supremacy of the proposed method over a recently published work.

Muscle satellite cells, also known as myogenic precursor cells, are the dedicated stem cells responsible for postnatal skeletal muscle growth, repair, and hypertrophy. Biological studies aimed at describing satellite cell activity on their host myofiber using timelapse light microscopy enable qualitative study, but high-throughput automatic tracking of satellite cells translocating on myofibers is very difficult due to their complex motion across the three-dimensional surface of myofibers and the lack of discriminating cell features. Other complicating factors include inhomogeneous illumination, fixed focal plane, low contrast, and stage motion. We propose a semi-automated approach for satellite cell tracking on myofibers consisting of registration with illumination correction, background subtraction and particle filtering. Initial experimental results show the effectiveness of the approach.

Traumatic injury of the pelvis is common and potentially devastating, with pelvic fractures being a major cause of trauma patient mortality. Detection and management of pelvic injuries is challenging due to varying injury patterns and resulting complications such as hemorrhage and infection. In this paper, we investigate the application of computer-aided detection (CAD) techniques for pelvic fracture detection. We propose a fast semi-automated method of pelvic fracture detection using a combination of (i) graph cuts and (ii) mean and Gaussian curvatures. A fracture is modeled as a minimum cut in a weighted graph. The same fracture is alternatively modeled as a valley based on the signs of mean and Gaussian curvatures. Each of these methods, in isolation, generates false positives in addition to the true fracture. We then combine the two methods and perform a neighborhood analysis to eliminate the false positives. Experimental results indicate that proposed method is very promising.

Study of cerebral white matter in the brain is an important medical problem which helps in better understanding of brain disorders like autism. The goal of this research is to segment the cerebral white matter from the input Magnetic Resonance Imaging (MRI) data. The present segmentation problem becomes extremely difficult due to i) the complex shape of the cerebral white matter and ii) the very low contrast between the white matter and the surrounding structures in the MRI data. We employ a novel probabilistic graph cut algorithm, where the edge capacity functions of the classical graph cut algorithm are modified according to the probabilities of pixels to belong to different segmentation classes. In order to separate the surrounding structures from the white matter, two appropriate geometric shape priors are introduced. Experimentation in 2D with 20 different datasets has yielded an average segmentation accuracy of 94.78%.

Automated visual tracking of cells from video microscopy has many important biomedical applications. In this paper, we model the problem of cell tracking over pairs of video microscopy image frames as a minimum weight matching problem in bipartite graphs. The bipartite matching essentially establishes one-to-one correspondences between the cells in different frames. A key advantage of using bipartite matching is the inherent scalability, which arises from its polynomial time-complexity. We propose two different tracking methods based on bipartite graph matching and properties of Gaussian distributions. In both the methods, i) the centers of the cells appearing in two frames are treated as vertices of a bipartite graph and ii) the weight matrix contains information about distance between the cells (in two frames) and cell velocity. In the first method, we identify fast-moving cells based on distance and filter them out using Gaussian distributions before the matching is applied. In the second method, we remove false matches using Gaussian distributions after the bipartite graph matching is employed. Experimental results indicate that both the methods are promising while the second method has higher accuracy.

In fuzzy abduction, one needs to evaluate the membership distribution of the premise (antecedent clause), when the membership distribution of the consequent clause, and the fuzzy implication relations between the antecedent and the consequent clauses are provided. The paper formulates and solves the problem of fuzzy abduction by using type-2 fuzzy sets. It presumes background knowledge about the primary and the secondary antecedent to consequent implication relations to uniquely determine the type-2 fuzzy set corresponding to the antecedent clause, when the same for the consequent clause is provided. The proposed methodology of abduction would serve many interesting applications on predictions, forecasting, and diagnosis, where the environmental factor can be modeled with type-2 secondary distributions.

Colon cancer is the second leading cause of cancer-related deaths per year in industrial nations. Virtual colonoscopy is a new, less invasive alternative to the usually practiced optical colonoscopy for colorectal polyp and cancer screening. In this paper, we present some physics-based modeling and pattern recognition techniques to identify anatomical landmarks in the human colon like the haustral folds and the tenia coli to further exploit the benefits of virtual colonoscopy. A combination of heat diffusion field algorithm and fuzzy c-means clustering algorithm is used to detect the haustral folds in human colon from volumetric computed tomography (CT) images. Each voxel on the corresponding colon surface is parameterized using the colon centerline information and associated local Frenet frames. The parameterized fold information is utilized to establish the tentative location of one tenia coli. Preliminary results on automated detection of tenia coli are shown on the colon surface.

Detection of hairline fractures, representing points or areas of discontinuity in the bone, is a clinically challenging task, especially in presence of noise. The above problem is equally appealing from a computer vision or pattern recognition perspective since (a) traditional techniques for detection of corners, denoting points of surface discontinuity, typically fail in such cases and, (b) one needs to implicitly handle unknown local degradation in the image. A novel two-phase scheme for hairline mandibular fracture detection, that is robust to noise, is proposed. In the first phase, the hairline fractures are coarsely localized using statistical correlation and by exploiting the bilateral symmetry of the human mandible. In the second phase, the fractures are precisely identified and highlighted using a Markov random field (MRF) modeling approach coupled with maximum a posteriori probability (MAP) estimation. Gibbs sampling is used to maximize the posterior probability. Experimental results on computer tomography (CT) scans from real patients are presented

A novel procedure for in-silico (virtual) craniofacial reconstruction of human mandibles with multiple fractures from a sequence of Computed Tomography (CT) images is presented. The problem is formulated as one of combinatorial pattern matching and solved in two stages. First, the opposable fracture surfaces are identified using a maximum weight graph matching algorithm where the fracture surfaces are modeled as the vertices of a weighted graph. The edge weights between pairs of vertices are treated as elements of a score matrix, whose values are a linear combination of (a) the Hausdorff distance, and (b) a score function based on fracture surface characteristics. Second, the pairs of opposable fracture surfaces identified in the first stage are actually registered using the Iterative Closest Point (ICP) algorithm enhanced with a graph theoretic improvisation. The correctness of the registration in the second stage is constantly monitored by volumetric matching of the reconstructed mandible with an intact mandible. Experimental results on simulated CT image sequences of broken human mandibles are presented.