In this research, we deal with the problem of visual question answering (VQA) in remote sensing. While remotely sensed images contain information significant for the task of identification and object detection, they pose a great challenge in their processing because of high dimensionality, volume and redundancy. Furthermore, processing image information jointly with language features adds additional constraints, such as mapping the corresponding image and language features. To handle this problem, we propose a cross attention based approach combined with information maximization. The CNN-LSTM based cross-attention highlights the information in the image and language modalities and establishes a connection between the two, while information maximization learns a low dimensional bottleneck layer, that has all the relevant information required to carry out the VQA task. We evaluate our method on two VQA remote sensing datasets of different resolutions. For the high resolution dataset, we achieve an overall accuracy of 79.11% and 73.87% for the two test sets while for the low resolution dataset, we achieve an overall accuracy of 85.98%.

Many signal processing applications involve designing an all-pass filter with a desired phase response. Earlier methods have largely focused on approximating the desired response using an optimisation based approach, or by using numerical computations of the group delay at specific frequencies with Blaschke interpolation. The former method does not offer any guarantee to match the estimated phase response at given points, whereas the latter one matches the phases at the input points, but is sensitive to the numerical precision of the specified group delays. In this work, we present a unimodular interpolation-based method to obtain all-pass filter coefficients that match the desired phase response at the given points without the need for group delays at the interpolating points. Through detailed simulations, we show that the proposed method is more suited for obtaining all-pass filters that match the target phases when compared to earlier approaches.

End-to-end models for robust automatic speech recognition (ASR) have not been sufficiently well-explored in prior work. With end-to-end models, one could choose to preprocess the input speech using speech enhancement techniques and train the model using enhanced speech. Another alternative is to pass the noisy speech as input and modify the model architecture to adapt to noisy speech. A systematic comparison of these two approaches for end-to-end robust ASR has not been attempted before. We address this gap and present a detailed comparison of speech enhancement-based techniques and three different model-based adaptation techniques covering data augmentation, multi-task learning, and adversarial learning for robust ASR. While adversarial learning is the best-performing technique on certain noise types, it comes at the cost of degrading clean speech WER. On other relatively stationary noise types, a new speech enhancement technique outperformed all the model-based adaptation techniques. This suggests that knowledge of the underlying noise type can meaningfully inform the choice of adaptation technique.

In this paper, we propose a novel framework for multi-image co-segmentation using class agnostic meta-learning strategy by generalizing to new classes given only a small number of training samples for each new class. We have developed a novel encoder-decoder network termed as DVICE (Directed Variational Inference Cross Encoder), which learns a continuous embedding space to ensure better similarity learning. We employ a combination of the proposed DVICE network and a novel few-shot learning approach to tackle the small sample size problem encountered in co-segmentation with small datasets like iCoseg and MSRC. Furthermore, the proposed framework does not use any semantic class labels and is entirely class agnostic. Through exhaustive experimentation over multiple datasets using only a small volume of training data, we have demonstrated that our approach outperforms all existing state-of-the-art techniques.

Anomalous human activity detection is the task of identifying human activities that differ from the usual. Existing techniques, in general, try to deploy some samples from an open-set (anomalous activities can not be represented as a closed set) to define the discriminator. However, it is non-trivial to obtain novel activity instances. To this end, we propose PoseCVAE, a novel anomalous human activity detection strategy using the notion of generative modeling. We adopt a hybrid training strategy comprising of self-supervised and unsupervised learning. The self-supervised learning helps the encoder and decoder to learn better latent space representation of human pose trajectories. We train our framework to predict future pose trajectory given a normal track of past poses, i.e., the goal is to learn a conditional posterior distribution that represents normal training data. To achieve this we use a novel adaptation of a conditional variational autoencoder (CVAE) and refer it as PoseCVAE. Future pose prediction will be erroneous if the given poses are sampled from a distribution different from the learnt posterior, which is indeed the case with abnormal activities. To further separate the abnormal class, we imitate abnormal poses in the encoded space by sampling from a distinct mixture of Gaussians (MoG). We use a binary cross-entropy (BCE) loss as a novel addition to the standard CVAE loss function to achieve this. We test our framework on three publicly available datasets and achieve comparable performance to existing unsupervised methods that exploit pose information.

Spiking Neural Networks (SNN) are energy-efficient computing architectures that exchange spikes for processing information, unlike classical Artificial Neural Networks (ANN). Due to this, SNNs are better suited for real-life deployments. However, similar to ANNs, SNNs also benefit from deeper architectures to obtain improved performance. Furthermore, like the deep ANNs, the memory, compute and power requirements of SNNs also increase with model size, and model compression becomes a necessity. Knowledge distillation is a model compression technique that enables transferring the learning of a large machine learning model to a smaller model with minimal loss in performance. In this paper, we propose techniques for knowledge distillation in spiking neural networks for the task of image classification. We present ways to distill spikes from a larger SNN, also called the teacher network, to a smaller one, also called the student network, while minimally impacting the classification accuracy. We demonstrate the effectiveness of the proposed method with detailed experiments on three standard datasets while proposing novel distillation methodologies and loss functions. We also present a multi-stage knowledge distillation technique for SNNs using an intermediate network to obtain higher performance from the student network. Our approach is expected to open up new avenues for deploying high performing large SNN models on resource-constrained hardware platforms.

A classical approach to abnormal activity detection is to learn a representation for normal activities from the training data and then use this learned representation to detect abnormal activities while testing. Typically, the methods based on this approach operate at a fixed timescale - either a single time-instant (e.g. frame-based) or a constant time duration (e.g. video-clip based). But human abnormal activities can take place at different timescales. For example, jumping is a short-term anomaly and loitering is a long-term anomaly in a surveillance scenario. A single and pre-defined timescale is not enough to capture the wide range of anomalies occurring with different time duration. In this paper, we propose a multi-timescale model to capture the temporal dynamics at different timescales. In particular, the proposed model makes future and past predictions at different timescales for a given input pose trajectory. The model is multi-layered where intermediate layers are responsible to generate predictions corresponding to different timescales. These predictions are combined to detect abnormal activities. In addition, we also introduce a single-camera abnormal activity dataset for research use that contains 483,566 annotated frames. Our experiments show that the proposed model can capture the anomalies of different time duration and outperforms existing methods.

The objective in image co-segmentation is to jointly segment unknown common objects from a given set of images. In this paper, we propose a novel deep convolution neural network based end-to-end co-segmentation model. It is composed of a metric learning and decision network leading to a novel conditional siamese encoder-decoder network for estimating a co-segmentation mask. The role of the metric learning network is to find an optimum latent feature space where objects of the same class are closer and that of different classes are separated by a certain margin. Depending on the extracted features, the decision network decides whether input images have common objects or not and the encoder-decoder network produces a cosegmentation mask accordingly. Key aspects of the architecture are as follows. First, it is completely class agnostic and does not require any semantic information. Second, in addition to producing masks, the decoder network also learns similarity across image pairs that improves co-segmentation significantly. Experimental results reflect an excellent performance of our method compared to state of-the-art methods on challenging co-segmentation datasets.

In this paper, we formulate image co-segmentation as a classification problem in an unsupervised framework with the classes being the common foreground and the remaining regions in the image set. We first find a set of superpixels across all images with high feature similarity such that the constituent superpixels in individual images are spatially compact and label them as seed for the common foreground. Those superpixels with high background probability are labeled as respective seeds for multiple background classes. Seed computation here is unsupervised and automated unlike some semi-supervised methods. Then, we compute discriminative features that separate the initially labeled classes using linear discriminant analysis. We use these features to perform spatially constrained label propagation and obtain labels for the unlabeled regions, and iterate this process till the seed regions grow to the common object. Experimental results demonstrate excellent robustness properties even while processing non-homogeneous image sets where the common object is present only in majority of the images.

In this paper, various regularizations on the room impulse response (RIR) are proposed to obtain better single-channel speech dereverberation in the non-negative matrix factorization (NMF) framework. The regularizations on the RIR are motivated by the spectral domain representation of the RIR. To obtain better estimates of the RIR and clean speech, we propose three modifications (i) to obtain a sparse RIR (ii) a frequency envelop constrained RIR and (iii) to include the early part of the RIR. The performance of the proposed regularizers are evaluated by considering speech enhancement measures. While it is observed that the regularizers lead to an improved estimate of the RIR, they do not necessarily lead to speech enhancement in all the cases. For the experiments conducted using the RIRs from the REVERB 2014 challenge and sentences from the TIMIT database, the regularizer that includes the early part of the RIR shows reasonable improvement in speech enhancement measures.

Imaging techniques involve counting of photons striking a detector. Due to fluctuations in the counting process, the measured photon counts are known to be corrupted by Poisson noise. In this paper, we propose a blind dictionary learning framework for the reconstruction of photographic image data from Poisson corrupted measurements acquired by a compressive camera. We exploit the inherent non-negativity of the data by modeling the dictionary as well as the sparse dictionary coefficients as non-negative entities, and infer these directly from the compressed measurements in a Poisson maximum likelihood framework. We experimentally demonstrate the advantage of this in situ dictionary learning over commonly used sparsifying bases such as DCT or wavelets, especially on color images.

In this paper, we propose an unsupervised two-stage algorithm to extract salient objects from images. In the first stage, the image is segmented into superpixels that are grouped together through k-means clustering, based on histogram features of superpixels. The saliency of each cluster is calculated using inter-cluster and intra-cluster feature dissimilarities. In the second stage, we use seam carving to obtain an object level segmentation of the image in the form of a bounding box around the salient object. We propose an automated approach for seam carving based on a novel energy function obtained by combining the saliency output with a texture removed input image. We compute the optimal number of seams to be removed to extract the salient object instead of manually providing it. The performance of the proposed method is demonstrated by processing different types of images.

Visual tracking is a critical task in surveillance and activity analysis. One of the major issues in visual target tracking is variations in illumination. In this paper, we propose a novel algorithm based on discrete cosine transform (DCT) to handle illumination variations, since illumination variations are mainly reflected in the low-frequency band. For instance, low illumination in a frame leads to low value DC coefficient as vias versa. We modify DC coefficient to achieve illumination invariance. Average of DC coefficients of particular numbers of neighboring frames of current frame is taken. The correction in DC coefficient is performed using maximum eigen value of the image co-variance matrix over N frames. The videos with corrected illumination are than used to track objects of interest using the Mean shift algorithm. The proposed algorithm is tested on an exhaustive database. The results demonstrate significantly improved tracking.

In fluorescence microscopy, thickness of the specimen is significant compared to the depth of focus of a high numerical aperture (Na) lens of the imaging system. An aberration is introduced when there is a variation in refractive index in the specimen. Aberrations decrease the resolution and contrast of the image. Deconvolution is a method used to restore the image computationally, and for a successful deconvolution an accurate knowledge of the imaging system's point spread function (PSF) is a must. Here, we derive an appropriate model to obtain shift varying PSF for a thick specimen. Deconvolution results obtained for such a specimen are significantly better compared to those obtained using the standard Gibson and Lanni PSF model.

We propose a radiance domain denoising frame work for the high dynamic range (HDR) imaging problem. The proposed method uses a maximum aposteriori probability (MAP) based reconstruction of the HDR image with total variation (TV) as the prior to avoid unnecessary smoothing of the radiance field. To make the computation with TV prior efficient, we extend the majorize-minimize method of upper bounding the total variation by a quadratic function to our case which has a nonlinear term arising from the camera response function. A theoretical justification for doing radiance domain denoising as opposed to image domain denoising is also provided. Our method yields better results, with the edges well preserved and noise reduced considerably.

We propose a particle filter tracker to track multiple maneuvering targets using a batch of range measurements. The state update is formulated through a locally linear motion model and the observability of the state vector is proved using geometrical arguments. The data likelihood treats the range observations as an image using template models derived from the state update equation, and incorporates the possibility of missing data as well as spurious range observations. The particle filter handles multiple targets, using a partitioned state-vector approach. The filter proposal function uses a Gaussian approximation of the full-posterior to cope with target maneuvers for improved efficiency. By treating the range measurements as images and using smoothness constraints, the particle filter is able to avoid the data association problems. Computer simulations demonstrate the performance of the tracking algorithm