Despite the recent progress in deep learning-based face image colorization techniques, there is still much room for improvement. One of the significant challenges is the bias toward specific skin color. Moreover, the conventional face colorization approaches aim to produce colored 2D face images, whereas the generation of colored 3D faces from monocular achromatic (gray-scale) images is beyond the scope of these methods despite having immense potential applications. To address these issues, we propose Self-Supervised COoperative COlorizaTion of Achromatic Faces (COCOTA) framework that contains chromatic and achromatic pipelines to jointly estimate the color and shape of 3D faces using monocular achromatic face images without inducing any specific color bias. On the challenging CelebA test dataset, COCOTA out-performs the current state-of-the-art method by a large margin (e.g., for 3D color-based error, a reduction from 5.12 ± 0.13 to 3.09 ± 0.08 leading to an improvement of 39.6%), demonstrating the effectiveness of the proposed method.

Recent 3D face reconstruction methods show encouraging results in retrieving 3D face shape and texture from monocular face images. However, these approaches pose several dependencies during testing, such as the requirement of facial landmark coordinates. Moreover, a large testing time presents a challenge for real-time applications. To address these issues, we propose REduced Dependency Fast UnsuperviSEd 3D Face Reconstruction (RED-FUSE) framework, which uses unprocessed face images to estimate reliable 3D face shape and texture, thus eliminating the need for prior land-mark knowledge, and considerable prediction time during testing. RED-FUSE outperforms the current state-of-the-art method on CelebA dataset e.g., for 3D shape and color-based errors, a reduction from 5.84 ± 0.16 to 3.14 ± 0.11 and from 3.50 ± 0.14 to 2.97 ± 0.09 is observed, leading to an improvement of 46.2% and 15.1%, respectively. In addition, the testing time reduces from 7.30 msec to 1.85 msec per face, showing the effectiveness of our method.

3D face reconstruction from a monocular face image is a mathematically ill-posed problem. Recently, we observed a surge of interest in deep learning-based approaches to address the issue. These methods possess extreme sensitivity towards occlusions. Thus, in this paper, we present a novel context-learning-based distillation approach to tackle the occlusions in the face images. Our training pipeline focuses on distilling the knowledge from a pre-trained occlusion-sensitive deep network. The proposed model learns the context of the target occluded face image. Hence our approach uses a weak model (unsuitable for occluded face images) to train a highly robust network towards partially and fully-occluded face images. We obtain a landmark accuracy of 0.77 against 5.84 of recent state-of-the-art-method for real-life challenging facial occlusions. Also, we propose a novel end-to-end training pipeline to reconstruct 3D faces from multiple variations of the target image per identity to emphasize the significance of visible facial features during learning. For this purpose, we leverage a novel composite multi-occlusion loss function. Our multi-occlusion per identity model shows a dip in the landmark error by a large margin of 6.67 in comparison to a recent state-of-the-art method. We deploy the occluded variations of the CelebA validation dataset and AFLW2000-3D face dataset: naturally-occluded and artificially occluded, for the comparisons. We comprehensively compare our results with the other approaches concerning the accuracy of the reconstructed 3D face mesh for occluded face images.

There have been a number of techniques that have demonstrated the generation of multimedia data for one modality at a time using GANs, such as the ability to generate images, videos, and audio. However, so far, the task of multi-modal generation of data, specifically for audio and videos both, has not been sufficiently well-explored. Towards this, we propose a method that demonstrates that we are able to generate naturalistic samples of video and audio data by the joint correlated generation of audio and video modalities. The proposed method uses multiple discriminators to ensure that the audio, video, and the joint output are also indistinguishable from real-world samples. We present a dataset for this task and show that we are able to generate realistic samples. This method is validated using various standard metrics such as Inception Score, Frechet Inception Distance (FID) and through human evaluation.

One of the major limitations of deep learning models is that they face catastrophic forgetting in an incremental learning scenario. There have been several approaches proposed to tackle the problem of incremental learning. Most of these methods are based on knowledge distillation and do not adequately utilize the information provided by older task models, such as uncertainty estimation in pre-dictions. The predictive uncertainty provides the distributional information can be applied to mitigate catastrophic forgetting in a deep learning framework. In the proposed work, we consider a Bayesian formulation to obtain the data and model uncertainties. We also incorporate self-attention framework to address the incremental learning problem. We define distillation losses in terms of aleatoric uncertainty and self-attention. In the proposed work, we investigate different ablation analyses on these losses. Furthermore, we are able to obtain better results in terms of accuracy on standard benchmarks.

Unsupervised Domain adaptation methods solve the adaptation problem for an unlabeled target set, assuming that the source dataset is available with all labels. However, the availability of actual source samples is not always possible in practical cases. It could be due to memory constraints, privacy concerns, and challenges in sharing data. This practical scenario creates a bottleneck in the domain adaptation problem. This paper addresses this challenging scenario by proposing a domain adaptation technique that does not need any source data. Instead of the source data, we are only provided with a classifier that is trained on the source data. Our proposed approach is based on a generative framework, where the trained classifier is used for generating samples from the source classes. We learn the joint distribution of data by using the energy-based modeling of the trained classifier. At the same time, a new classifier is also adapted for the target domain. We perform various ablation analysis under different experimental setups and demonstrate that the proposed approach achieves better results than the baseline models in this extremely novel scenario.

In image processing, it is well known that mean square error criteria is perceptually inadequate. Consequently, image quality assessment (IQA) has emerged as a new branch to overcome this issue, and this has led to the discovery of one of the most popular perceptual measures, namely, the structural similarity index (SSIM). This measure is mathematically simple, yet powerful enough to express the quality of an image. Therefore, it is natural to deploy SSIM in model based applications, such as denoising, restoration, classification, etc. However, the non-convex nature of this measure makes this task difficult. Our attempt in this work is to discuss problems associated with its convex program and take remedial action in the process of obtaining a generalized convex framework. The obtained framework has been seen as a component of an alternative learning scheme for the case of a regularized linear model. Subsequently, we develop a relevant dictionary learning module as a part of alternative learning. This alternative learning scheme with sparsity prior is finally used in denoising and deblurring applications. To further boost the performance, an iterative scheme is developed based on the statistical nature of added noise. Experiments on image denoising and deblurring validate the effectiveness of the proposed scheme. Furthermore, it has been shown that the proposed framework achieves highly competitive performance with respect to other schemes in literature and performs better in natural images in terms of SSIM and visual inspection.

Conventionally, the point spread function (PSF) is understood as a characteristic function of any optical system. It captures the information about the amount of blur present along all the directions for a point in the scene. However, the dependence of blur on the PSF is in the form of convolution for any object other than a point source present in the scene and hence their relationship is less explicit. The authors propose a blur parameter locus curve (BPLC) as a system representation which has a one to one relationship with blur. BPLC simply is a chart of blur amounts in all directions of a given PSF with respect to the selected measurement function. They further characterise the PSF by decomposing the variation of BPLC across all directions based on the study performed for different possible forms of the blur kernels. Such decomposition provides powerful tools for various analysis. As PSF can be anisotropic, the computation of BPLC becomes an essential intermediate step to obtain the scale map as at the same scale, blur is different in different directions. Furthermore, they demonstrate the use of BPLC to obtain other system characteristics function such as PSF.

The occurrence of motion blur along with defocus blur is a common phenomena in natural images. Usually, these blurs are spatially varying in nature for any general image and estimation of one type of blur is affected by presence of other. In this paper, we propose a novel method to estimate the concurrent defocus and motion blurs in a single image. Unlike the recent methods, which perform well only on simulated conditions or in presence of single type of blur, proposed method works well for real images as well as for compressed images. In this paper, we consider only commonly associated motion and defocus blurs for analysis. Decoupling of motion and defocus blur provides a fundamental tool that can be used for various analysis and applications.

Dehazing is an important process as it can significantly improve the performance of computer vision applications in outdoor environments. The two main requirements that an online dehazing system demands are low processing time and high visual range. The authors present a novel dehazing algorithm based on colour uniformity principle (CUP) which meets the desired requirements of a realtime implementation. Estimation of atmospheric scattering parameter and transmission map forms the key step in dehazing problem. At first, the authors use CUP to generate the transmission map and refine it further by Fast Guided Filter. They estimate the atmospheric scattering parameter with the help of the estimated transmission map. Experimental results show that the quality of dehazed output, produced in real‐time using the proposed method, is comparable with the results achieved by the state of the art techniques. The proposed dehazing method produces reliable dehazed output in varying haze conditions, unlike current methods.

Significant recent developments in 3D display technology have focused on techniques for converting 2D media into 3D. Depth map is an integral part of 2D-to-3D conversion. Combining multiple depth cues results in a more accurate depth map as it compensates for the errors caused by one depth cue as well as its absence by other depth cues. In this paper, we present a novel framework to generate a more accurate depth map for video using defocus and motion cues. The moving objects present in the scene are the source of errors in both defocus and motion-based depth map estimation. The proposed method rectifies these errors in the depth map by integrating defocus blur and motion cues. In addition, it also corrects the errors in other parts of depth map caused by inaccurate estimation of defocus blur and motion. Since the proposed integration approach relies on the characteristics of point spread functions of defocus and motion blur along with their relations to camera parameters, it is more accurate and reliable.

Video summarization is an economical way of representing video contents and is useful for effective and quick browsing of the relevant activity present in the video. Most current video summarization approaches do not provide the correct insight of the events occurring in the video, such as moving the objects in a nonchronological order and tampering with the background and size of the objects. In this paper, we present an approach to create a video summarization that is a precise representation of the video content. First, our approach finds out the salient activities that are taking place in the video using an optimization framework for static and dynamic scenes. Second, the frames with the salient activities are stitched using alpha matting to form a single frame. Third, the summarized frame over multiple video shots obtained by our approach gives superior retrieval performance with image queries while reducing retrieval latency and memory requirement. This paper proposes a single frame-based indexing of the video database instead of multi-frame indexing as it is an interesting and potentially promising approach. Finally, several experiments are carried out to evaluate the proposed approach. The experiments demonstrate that the video summarization produced by our approach gives a realistic view of the events in the video, while the reduction in computational complexity and memory requirement is high.

This paper presents a novel automatic facial expressions recognition system (AFERS) using the deep network framework. The proposed AFERS consists of four steps: 1) geometric features extraction; 2) regional local binary pattern (LBP) features extraction; 3) fusion of both the features using autoencoders; and 4) classification using Kohonen self-organizing map (SOM)-based classifier. This paper makes three distinct contributions. The proposed deep network consisting of autoencoders and the SOM-based classifier is computationally more efficient and performance wise more accurate. The fusion of geometric features with LBP features using autoencoders provides better representation of facial expression. The SOM-based classifier proposed in this paper has been improved by making use of a soft-threshold logic and a better learning algorithm. The performance of the proposed approach is validated on two widely used databases (DBs): 1) MMI and 2) extended Cohn-Kanade (CK+). An average recognition accuracy of 97.55% in MMI DB and 98.95% in CK+ DB are obtained using the proposed algorithm. The recognition results obtained from fused features are found to be distinctly superior to both recognition using individual features as well as recognition with a direct concatenation of the individual feature vectors. Simulation results validate that the proposed AFERS is more efficient as compared to the existing approaches.

We propose a novel technique for detection of visual saliency in dynamic video based on video decomposition. The decomposition obtains the sparse features in a particular orientation by exploiting the spatiotemporal discontinuities present in a video cube. A weighted sum of the sparse features along three orthogonal directions determines the salient regions in the video cubes. The weights computed using the frame correlation along three directions are based on the characteristic of human visual system that identifies the sparsest feature as the most salient feature in a video. Unlike the existing methods, which detect the salient region as blob, the proposed approach detects the exact boundaries of salient region with minimum false detection. The experimental results confirm that the detected salient regions of a video closely resemble the salient regions detected by actual tracking of human eyes. The algorithm is tested on different types of video contents and compared with the several state-of-the-art methods to establish the effectiveness of the proposed method.

Roads are the vital mode of transportation for people and goods around the globe and its use has grown dramatically over the years. There is one death every four minutes due to road accidents in the developing nations. This is of deep concern to the entire humanity. Road accident detection and vehicle behavior analysis is of great interest to the research community in intelligent transportation systems. It is very difficult from the state of the art techniques to provide the abstract form of salient parts of accidents from road surveillance videos. To resolve these issues, we present perceptual video summarization techniques to enrich the speed of visualizing the accident content from a stack of videos. The problem of vehicle analysis is formulated as an optimization problem. To the best of our knowledge, this is the first time we solve an accident detection as an optimization problem and filter the frames to be selected, through a single formulation. With the camera in a surrounding infrastructure and capturing a video, we exploited the properties of sub modularity to provide a relevant and condensed key frame summary. We have studied it for various real world traffic surveillance videos comprising of vehicular accidents and thus making it a promising approach.

Real world scenes have huge intensity variations which are not in the control of the capture process. While human eye has an excellent dynamic range that enables us to visualize precise contrast variations and dynamically adapts to illumination variations, the dynamic range of conventional imaging devices is limited because of the physical constraints of the sensors. As a result of limited capabilities of the sensors, image saturation is observed often when lighting conditions are unfavorable (very bright, dark or uneven). In such scenarios, the captured image will have some optimally illuminated parts while some parts may undergo saturation (underexposure or overexposure). This makes the captured scene visually unappealing and the capture suffers from significant information loss. In this work, we propose an imaging solution to recover the scene information lost due to saturation, and hence, produce a better quality image ensuring no or minimal saturation.

The enormous growth of video content in recent times has raised the need to abbreviate the content for human consumption. Thus, there is a need for summaries of a quality that meets the requirements of human users. This also means that the summarization must incorporate the peculiar features of human perception. We present a new framework for video summarization in this paper. Unlike many available summarization algorithms that utilize only statistical redundancy, we introduce for the first time the features of the human visual system within the summarization framework itself to allow for the emphasis of perceptually significant events while simultaneously eliminating perceptual redundancy from the summaries. The subjective and objective evaluation scores have evaluated the framework.

The ability to follow a human is an important requirement for a service robot designed to work along side humans in homes or in work places. This paper describes the development and implementation of a novel robust visual controller for the human-following robot. This visual controller consists of two parts: 1) a robust algorithm that tracks a human visible in its camera view and 2) a servo controller that generates necessary motion commands so that the robot can follow the target human. The tracking algorithm uses point-based features, like speeded up robust feature, to detect human under challenging conditions, such as, variation in illumination, pose change, full or partial occlusion, and abrupt camera motion. The novel contributions in the tracking algorithm include the following: 1) a dynamic object model that evolves over time to deal with short-term changes, while maintaining stability over long run; 2) an online K-D tree-based classifier along with a Kalman filter is used to differentiate a case of pose change from a case of partial or full occlusion; and 3) a method is proposed to detect pose change due to out-of-plane rotations, which is a difficult problem that leads to frequent tracking failures in a human following robot. An improved version of a visual servo controller is proposed that uses feedback linearization to overcome the chattering phenomenon present in sliding mode-based controllers used previously. The efficacy of the proposed approach is demonstrated through various simulations and real-life experiments with an actual mobile robot platform.

Estimation of salient regions in an input video is an active area of research due to its wide applications. In this paper, we propose a novel algorithm to estimate the eye gaze movement in a video using motion, color and structural cues with minimum outliers. The algorithm is generalized to capture salient information for the videos taken under different camera motions. The entire algorithm is parallelizable and ensures faster estimation of salient regions. Using different standard datasets, the estimations of proposed algorithm are compared with state-of-the-art approaches. It is observed that the proposed method produces estimations closer to the ground-truth eye tracker data with minimum outliers.

Images captured in presence of fog, haze or snow usually suffer from poor contrast and visibility. In this paper we propose a novel dehazing method to increase visibility from a single view without using any prior knowledge about the outdoor scene. The proposed method estimates a visibility map of the scene from the input image and uses stochastic iterative algorithm to remove fog and haze. The method can be applied to color and grayscale images. Experimental results show that the proposed algorithm outperforms most of the state-of-the-art algorithms in terms of contrast, colorfulness and visibility.

Sparse representation based image and video processing have recently drawn much attention. Dictionary learning is an essential task in this framework. Our novel proposition involves direct computation of the dictionary by analyzing the distribution of training data in the metric space. The resulting representation is applied in the domain of grey scale image denoising. Denoising is one of the fundamental problems in image processing. Sparse representation deals efficiently with this problem. In this regard, dictionary learning from noisy images, improves denoising performance. Experimental results indicate that our proposed approach outperforms the ones using K-SVD for additive high-level Gaussian noise while for the medium range of noise level, our results are comparable.

In this paper, we present a new method for range data fusion from two heterogeneous range scanners for accurate surface modeling of rough and highly unstructured terrain. First, we present the segmentation of RGB-D images using the new framework of the GMM by employing the convex relaxation technique. After segmentation of RGB-D images, we transform both the range data to a common reference frame using PCA algorithm and apply the ICP algorithm to align both data in the reference frame. Based on a threshold criterion, we fuse the range data in such a way that the coarser regions are obtained from Kinect sensor and finer regions of plane are obtained from the Laser range sensor. After fusion, we apply Delaunay triangulation algorithm to generate the highly accurate surface model of the terrain. Finally, the experimental results show the robustness of the proposed approach.

This paper presents an appearance feature based facial expression recognition system using Kohonen Self-Organizing Map (KSOM). Appearance features are extracted using uniform Local binary patterns (LBPs) from equally sub-divided blocks applied over face image. The dimensionality of the LBP feature vector is further reduced using principal component analysis (PCA) to remove the redundant data that leads to unnecessary computation cost. Using our proposed KSOM based classification approach, we train only 59 dimensional LBP features extracted from whole facial region. The classifier is designed to categorize six basic facial expressions (happiness, sadness, disgust, anger, surprise and fear). To validate the performance of the reduced 59 dimensional LBP feature vector, we also train the original data of dimension 944 using the KSOM. The results demonstrates, that with marginal degradation in overall recognition performance, the reduced 59 dimensional data obtains very good classification results. The paper also presents three more comparative studies based on widely used classifiers like; Support vector machine (SVM), Radial basis functions network (RBFN) and Multi-layer perceptron (MLP3). Our KSOM based approach outperforms all other classification methods with average recognition accuracy 69.18%. Whereas, the average recognition rated obtained by SVM, RBFN and MLP3 are 65.78%, 68.09% and 62.73% respectively.

Occlusion is often thought of as a challenge for visual algorithms, specially tracking. Existing literature, however, has identified a number of occlusion categories in the context of tracking in ad hoc manner. We propose a systematic approach to formulate a set of occlusion cases by considering the spatial relations among object support(s) (projections on the image plane) with the detected foreground blob(s), to show that only 7 occlusion states are possible. We designate the resulting qualitative formalism as Oc-7, and show how these occlusion states can be detected and used effectively for the task of multi-object tracking under occlusion of various types. The object support is decomposed into overlapping patches which are tracked independently on the occurrence of occlusions. As a demonstration of the application of these occlusion states, we propose a reasoning scheme for selective tracker execution and object feature updates to track multiple objects in complex environments.

A novel approach to real-time tracking of three-finger planar grasp points for deforming objects is proposed. The search space of possible grasping configurations is reduced in two stages - firstly, by fixing one finger at the boundary point nearest to the object centroid and secondly, through a heuristic partitioning of the object boundary where the remaining two fingers are localized. The potential grasping configurations satisfying force closure conditions are evaluated through an objective function that maximizes the grasping span while minimizing the distance between the object centroid and the intersection of the contour normals at the finger contact points. A population based stochastic search strategy is adopted for computing the optimal grasping configurations and re-localizing them as the shape undergoes drastic translations, rotations, scaling and local deformations. Experimental results of grasp point tracking are presented for deforming planar shapes extracted from both real and synthetic image sequences. The current implementation of the proposed scheme operates at 10 Hz for grasp point tracking on shapes extracted through visual feedback