Neural Radiance Fields (NeRFs) have made significant advances in rendering novel photorealistic views for both static and dynamic scenes. However, most prior works assume ideal conditions of artifact-free visual inputs i.e., images and videos. In real scenarios, artifacts such as object motion blur, camera motion blur, or lens defocus blur are ubiquitous. Some recent studies have explored novel view synthesis using blurred input frames by examining either camera motion blur, defocus blur, or both. However, these studies are limited to static scenes. In this work, we enable NeRFs to deal with object motion blur whose local nature stems from the interplay between object velocity and camera exposure time. Often, the object motion is unknown and time varying, and this adds to the complexity of scene reconstruction. Sports videos are a prime example of how rapid object motion can significantly degrade video quality for static cameras by introducing motion blur. We present an approach for realizing motion blur-free novel views of dynamic scenes from input videos with object motion blur captured from static cameras spanning multiple poses. We propose a NeRF-based analytical framework that elegantly correlates object three-dimensional (3D) motion across views as well as time to the observed blurry videos. Our proposed method DynaMoDeNeRF (Dynamic Motion-aware Deblurring NeRF) is self-supervised and reconstructs the dynamic 3D scene, renders sharp novel views by blind deblurring, and recovers the underlying 3D motion and blur parameters. We provide comprehensive experimental analysis on synthetic and real data to validate our approach. To the best of our knowledge, this is the first work to address localized object motion blur in the NeRF domain. The dataset is available at: https://github.com/akumar005/DynaMoDe-NeRF

Underwater (UW) robotics applications require depth and restored images simultaneously in real-time, irrespective of whether the UW images are captured in good lighting conditions or not. Most of the UW image restoration and depth estimation methods have been devised for images under normal lighting. Consequently, they struggle to perform on poorly lit images. Even though artificial illumination can be used when there is insufficient ambient light, it can introduce non-uniform lighting artifacts in the restored images. Hence, the recovery of depth and restored images directly from Low-Light UW (LLUW) images is a critical requirement in marine applications. While a few works have attempted LLUW image restoration, there are no reported works on joint recovery of depth and clean image from LLUW images. We propose a Self-Supervised Low-Light Underwater Image and Depth recovery network (SelfLUID-Net) for joint estimation of depth and restored image in real-time from a single LLUW image. We have collected an Underwater Low-Light Stereo Video (ULVStereo) dataset which is the first-ever UW dataset with stereo pairs of low-light and normally-lit UW images. For the dual tasks of image and depth recovery from a LLUW image, we effectively utilize the stereo data from ULVStereo that provides cues for both depth and illumination-independent clean image. We harness a combination of the UW image formation process, the Retinex model, and constraints enforced by the scene geometry for our self-supervised training. To handle occlusions, we additionally utilize monocular frames from our video dataset and propose a masking scheme to prevent dynamic transients, that do not respect the underlying scene geometry, from misguiding the learning process. Evaluations on five LLUW datasets demonstrate the superiority and generalization ability of our proposed SelfLUID-Net over existing state-of-the-art methods. The dataset ULVStereo is available at https://github.com/nishavarghese15/ULVStereo.

In this paper, we address the weakly-supervised Audio-Visual Video Parsing (AVVP) problem, which aims at labeling events in a video as audible, visible, or both, and temporally localizing and classifying them into known categories. This is challenging since we only have access to video-level (weak) event labels when training but need to predict event labels at the segment (frame) level at test time. Recent methods employ multiple-instance learning (MIL) techniques that tend to focus solely on the most discriminative segments, resulting in frequent misclassifications. Our idea is to first construct several “prototype” features for each event class by clustering key segments identified for the event in the training data. We then assign pseudo labels to all training segments based on their feature similarities with these prototypes and retrain the model under weak and strong supervision. We facilitate this by structuring the feature space with contrastive learning using pseudo labels. Experiments show that we outperform existing methods for weakly-supervised AVVP. We also show that learning with weak and iteratively re-estimated pseudo labels can be interpreted as an expectation-maximization (EM) algorithm, providing further insight for our training procedure.

The success of Deep Generative Models at high-resolution image generation has led to their extensive utilization for style editing of real images. Most existing methods work on the principle of inverting real images onto their latent space, followed by determining controllable directions. Both inversion of real images and determination of controllable latent directions are computationally expensive operations. Moreover, the determination of controllable latent directions requires additional human supervision. This work aims to explore the efficacy of mask-guided feature modulation in the latent space of a Deep Generative Model as a solution to these bottlenecks. To this end, we present the SemanticStyle Autoencoder (SSAE), a deep Generative Autoencoder model that leverages semantic mask-guided latent space manipulation for highly localized photorealistic style editing of real images. We present qualitative and quantitative results for the same and their analysis. This work shall serve as a guiding primer for future work.

Semantic segmentation involves classifying each pixel into one of a pre-defined set of object/stuff classes. Such a fine-grained detection and localization of objects in the scene is challenging by itself. The complexity increases manifold in the presence of blur. With cameras becoming increasingly light-weight and compact, blur caused by motion during capture time has become unavoidable. Most research has focused on improving segmentation performance for sharp clean images and the few works that deal with degradations, consider motion-blur as one of many generic degradations. In this work, we focus exclusively on motion-blur and attempt to achieve robustness for semantic segmentation in its presence. Based on the observation that segmentation annotations can be used to generate synthetic space-variant blur, we propose a Class-Centric Motion-Blur Augmentation (CCMBA) strategy. Our approach involves randomly selecting a subset of semantic classes present in the image and using the segmentation map annotations to blur only the corresponding regions. This enables the network to simultaneously learn semantic segmentation for clean images, images with egomotion blur, as well as images with dynamic scene blur. We demonstrate the effectiveness of our approach for both CNN and Vision Transformer-based semantic segmentation networks on PASCAL VOC and Cityscapes datasets. We also illustrate the improved generalizability of our method to complex real-world blur by evaluating on the commonly used deblurring datasets GoPro and REDS.

In this paper, we address the problem of weakly supervised Audio-Visual Video Parsing (AVVP), where the goal is to temporally localize events that are audible or visible and simultaneously classify them into known event categories. This is a challenging task, as we only have access to the video-level event labels during training but need to predict event labels at the segment level during evaluation. Existing multiple-instance learning (MIL) based methods use a form of attentive pooling over segment-level predictions. These methods only optimize for a subset of most discriminative segments that satisfy the weak-supervision constraints, which miss identifying positive segments. To address this, we focus on improving the proportion of positive segments detected in a video. To this end, we model the number of positive segments in a video as a latent variable and show that it can be modeled as Poisson binomial distribution over segment-level predictions, which can be computed exactly. Given the absence of fine-grained supervision, we propose an Expectation-Maximization approach to learn the model parameters by maximizing the evidence lower bound (ELBO). We iteratively estimate the minimum positive segments in a video and refine them to capture more positive segments. We conducted extensive experiments on AVVP tasks to evaluate the effectiveness of our proposed approach, and the results clearly demonstrate that it increases the number of positive segments captured compared to existing methods. Additionally, our experiments on Temporal Action Localization (TAL) demonstrate the potential of our method for generalization to similar MIL tasks.

Underwater (UW) depth estimation and image restoration is a challenging task due to its fundamental ill-posedness and the unavailability of real large-scale UW-paired datasets. UW depth estimation has been attempted before by utilizing either the haze information present or the geometry cue from stereo images or the adjacent frames in a video. To obtain improved estimates of depth from a single UW image, we propose a deep learning (DL) method that utilizes both haze and geometry during training. By harnessing the physical model for UW image formation in conjunction with the view-synthesis constraint on neighboring frames in monocular videos, we perform disentanglement of the input image to also get an estimate of the scene radiance. The proposed method is completely self-supervised and simultaneously outputs the depth map and the restored image in real-time (55 fps). We call this first-ever Underwater Self-supervised deep learning network for simultaneous Recovery of Depth and Image as USe-ReDI-Net. To facilitate monocular self-supervision, we collected a Dataset of Real-world Underwater Videos of Artifacts (DRUVA) in shallow sea waters. DRUVA is the first UW video dataset that contains video sequences of 20 different submerged artifacts with almost full azimuthal coverage of each artifact. Extensive experiments on our DRUVA dataset and other UW datasets establish the superiority of our proposed USe-ReDI-Net over prior art for both UW depth and image recovery. The dataset DRUVA is available at https://github.com/nishavarghese15/DRUVA.

Supervised networks address the task of low-light enhancement using paired images. However, collecting a wide variety of low-light/clean paired images is tedious as the scene needs to remain static during imaging. In this paper, we propose an unsupervised low-light enhancement network using context-guided illumination-adaptive norm (CIN). Inspired by coarse to fine methods, we propose to address this task in two stages. In stage- I, a pixel amplifier module (PAM) is used to generate a coarse estimate with an overall improvement in visibility and aesthetic quality. Stage- II further enhances the saturated dark pixels and scene properties of the image using CIN. Different ablation studies show the importance of PAM and CIN in improving the visible quality of the image. Next, we propose a region-adaptive single input multiple output (SIMO) model that can generate multiple enhanced images from a single low-light image. The objective of SIMO is to let users choose the image of their liking from a pool of enhanced images. Human subjective analysis of SIMO results shows that the distribution of preferred images varies, endorsing the importance of SIMO-type models. Lastly, we propose a low-light road scene (LLRS) dataset having an unpaired collection of low-light and clean scenes. Unlike existing datasets, the clean and low-light scenes in LLRS are real and captured using fixed camera settings. Exhaustive comparisons on publicly available datasets, and the proposed dataset reveal that the results of our model outperform prior art quantitatively and qualitatively.

Camera gimbal systems pervade various applications such as navigation, target tracking, security and surveillance. The need for higher steering rate (rotation angle per second) of gimbal often results in motion blur in the captured video frames. Motion deblurring in real-time is difficult with existing blind restoration methods which incur large execution times while attempting to retrieve latent images from blurry inputs using high-dimensional optimization. On the other hand, deep learning methods for motion deblurring, though fast, do not generalize satisfactorily with domain shifts. In this work, we address the problem of real-time motion deblurring in infrared (IR) images captured by a real gimbal-based system. We propose two blur-kernel estimation methods and reveal howa prioriknowledge of the blur-kernel can be used in conjunction with non-blind deblurring methods to achieve real-time performance. We experimentally show that, in comparison to the state-of-the-art techniques in deblurring, our method is better-suited for practical gimbal-based imaging systems.

Satellite images are typically subject to multiple distortions. Different factors affect the quality of satellite images, including changes in atmosphere, surface reflectance, sun illumination, viewing geometries etc., limiting its application to downstream tasks. In supervised networks, the availability of paired datasets is a strong assumption. Consequently, many unsupervised algorithms have been proposed to address this problem. These methods synthetically generate a large dataset of degraded images using image formation models. A neural network is then trained with an adversarial loss to discriminate between images from distorted and clean domains. However, these methods yield suboptimal performance when tested on real images that do not necessarily conform to the generation mechanism. Also, they require a large amount of training data and are rendered unsuitable when only a few images are available. We propose a distortion disentanglement and knowledge distillation framework for satellite image restoration to address these important issues. Our algorithm requires only two images: the distorted satellite image to be restored and a reference image with similar semantics. Specifically, we first propose a mechanism to disentangle distortion. This enables us to generate images with varying degrees of distortion using the disentangled distortion and the reference image. We then propose the use of knowledge distillation to train a restoration network using the generated image pairs. As a final step, the distorted image is passed through the restoration network to get the final output. Ablation studies show that our proposed mechanism successfully disentangles distortion. Exhaustive experiments on different timestamps of Google-Earth images and publicly available datasets, LEVIR-CD and SZTAKI, show that our proposed mechanism can tackle a variety of distortions and outperforms existing state-of-the-art restoration methods visually as well as on quantitative metrics.

Video deblurring remains a challenging task due to the complexity of spatially and temporally varying blur. Most of the existing works depend on implicit or explicit alignment for temporal information fusion, which either increases the computational cost or results in suboptimal performance due to misalignment. In this work, we investigate two key factors responsible for deblurring quality: how to fuse spatio-temporal information and from where to collect it. We propose a factorized gated spatio-temporal attention module to perform non-local operations across space and time to fully utilize the available information without depending on alignment. First, we perform spatial aggregation followed by a temporal aggregation step. Next, we adaptively distribute the global spatio-temporal information to each pixel. It shows superior performance compared to existing non-local fusion techniques while being considerably more efficient. To complement the attention module, we propose a reinforcement learning-based framework for selecting keyframes from the neighborhood with the most complementary and useful information. Moreover, our adaptive approach can increase or decrease the frame usage at inference time, depending on the user’s need. Extensive experiments on multiple datasets demonstrate the superiority of our method.

We present a general learning-based solution for restoring images suffering from spatially-varying degradations. Prior approaches are typically degradation-specific and employ the same processing across different images and different pixels within. However, we hypothesize that such spatially rigid processing is suboptimal for simultaneously restoring the degraded pixels as well as reconstructing the clean regions of the image. To overcome this limitation, we propose SPAIR, a network design that harnesses distortion-localization information and dynamically adjusts computation to difficult regions in the image. SPAIR comprises of two components, (1) a localization network that identifies degraded pixels, and (2) a restoration network that exploits knowledge from the localization network in filter and feature domain to selectively and adaptively restore degraded pixels. Our key idea is to exploit the non-uniformity of heavy degradations in spatial-domain and suitably embed this knowledge within distortion-guided modules performing sparse normalization, feature extraction and attention. Our architecture is agnostic to physical formation model and generalizes across several types of spatially-varying degradations. We demonstrate the efficacy of SPAIR individually on four restoration tasks- removal of rain-streaks, raindrops, shadows and motion blur. Extensive qualitative and quantitative comparisons with prior art on 11 benchmark datasets demonstrate that our degradation-agnostic network design offers significant performance gains over state-of-the-art degradation-specific architectures. Code available at https://github.com/humananalysis/spatially-adaptive-image-restoration.

Videos with binaural audios provide immersive viewing experience by enabling 3D sound sensation. Recent works attempt to generate binaural audio in a multimodal learning framework using large quantities of videos with accompanying binaural audio. In contrast, we attempt a more challenging problem – synthesizing binaural audios for a video with monaural audio in a weakly semi-supervised setting. Our key idea is that any down-stream task that can be solved only using binaural audios can be used to provide proxy supervision for binaural audio generation, thereby reducing the reliance on explicit supervision. In this work, as a proxy-task for weak supervision, we use Sound Source Localization with only audio. We design a two-stage architecture called Localize-to-Binauralize Network (L2BNet). The first stage of L2BNet is a Stereo Generation (SG) network employed to generate two-stream audio from monaural audio using visual frame information as guidance. In the second stage, an Audio Localization (AL) network is designed to use the synthesized two-stream audio to localize sound sources in visual frames. The entire network is trained end-to-end so that the AL network provides necessary supervision for the SG network. We experimentally show that our weakly-supervised framework generates two-stream audio containing binaural cues. Through user study, we further validate that our proposed approach generates binaural-quality audio using as little as 10% of explicit binaural supervision data for the SG network.

Image inpainting methods have shown significant improvements by using deep neural networks recently. However, many of these techniques often create distorted structures or blurry inconsistent textures. The problem is rooted in the encoder layers’ ineffectiveness in building a complete and faithful embedding of the missing regions from scratch. Existing solutions like course-to-fine, progressive refinement, structural guidance, etc. suffer from huge computational overheads owing to multiple generator networks, limited ability of handcrafted features, and sub-optimal utilization of the information present in the ground truth. We propose a distillation-based approach for inpainting, where we provide direct feature level supervision while training. We deploy cross and self-distillation techniques and design a dedicated completion-block in encoder to produce more accurate encoding of the holes. Next, we demonstrate how an inpainting network’s attention module can improve by leveraging a distillation-based attention transfer technique and further enhance coherence by using a pixeladaptive global-local feature fusion. We conduct extensive evaluations on multiple datasets to validate our method. Along with achieving significant improvements over previous SOTA methods, the proposed approach’s effectiveness is also demonstrated through its ability to improve existing inpainting works.

Dual-lens (DL) cameras capture depth information, and hence enable several important vision applications. Most present-day DL cameras employ unconstrained settings in the two views in order to support extended functionalities. But a natural hindrance to their working is the ubiquitous motion blur encountered due to camera motion, object motion, or both. However, there exists not a single work for the prospective unconstrained DL cameras that addresses this problem (so called dynamic scene deblurring). Due to the unconstrained settings, degradations in the two views need not be the same, and consequently, naive deblurring approaches produce inconsistent left-right views and disrupt scene-consistent disparities. In this paper, we address this problem using Deep Learning and make three important contributions. First, we address the root cause of view-inconsistency in standard deblurring architectures using a Coherent Fusion Module. Second, we address an inherent problem in unconstrained DL deblurring that disrupts scene-consistent disparities by introducing a memory-efficient Adaptive Scale-space Approach. This signal processing formulation allows accommodation of different image-scales in the same network without increasing the number of parameters. Finally, we propose a module to address the Space-variant and Image-dependent nature of dynamic scene blur. We experimentally show that our proposed techniques have substantial practical merit.

In this paper, we address the problem of dynamic scene deblurring in the presence of motion blur. Restoration of images affected by severe blur necessitates a network design with a large receptive field, which existing networks attempt to achieve through simple increment in the number of generic convolution layers, kernel-size, or the scales at which the image is processed. However, these techniques ignore the non-uniform nature of blur, and they come at the expense of an increase in model size and inference time. We present a new architecture composed of region adaptive dense deformable modules that implicitly discover the spatially varying shifts responsible for non-uniform blur in the input image and learn to modulate the filters. This capability is complemented by a self-attentive module which captures non-local spatial relationships among the intermediate features and enhances the spatially varying processing capability. We incorporate these modules into a densely connected encoder-decoder design which utilizes pre-trained Densenet filters to further improve the performance. Our network facilitates interpretable modeling of the spatially-varying deblurring process while dispensing with multi-scale processing and large filters entirely. Extensive comparisons with prior art on benchmark dynamic scene deblurring datasets clearly demonstrate the superiority of the proposed networks via significant improvements in accuracy and speed, enabling almost real-time deblurring.

Dense video captioning is an extremely challenging task since an accurate and faithful description of events in a video requires a holistic knowledge of the video contents as well as contextual reasoning of individual events. Most existing approaches handle this problem by first proposing event boundaries from a video and then captioning on a subset of the proposals. Generation of dense temporal annotations and corresponding captions from long videos can be dramatically source consuming. In this paper, we focus on the task of generating a dense description of temporally untrimmed videos and aim to significantly reduce the computational cost by processing fewer frames while maintaining accuracy. Existing video captioning methods sample frames with a predefined frequency over the entire video or use all the frames. Instead, we propose a deep reinforcement-based approach which enables an agent to describe multiple events in a video by watching a portion of the frames. The agent needs to watch more frames when it is processing an informative part of the video, and skip frames when there is redundancy. The agent is trained using actor-critic algorithm, where the actor determines the frames to be watched from a video and the critic assesses the optimality of the decisions taken by the actor. Such an efficient frame selection simplifies the event proposal task considerably. This has the added effect of reducing the occurrence of unwanted proposals. The encoded state representation of the frame selection agent is further utilized for guiding event proposal and caption generation tasks. We also leverage the idea of knowledge distillation to improve the accuracy. We conduct extensive evaluations on ActivityNet captions dataset to validate our method.

This paper tackles the problem of motion deblurring of dynamic scenes. Although end-to-end fully convolutional designs have recently advanced the state-of-the-art in non-uniform motion deblurring, their performance-complexity trade-off is still sub-optimal. Existing approaches achieve a large receptive field by increasing the number of generic convolution layers and kernel-size, but this comesat the expense of of the increase in model size and inference speed. In this work, we propose an efficient pixel adaptive and feature attentive design for handling large blur variations across different spatial locations and process each test image adaptively. We also propose an effective content-aware global-local filtering module that significantly improves performance by considering not only global dependencies but also by dynamically exploiting neighboring pixel information. We use a patch-hierarchical attentive architecture composed of the above module that implicitly discovers the spatial variations in the blur present in the input image and in turn, performs local and global modulation of intermediate features. Extensive qualitative and quantitative comparisons with prior art on deblurring benchmarks demonstrate that our design offers significant improvements over the state-of-the-art in accuracy as well as speed.

Deep learning approaches in image processing predominantly resort to supervised learning. A majority of methods for image denoising are no exception to this rule and hence demand pairs of noisy and corresponding clean images. Only recently has there been the emergence of methods such as Noise2Void, where a deep neural network learns to denoise solely from noisy images. However, when clean images that do not directly correspond to any of the noisy images are actually available, there is room for improvement as these clean images contain useful information that fully unsupervised methods do not exploit. In this paper, we propose a method for image denoising in this setting. First, we use a flow-based generative model to learn a prior from clean images. We then use it to train a denoising network without the need for any clean targets. We demonstrate the efficacy of our method through extensive experiments and comparisons.

Atmospheric medium often constrains the visibility of outdoor scenes due to scattering of light rays. This causes attenuation in the irradiance reaching the imaging device along with an additive component to render a hazy effect in the image. The visibility is further reduced for poorly illuminated scenes. The attenuation becomes wavelength dependent in underwater scenario, causing undesired color cast along with hazy effect. In order to suppress the effect of different atmospheric/underwater conditions such as haze and to enhance the contrast of such images, we reformulate local haziness in a generalized manner. The parameters are estimated by harnessing the similarity of patches within a local neighborhood. Unlike existing methods, our approach is developed based on the assumption that for outdoor scenes the depth of patches changes gradually in a local neighborhood surrounding the patch. This change in depth can be approximated by patch similarity in that neighborhood. As the attenuation in irradiance of an image in presence of atmospheric medium relies on the depth of the scene, the coefficients related to the attenuation are estimated from the weights of patch similarity. The additive haze effect is deduced using non-local mean of the patch. Our experimental results demonstrate the effectiveness of our approach in reducing the haze component as well as in enhancing the image under different conditions of haze (daytime, nighttime, and underwater).

We present a solution for the goal of extracting a video from a single motion blurred image to sequentially reconstruct the clear views of a scene as beheld by the camera during the time of exposure. We first learn motion representation from sharp videos in an unsupervised manner through training of a convolutional recurrent video autoencoder network that performs a surrogate task of video reconstruction. Once trained, it is employed for guided training of a motion encoder for blurred images. This network extracts embedded motion information from the blurred image to generate a sharp video in conjunction with the trained recurrent video decoder. As an intermediate step, we also design an efficient architecture that enables real-time single image deblurring and outperforms competing methods across all factors: accuracy, speed, and compactness. Experiments on real scenes and standard datasets demonstrate the superiority of our framework over the state-of-the-art and its ability to generate a plausible sequence of temporally consistent sharp frames.

Convolutional Neural Networks (CNN) have recently advanced the state-of-the-art in generalized motion deblurring. Literature suggests that restoration of high-resolution blurred images requires a design with a large receptive field, which existing networks achieve by increasing the number of generic convolution layers, kernel-size, or the scales at which the image is processed. However, increasing the network capacity in this form comes with the burden of increased model size and lower speed. To resolve this, we propose a novel architecture composed of dynamic convolutional modules, namely feature transformation (FT) and spatial attention (SA). An FT module addresses the camera shifts responsible for the global blur in the input image, while a SA module addresses spatially varying blur due to dynamic objects and depth changes. Qualitative and quantitative comparisons on deblurring benchmarks demonstrate that our network outperforms prior art across factors of accuracy, compactness, and speed, enabling real-time deblurring.

The increasing popularity of computational light field (LF) cameras has necessitated the need for tackling motion blur which is a ubiquitous phenomenon in hand-held photography. The state-of-the-art method for blind deblurring of LFs of general 3D scenes is limited to handling only downsampled LF, both in spatial and angular resolution. This is due to the computational overhead involved in processing data-hungry full-resolution 4D LF altogether. Moreover, the method warrants high-end GPUs for optimization and is ineffective for wide-angle settings and irregular camera motion. In this paper, we introduce a new blind motion deblurring strategy for LFs which alleviates these limitations significantly. Our model achieves this by isolating 4D LF motion blur across the 2D subaperture images, thus paving the way for independent deblurring of these subaperture images. Furthermore, our model accommodates common camera motion parameterization across the subaperture images. Consequently, blind deblurring of any single subaperture image elegantly paves the way for cost-effective non-blind deblurring of the other subaperture images. Our approach is CPU-efficient computationally and can effectively deblur full-resolution LFs.

Blind motion deblurring methods are primarily responsible for recovering an accurate estimate of the blur kernel. Non-blind deblurring (NBD) methods, on the other hand, attempt to faithfully restore the original image, given the blur estimate. However, NBD is quite susceptible to errors in blur kernel. In this work, we present a convolutional neural network-based approach to handle kernel uncertainty in non-blind motion deblurring. We provide multiple latent image estimates corresponding to different prior strengths obtained from a given blurry observation in order to exploit the complementarity of these inputs for improved learning. To generalize the performance to tackle arbitrary kernel noise, we train our network with a large number of real and synthetic noisy blur kernels. Our network mitigates the effects of kernel noise so as to yield detail-preserving and artifact-free restoration. Our quantitative and qualitative evaluations on benchmark datasets demonstrate that the proposed method delivers state-of-the-art results. To further underscore the benefits that can be achieved from our network, we propose two adaptations of our method to improve kernel estimates, and image deblurring quality, respectively.

A vast majority of contemporary cameras employ rolling shutter (RS) mechanism to capture images. Due to the sequential mechanism, images acquired with a moving camera are subjected to rolling shutter effect which manifests as geometric distortions. In this work, we consider the specific scenario of a fast moving camera wherein the rolling shutter distortions not only are predominant but also become depth-dependent which in turn results in intra-frame occlusions. To this end, we develop a first-of-its-kind pipeline to recover the latent image of a 3D scene from a set of such RS distorted images. The proposed approach sequentially recovers both the camera motion and scene structure while accounting for RS and occlusion effects. Subsequently, we perform depth and occlusion-aware rectification of RS images to yield the desired latent image. Our experiments on synthetic and real image sequences reveal that the proposed approach achieves state-of-the-art results.

We address the problem of camera motion estimation from a single blurred image with the aid of deep convolutional neural networks. Unlike learning-based prior works that estimate a space-invariant blur kernel, we solve for the global camera motion which in turn represents the space-variant blur at each pixel. Leveraging the camera motion as well as the clean reference image during training, we resort to a semi -supervised training scheme that utilizes the strengths of both supervised and unsupervised learning to solve for the camera motion undergone by a space-variant blurred image. Finally, we show the effectiveness of such a motion estimation network with applications in space-variant deblurring and change detection.

This paper addresses the problem of obtaining a blur-based segmentation map from a single image affected by motion or defocus blur. Since traditional hand-designed priors have fundamental limitations, we utilise deep neural networks to learn features related to blur and enable a pixel-level blur classification. Our approach mitigates the ambiguities present in blur detection task by introducing joint learning of global context and local features into the framework. Specifically, we train two sub-networks to perform the task at global (image) and local (patch) levels. We aggregate the pixel-level probabilities estimated by two networks and feed them to a MRF based framework which returns a refined and dense segmentation-map of the image with respect to blur. We also demonstrate via both qualitative and quantitative evaluation, that our approach performs favorably against state-of-the-art blur detection or segmentation works, and show its utility to applications of automatic image matting and blur magnification.

Images captured from consumer cameras are often prone to camera shake resulting in motion blur. Effect of motion blur is more common in high dynamic range imaging applications where multiple images are captured over a wide range of exposure settings. In this paper, we propose a unified approach to perform high dynamic range super-resolution (HDR-SR) imaging from a sequence of low dynamic range and low-resolution motion-blurred images. While existing works on HDR-SR assume the availability of blur-free input images, we propose an approach which is designed to handle blurring effects caused by the camera motion. Our approach attempts to harness the complementarity present in terms of the sensor exposure and blur to yield a high-quality image which has both higher spatial resolution as well as dynamic range. Experiments on synthetic and real examples demonstrate that the proposed method delivers state-of-the-art results.

Inevitable camera motion during exposure does not augur well for free-hand photography. Distortions introduced in images can be of different types and mainly depend on the structure of the scene, the nature of camera motion, and the shutter mechanism of the camera. In this paper, we address the problem of registering images taken from global shutter and rolling shutter cameras and reveal the constraints on camera motion that admit registration, change detection, and rectification. Our analysis encompasses degradations arising from camera motion during exposure and differences in shutter mechanisms. We also investigate conditions under which camera motions causing distortions in reference and target image can be decoupled to yield the underlying latent image through RS rectification. We validate our approach using several synthetic and real examples.

Row-wise exposure delay present in CMOS cameras is responsible for skew and curvature distortions known as the rolling shutter (RS) effect while imaging under camera motion. Existing RS correction methods resort to using multiple images or tailor scene-specific correction schemes. We propose a convolutional neural network (CNN) architecture that automatically learns essential scene features from a single RS image to estimate the row-wise camera motion and undo RS distortions back to the time of first-row exposure. We employ long rectangular kernels to specifically learn the effects produced by the row-wise exposure. Experiments reveal that our proposed architecture performs better than the conventional CNN employing square kernels. Our single-image correction method fares well even operating in a frame-by-frame manner against video-based methods and performs better than scene-specific correction schemes even under challenging situations.

In this work, we investigate the relation between the edge profiles present in a motion blurred image and the underlying camera motion responsible for causing the motion blur. While related works on camera motion estimation (CME) rely on the strong assumption of space-invariant blur, we handle the challenging case of general camera motion. We first show how edge profiles alone can be harnessed to perform direct CME from a single observation. While it is routine for conventional methods to jointly estimate the latent image too through alternating minimization, our above scheme is best-suited when such a pursuit is either impractical or inefficacious. For applications that actually favor an alternating minimization strategy, the edge profiles can serve as a valuable cue. We incorporate a suitably derived constraint from edge profiles into an existing blind deblurring framework and demonstrate improved restoration performance. Experiments reveal that this approach yields state-of-the-art results for the blind deblurring problem.

In this paper, we investigate deep neural networks for blind motion deblurring. Instead of regressing for the motion blur kernel and performing non-blind deblurring outside of the network (as most methods do), we propose a compact and elegant end-to-end deblurring network. Inspired by the data-driven sparse-coding approaches that are capable of capturing linear dependencies in data, we generalize this notion by embedding non-linearities into the learning process. We propose a new architecture for blind motion deblurring that consists of an autoencoder that learns the data prior, and an adversarial network that attempts to generate and discriminate between clean and blurred features. Once the network is trained, the generator learns a blur-invariant data representation which when fed through the decoder results in the final deblurred output.

In this paper, we address the problem of registering a distorted image and a reference image of the same scene by estimating the camera motion that had caused the distortion. We simultaneously detect the regions of changes between the two images. We attend to the coalesced effect of rolling shutter and motion blur that occurs frequently in moving CMOS cameras. We first model a general image formation framework for a 3D scene following a layered approach in the presence of rolling shutter and motion blur. We then develop an algorithm which performs layered registration to detect changes. This algorithm includes an optimisation problem that leverages the sparsity of the camera trajectory in the pose space and the sparsity of changes in the spatial domain. We create a synthetic dataset for change detection in the presence of motion blur and rolling shutter effect covering different types of camera motion for both planar and 3D scenes. We compare our method with existing registration methods and also show several real examples captured with CMOS cameras.

We address the problem of estimating the latent high-resolution (HR) image of a 3D scene from a set of non-uniformly motion blurred low-resolution (LR) images captured in the burst mode using a hand-held camera. Existing blind super-resolution (SR) techniques that account for motion blur are restricted to fronto-parallel planar scenes. We initially develop an SR motion blur model to explain the image formation process in 3D scenes. We then use this model to solve for the three unknowns-the camera trajectories, the depth map of the scene, and the latent HR image. We first compute the global HR camera motion corresponding to each LR observation from patches lying on a reference depth layer in the input images. Using the estimated trajectories, we compute the latent HR image and the underlying depth map iteratively using an alternating minimization framework. Experiments on synthetic and real data reveal that our proposed method outperforms the state-of-the-art techniques by a significant margin.

The rule of perspectivity that 'straight-lines-mustremain-straight' is easily inflected in CMOS cameras by distortions introduced by motion. Lines can be rendered as curves due to the row-wise exposure mechanism known as rolling shutter (RS). We solve the problem of correcting distortions arising from handheld cameras due to RS effect from a single image free from motion blur with special relevance to urban scenes. We develop a procedure to extract prominent curves from the RS image since this is essential for deciphering the varying row-wise motion. We pose an optimization problem with line desirability costs based on straightness, angle, and length, to resolve the geometric ambiguities while estimating the camera motion based on a rotation-only model assuming known camera intrinsic matrix. Finally, we rectify the RS image based on the estimated camera trajectory using inverse mapping. We show rectification results for RS images captured using mobile phone cameras. We also compare our single image method against existing video and nonblind RS rectification methods that typically require multiple images.

Taking good photos in low-light conditions using a smart-phone camera is quite challenging. In this paper, we propose a method to produce a sharp well-exposed image under dim light scenarios using exposure bracketing. These images captured from a hand-held camera can be viewed as a set of blurred (high exposure) and noisy images (low exposure). We first describe an algorithm for estimating the sharp latent image and depth map from a noisy-blurred pair by employing a gradient prior derived from the noisy image. We also show how our approach can be extended for extracting the high dynamic range information embedded in differently exposed images. Our framework can comfortably account for mis-alignments, depth variations and space-variant blurring.

We propose a passive forgery detection technique for locating spliced regions in motion blurred images of 3D scenes. We consider general camera motion in hand-held cameras and utilize discrepancies in local motion blur patterns as a cue for splicing detection. We first devise an automatic and computationally efficient scheme to estimate the camera motion using only the blur kernels from authentic region. Next, we utilize the relationship among blur kernels, camera trajectory and local depth to predict a set of authentic blur kernels for any depth map to directly flag spliced regions.

Capturing multiple images using the burst mode of handheld cameras can be a boon to obtain a high resolution (HR) image by exploiting the subpixel motion among the captured images arising from handshake. However, the caveat with mobile phone cameras is that they produce rolling shutter (RS) distortions that must be accounted for in the super-resolution process. We propose a method in which we obtain an RS-free HR image using HR camera trajectory estimated by leveraging the intra- and inter-frame continuity of the camera motion. Experimental evaluations demonstrate that our approach can effectively recover a super-resolved image free from RS artifacts.

Classical multi-image super-resolution (SR) algorithms, designed for CCD cameras, assume that the motion among the images is global. But CMOS sensors that have increasingly started to replace their more expensive CCD counterparts in many applications do not respect this assumption if there is a motion of the camera relative to the scene during the exposure duration of an image because of the row-wise acquisition mechanism. In this paper, we study the hitherto unexplored topic of multi-image SR in CMOS cameras. We initially develop an SR observation model that accounts for the row-wise distortions called the "rolling shutter" (RS) effect observed in images captured using non-stationary CMOS cameras. We then propose a unified RS-SR framework to obtain an RS-free high-resolution image (and the row-wise motion) from distorted low-resolution images. We demonstrate the efficacy of the proposed scheme using synthetic data as well as real images captured using a hand-held CMOS camera. Quantitative and qualitative assessments reveal that our method significantly advances the state-of-the-art.

We propose a framework for robust estimation of normal of a planar scene from a single motion blurred observation. We first reveal how feature points can be extracted from blur kernels and matched to generate several point correspondences. Although these points correspond to different homographies, the fact that they conform to the same normal yields a rank-3 constraint which we harness within a hierarchical clustering framework to estimate the normal accurately.

We address the problem of restoring a static planar scene degraded by skewing effect when imaged through adynamic water surface. In particular, we investigate geometric distortions due to unidirectional cyclic waves and circular ripples,phenomena that are most prevalent in fluid flow. Although the camera and scene are stationary, light rays emanating from a scene undergo refraction at the fluid–air interface. This refraction effect is time varying for dynamic fluids and results in nonrigid distortions (skew) in the captured image. These distortions can be associated with motion blur depending on the exposure time of the camera. In the first part of this paper, we establish the condition under which the blur induced due to unidirectional cyclic waves can be treated as space invariant. We proceed to derive a mathematical model for blur formation and propose a restoration scheme using a single degraded observation. In the second part, we reveal how the blur induced by circular ripples(though space variant) can be modeled as uniform in the polar domain and develop a method for deskewing. The proposed methods are tested on synthetic as well as real examples.

In this paper, we extend traditional Shape from focus (SFF) to the underwater scenario. Specifically, we show how 3D shape of objects immersed in water can be extracted from images captured under an optical microscope. Traditional SFF employs telecentric optics to achieve geometric registration among the frames. We extend conventional SFF to underwater objects under the assumption of negligible scattering. We establish that the property of telecentricity holds for objects immersed in water provided the numerical aperture is small. By modeling geometrical distortions due to refraction effects on the water surface, we prove that the depth map obtained in the presence of water is a scaled version of the original depth map. We also reveal that this scale factor is directly related to the refractive index of water. We validate performance with real experiments.

We present a scheme for recovering the orientation of a planar scene from a single translation ally-motion blurred image. By leveraging the homography relationship among image coordinates of 3D points lying on a plane, and by exploiting natural correspondences among the extremities of the blur kernels derived from the motion blurred observation, the proposed method can accurately infer the normal of the planar surface. We validate our approach on synthetic as well as real planar scenes.

Temporal artifacts due to sequential acquisition of measurements in compressed sensing manifest differently from a conventional optical camera. We propose a framework for dynamic scenes to estimate the relative global motion between camera and scene from measurements acquired using a compressed sensing camera. We follow an adaptive block approach where the resolution of the estimated motion path depends on the motion trajectory. To underline the importance of the proposed motion estimation framework, we develop a face recognition algorithm in the compressive sensing domain by factoring in the time-varying nature of the acquisition process.

The extensive availability of sophisticated image editing tools has rendered it relatively easy to produce fake images. Image splicing is a form of tampering in which an original image is altered by copying a portion from a different source. Because the phenomenon of motion blur is a common occurrence in hand-held cameras, we propose a passive method to automatically detect image splicing using blur as a cue. Specifically, we address the scenario of a static scene in which the cause of blur is due to hand shake. Existing methods for dealing with this problem work only in the presence of uniform space-invariant blur. In contrast, our method can expose the presence of splicing by evaluating inconsistencies in motion blur even under space-variant blurring situations. We validate our method on several examples for different scene situations and camera motions of interest.

We address the problem of estimating the latent image of a static bilayer scene (consisting of a foreground and a background at different depths) from motion blurred observations captured with a handheld camera. The camera motion is considered to be composed of in-plane rotations and translations. Since the blur at an image location depends both on camera motion and depth, deblurring becomes a difficult task. We initially propose a method to estimate the transformation spread function (TSF) corresponding to one of the depth layers. The estimated TSF (which reveals the camera motion during exposure) is used to segment the scene into the foreground and background layers and determine the relative depth value. The deblurred image of the scene is finally estimated within a regularization framework by accounting for blur variations due to camera motion as well as depth.

In this paper, we develop a method for motion segmentation using blur kernels. A blur kernel represents the apparent motion undergone by a scene point in the image plane. When the relative motion between the camera and scene is not restricted to fronto-parallel translations, the shape of the blur kernels can vary across image points. For a dynamic scene, we effectively model motion blur using transformation spread functions (TSFs) which represent the relative motions. Given a set of blur kernels that are estimated at different points across an image, we develop a method to segment them according to their relative motion. We initially group the blur kernels based on their `compatibility'. We refine this initial segmentation by jointly estimating the TSF and removing the outliers.

We address the problem of automatically detecting occluded regions given a blurred/unblurred image pair of a scene taken from different viewpoints. The occlusion can be due to single or multiple objects. We present a unified framework for detecting occluder(s) that is reasonably robust to non-uniform motion blur as well as variations in camera pose (without the need for deblurring). We assume that the occluded pixels occupy only a relatively small area and that the camera motion trajectory is sparse in the camera motion space. We validate the performance of our algorithm with experiments on synthetic and real data.

Image tampering has become rampant in today's world due to availability of sophisticated image editing tools. In this paper, we deal with the problem of image splicing which is one form of tampering. We propose a passive method to detect the presence of splicing in a given image based on inconsistencies derived from motion blur. Both planar and 3D scenes are considered. The cause of blurring in the image is restricted to translation camera motion while the scene is assumed to be static. We validate our approach on synthetic as well as real examples.

Existing single image defogging techniques can restore contrast loss and yield a rough estimate of the depth map of a scene. The ubiquity of hand-held imaging devices has attracted considerable attention to motion blur but this has not been addressed in the context of images captured under foggy conditions. In this paper, we show how to restore foggy motion-blurred images using depth cues derived from fog itself. Initially, we address restoration of images blurred primarily due to in-plane translational camera motion. This is followed by a scheme for handling general camera motion blur with a projective blur model. We demonstrate that foggy road scene images can be segmented into road, left, right and sky planes, and that each of these planes can be deblurred individually.

Hand-held cameras inevitably result in blurred images caused by camera-shake, and even more so in high dynamic range imaging applications where multiple images are captured over a wide range of exposure settings. The degree of blurring depends on many factors such as exposure time, stability of the platform, and user experience. Camera shake involves not only translations but also rotations resulting in nonuniform blurring. In this paper, we develop a method that takes input non-uniformly blurred and differently exposed images to extract the deblurred, latent irradiance image. We use transformation spread function (TSF) to effectively model the blur caused by camera motion. We first estimate the TSFs of the blurred images from locally derived point spread functions by exploiting their linear relationship. The scene irradiance is then estimated by minimizing a suitably derived cost functional. Two important cases are investigated wherein 1) only the higher exposures are blurred and 2) all the captured frames are blurred.

In this correspondence, we address the task of recovering shape-from-focus (SFF) as a perceptual organization problem in 3-D. Using tensor voting, depth hypotheses from different focus operators are validated based on their likelihood to be part of a coherent 3-D surface, thereby exploiting scene geometry and focus information to generate reliable depth estimates. The proposed method is fast and yields significantly better results compared with existing SFF methods.

Space-variantly blurred images of a scene contain valuable depth information. In this paper, our objective is to recover the 3-D structure of a scene from motion blur/optical defocus. In the proposed approach, the difference of blur between two observations is used as a cue for recovering depth, within a recursive state estimation framework. For motion blur, we use an unblurred-blurred image pair. Since the relationship between the observation and the scale factor of the point spread function associated with the depth at a point is nonlinear, we propose and develop a formulation of unscented Kalman filter for depth estimation. There are no restrictions on the shape of the blur kernel. Furthermore, within the same formulation, we address a special and challenging scenario of depth from defocus with translational jitter. The effectiveness of our approach is evaluated on synthetic as well as real data, and its performance is also compared with contemporary techniques.

A new approach for image matting is proposed based on the Kalman filter, to extract the matte and original foreground, despite the presence of noise in the observed image. Different filter formulations with a discontinuity-adaptive Markov random field prior are proposed for handling additive white Gaussian noise and film-grain noise.

We propose a new method that extends the capability of shape-from-focus (SFF) to estimate the depth profile of 3-D objects in the presence of structure-dependent pixel motion. Existing SFF techniques work under the constraint that there is no parallax in the captured stack of frames. However, in off-the-shelf cameras, there can be appreciable pixel motion among the observations when there is relative motion between the object and the camera. In such a scenario, the depth estimates will be erroneous if the parallax effect is not factored in. Our degradation model accounts for pixel migration effects in the observations due to parallax resulting in a generalization of the SFF technique. We show that pixel motion and defocus blur therein are tightly coupled to the underlying shape of the 3-D object. Simultaneous reconstruction of the underlying 3-D structure and the all-in-focus image is carried out within an optimization framework using local image operations. The proposed method when tested on many examples, both synthetic and real, is very effective and delivers state-of-the-art performance.

Unrestricted camera motion and the ability to operate over a range of lens parameters are often desirable when using an off-the-shelf camera. Variations in intrinsic and extrinsic parameters induce defocus and pixel motion, both of which relate to scene structure. We propose a depth estimation approach by elegantly coupling the motion and defocus cues. We further advocate a natural extension of our framework for inpainting both depth and image, using the motion cue. Unlike traditional inpainting, our approach also considers defocus blur. This ensures that the image inpainting is coherent with respect to defocus. We use the belief propagation method in our estimation approach, which also handles occlusions and uses the color image segmentation cue.

This paper deals with the problem of estimating structure of 3D scenes and image transformations from observations that are blurred due to unconstrained camera motion. Initially, we consider a fronto-parallel planar scene and relate the reference image of the scene to its motion-blurred observation by finding the reference image transformations. The blur kernel at every image point can be determined from these transformations. For 3D scenes, the extent of blurring in the image is related to the camera motion as well as the scene structure. We propose a technique to estimate the scene depth with the knowledge of the estimated image transformations. The proposed method is validated by testing on real and synthetic experiments.

We propose a technique to in paint large missing regions in range images. Such a technique can be used to restore degraded/occluded range maps. It can also serve to reconstruct dense depth maps from sparse measurements which can speed up the acquisition. Our method uses the visual cue from segmentation of an intensity image registered to the range image. Our approach enforces that pixels in the same segment should have similar range. Our simple strategy involves plane-fitting and local medians over segments to compute local energies for labeling unknown pixels. Our results exhibit high quality in painting with very low errors.

In this paper, we propose a video matting method with simultaneous noise reduction based on the Unscented Kalman filter (UKF). This recursive approach extracts the alpha mattes and denoised foregrounds from noisy videos, in a unified framework. No assumptions are made about the type of motion of the camera or of the foreground object in the video. Moreover, user-specified trimaps are required only once every ten frames. In order to accurately extract information at the borders between the foreground and the background, we include a discontinuity-adaptive Markov random field (MRF) prior. It incorporates spatio-temporal information from the current and previous frame during estimation of the alpha matte as well as the foreground. Results are given on videos with real film-grain noise.

Under stereo settings, the twin problems of image superresolution (SR) and high-resolution (HR) depth estimation are intertwined. The subpixel registration information required for image superresolution is tightly coupled to the 3D structure. The effects of parallax and pixel averaging (inherent in the downsampling process) preclude a priori estimation of pixel motion for superresolution. These factors also compound the correspondence problem at low resolution (LR), which in turn affects the quality of the LR depth estimates. In this paper, we propose an integrated approach to estimate the HR depth and the SR image from multiple LR stereo observations. Our results demonstrate the efficacy of the proposed method in not only being able to bring out image details but also in enhancing the HR depth over its LR counterpart.

Given an off-the-shelf camera, one has the freedom to move the camera or play around with its intrinsic parameters such as zoom or aperture settings. We propose a framework for depth estimation from a set of calibrated images, captured under general camera motion and parameter variation. Our framework considers the practical trade-offs in a camera and hence essentially generalizes the more constrained areas such as lateral or axial stereo, shape from defocus/focus etc. We discuss practical issues where such an approach becomes important. We pose the problem in a MAP framework and compute the depth estimates efficiently using belief propagation (BP). We also incorporate the visibility consideration to handle occlusions. Moreover, we use the vital cue from color image segmentation to constrain the estimation process. Our results demonstrate the effectiveness of our approach to localize discontinuities and handle low-textured regions

Shape from focus (SFF) which uses a sequence of space-variantly defocused frames works under the constraint that there is ‘no magnification’ in the stack. In the presence of sensor damage and/or occlusions, there will be missing data in the observations and SFF cannot recover structure in those regions. In many applications, the capability of fillingin missing data is of critical importance. In this paper, we investigate the effect of motion parallax in SFF and demonstrate the interesting possibility of how it can be judiciously used to jointly inpaint image and depth profiles. When there is relative motion between the 3D specimen and the camera, by virtue of the inherent pixel motion in each of the frames, it is possible to obtain a focused image and depth map of the scene despite missing regions in the observations.

Traditional stereo algorithms estimate disparity at the same resolution as the observations. In this work we address the problem of estimating disparity and occlusion information at a higher resolution (HR). We draw on the image formation model from the motion super-resolution domain to relate HR disparity and the observations. This approach estimates both the HR disparity and HR intensity. We minimize a suitably constructed cost function using graph cuts and iterated conditional modes (ICM) for disparity and intensity, respectively.

We propose a new technique in which line segments and elliptical arcs are used as features for recognizing image patterns. By using this approach, the process of locating a model in a given image is efficient since the number of features to be compared is few. We propose distance measures to evaluate the similarity between the features of the model and that of the image. The model transformation parameters are found by searching the transformation space using cell decomposition.

We propose a recursive spatial-domain speckle reduction algorithm for synthetic aperture radar (SAR) imagery based on the unscented Kalman filter (UKF) with a discontinuity-adaptive Markov random field (DAMRF) prior. The capability of the UKF in handling speckle noise and the feature preservation ability of the DAMRF model are explored within a unified framework through importance sampling.

This correspondence proposes a recursive algorithm for noise reduction in synthetic aperture radar imagery. Excellent despeckling in conjunction with feature preservation is achieved by incorporating a discontinuity-adaptive Markov random field prior within the unscented Kalman filter framework through importance sampling. The performance of this method is demonstrated on both synthetic and real examples.

In the Shape from Focus (SFF) method, a sequence of images of a 3D object is captured for computing its depth profile. However, it is useful in several applications to also derive a high resolution focused image of the 3D object. Given the space-variantly blurred frames and the depth map, we propose a method to optimally estimate a high resolution image of the object within the SFF framework.

This paper presents a novel approach based on the unscented Kalman filter (UKF) for image estimation in film-grain noise. The image prior is modeled as non-Gaussian and is incorporated within the UKF frame work using importance sampling. A small carefully chosen deterministic set of sigma points is used to capture the prior and is propagated through film-grain nonlinearity to compute image statistics. Experimental results are given to demonstrate the efficacy of the proposed method.

Reconstruction-based super-resolution algorithms use either sub-pixel shifts or relative blur among low-resolution observations as a cue to obtain a high-resolution image. In this paper, we propose a super-resolution algorithm that exploits the information available in the low-resolution observations due to both sub-pixel shifts and relative blur to yield a better quality image. Performance analysis is carried out based on the Cramer-Rao lower bound. Several experimental results on synthetic and real images are given for validation.

This paper presents discontinuity adaptive image estimation within the Kalman filter framework by non-Gaussian modeling of the image prior. A generalized methodology is proposed for specifying state-dynamics using the conditional density of the state given its neighbors, without explicitly defining the state equation. The novelty of our approach lies in directly obtaining the predicted mean and variance of the non-Gaussian state conditional density by importance sampling and incorporating them in the update step of the Kalman filter. Experimental results are given to demonstrate the effectiveness of the proposed method in preserving edges.

The shape-from-focus (SFF) method uses a sequence of frames to estimate the structure of a 3-D object. Its accuracy depends on the step size by which the translational table is moved while capturing the images. Existing SFF algorithms use an ad hoc interpolation strategy to account for the error due to the finite step size. We propose an improved SFF method that uses relative defocus blur derived from actual image data to arrive at the final estimates of the structure of the object. A space-variant image restoration scheme is also proposed to obtain a focused image of the 3-D object. The reconstructed 3-D structure as well as the quality of the restored image are superior for the proposed method in comparison to traditional SFF.

In this paper, a novel method to enhance license plate numbers of moving vehicles in real traffic videos is proposed. A high-resolution image of the number plate is obtained by fusing the information derived from multiple, subpixel shifted, and noisy low-resolution observations. The image to be superresolved is modeled as a Markov random field and is estimated from the observations by a graduated nonconvexity optimization procedure. A discontinuity adaptive regularizer is used to preserve the edges in the reconstructed number plate for improved readability. Experimental results are given on several traffic sequences to demonstrate the robustness of the proposed method to potential errors in motion and blur estimates. The method is computationally efficient as all operations can be implemented locally in the image domain

We present a learning-based method to super-resolve face images using a kernel principal component analysis-based prior model. A prior probability is formulated based on the energy lying outside the span of principal components identified in a higher-dimensional feature space. This is used to regularize the reconstruction of the high-resolution image. We demonstrate with experiments that including higher-order correlations results in significant improvements

A method based on the particle filter for recovering images degraded by film-grain noise is proposed. Due to the nonlinear relationship between the silver density and exposure, film-grain noise manifests itself as multiplicative non-Gaussian noise in the exposure domain. Since the posterior density is non-Gaussian, the proposed method works by representing it by a set of samples with associated weights. These samples are propagated in a recursive framework to obtain an optimal estimate of the original image. The effectiveness of the method is demonstrated with examples.

We propose a new method within the framework of principal component analysis (PCA) to robustly recognize faces in the presence of clutter. The traditional eigenface recognition (EFR) method, which is based on PCA, works quite well when the input test patterns are faces. However, when confronted with the more general task of recognizing faces appearing against a background, the performance of the EFR method can be quite poor. It may miss faces completely or may wrongly associate many of the background image patterns to faces in the training set. In order to improve performance in the presence of background, we argue in favor of learning the distribution of background patterns and show how this can be done for a given test image. An eigenbackground space is constructed corresponding to the given test image and this space in conjunction with the eigenface space is used to impart robustness. A suitable classifier is derived to distinguish nonface patterns from faces. When tested on images depicting face recognition in real situations against cluttered background, the performance of the proposed method is quite good with fewer false alarms.

We propose a method for estimating depth from images captured with a real aperture camera by fusing defocus and stereo cues. The idea is to use stereo-based constraints in conjunction with defocusing to obtain improved estimates of depth over those of stereo or defocus alone. The depth map as well as the original image of the scene are modeled as Markov random fields with a smoothness prior, and their estimates are obtained by minimizing a suitable energy function using simulated annealing. The main advantage of the proposed method, despite being computationally less efficient than the standard stereo or DFD method, is simultaneous recovery of depth as well as space-variant restoration of the original focused image of the scene.

We propose a view-based approach to recognize humans from their gait. Two different image features have been considered: the width of the outer contour of the binarized silhouette of the walking person and the entire binary silhouette itself. To obtain the observation vector from the image features, we employ two different methods. In the first method, referred to as the indirect approach, the high-dimensional image feature is transformed to a lower dimensional space by generating what we call the frame to exemplar (FED) distance. The FED vector captures both structural and dynamic traits of each individual. For compact and effective gait representation and recognition, the gait information in the FED vector sequences is captured in a hidden Markov model (HMM). In the second method, referred to as the direct approach, we work with the feature vector directly (as opposed to computing the FED) and train an HMM. We estimate the HMM parameters (specifically the observation probability B) based on the distance between the exemplars and the image features. In this way, we avoid learning high-dimensional probability density functions. The statistical nature of the HMM lends overall robustness to representation and recognition. The performance of the methods is illustrated using several databases.

We present a scheme for vehicle detection and tracking in video. The proposed method effectively combines statistical knowledge about the class of vehicles with motion information. The unknown distribution of the image patterns of vehicles is approximately modeled using higher-order statistical information derived from sample images. Statistical information about the background is learnt "on the fly". A motion detector identifies regions of activity. The classifier uses a higher-order statistical closeness measure to determine which of the objects actually correspond to moving vehicles. The tracking module uses position co-ordinates and difference measurement values for correspondence. Results on real video sequences are given.

We describe a higher-order spectral analysis-based approach for detecting people by recognizing human motion such as walking or running. The periodic attribute of human motion lends itself to efficient spectral inspection. In the proposed method, the stride length is determined in every frame as the image sequence evolves. The bispectrum which is the Fourier transform of the triple correlation is a robust indicator of presence of periodicity. Triple correlation is robust as it is immune to any symmetrically distributed noise. The method is successfully tested on real video sequences.

We propose a method for simultaneous recovery of depth and restoration of scene intensity, given two defocused images of a scene. The space-variant blur parameter and the focused image of the scene are modeled as Markov random fields (MRFs). Line fields are included to preserve discontinuities. The joint posterior distribution of the blur parameter and the intensity process is examined for locality property and we derive an important result that the posterior is again Markov. The result enables us to obtain the maximum a posterior (MAP) estimates of the blur parameter and the focused image, within reasonable computational limits. The estimates of depth and the quality of the restored image are found to be quite good, even in the presence of discontinuities.

The paper describes a scheme for detecting vehicles in images. The proposed method approximately models the unknown distribution of the images of vehicles by learning higher order statistics (HOS) information of the 'vehicle class' from sample images. Given a test image, statistical information about the background is learnt 'on the fly'. An HOS-based decision measure then classifies test patterns as vehicles or otherwise. When tested on real images of aerial views of vehicular activity, the method gives good results even on complicated scenes. It does not require any a priori information about the site. However, it is amenable to augmentation with contextual information. The method can serve as an important step towards building an automated roadway monitoring system.

The paper describes a scheme for detecting and tracking people in images. The method effectively combines statistical information about the class of people with motion information for classification and tracking. In this scheme, the unknown distribution of the images of people is approximately modeled by learning higher order statistics (HOS) information of the "people class" from sample images. Given a test image, statistical information about the background is learnt dynamically. A motion detector identifies regions of activity in the image sequence. A classifier based on an HOS-based closeness measure then determines which of the moving objects actually correspond to people in motion. The tracking module uses position information and an HOS-based difference measurement vector to establish correspondence. When tested on real video data with a cluttered background, the performance of the method is found to be quite good. The method can also detect people in static imagery.

In this paper, we propose a MAP-Markov random field (MRF) based scheme for recovering the depth and the focused image of a scene from two defocused images. The space-variant blur parameter and the focused image of the scene are both modeled as MRFs and their MAP estimates are obtained using simulated annealing. The scheme is amenable to the incorporation of smoothness constraints on the spatial variations of the blur parameter as well as the scene intensity. It also allows for inclusion of line fields to preserve discontinuities. The performance of the proposed scheme is tested on synthetic as well as real data and the estimates of the depth are found to be better than that of the existing window-based depth from defocus technique. The quality of the space-variant restored image of the scene is quite good even under severe space-varying blurring conditions.

A MAP-MRF based scheme is proposed for simultaneous recovery of the depth and the focused image of a scene from two defocused images. The space-variant blur parameter and the focused image of the scene are both modeled as MRFs and their MAP estimates are obtained using simulated annealing. The performance of the proposed scheme is tested on synthetic as well as real data and the estimates of the depth are found to be better than that of existing window-based techniques.

Two new schemes are presented for finding human faces in a photograph. The first scheme approximates the unknown distributions of the face and the face-like manifolds wing higher order statistics (HOS). An HOS-based data clustering algorithm is also proposed. In the second scheme, the face to non-face and non-face to face transitions are learnt using a hidden Markov model (HMM). The HMM parameters are estimated corresponding to a given photograph and the faces are located by examining the optimal state sequence of the HMM. Experimental results are presented on the performance of both the schemes.

A maximum likelihood-based method is proposed for blur identification from multiple observations of a scene. When the relations among the blurring functions are known, the estimate of blur obtained using the proposed method is very good. Since direct computation of the likelihood function becomes difficult as the number of images increases, we propose an algorithm to compute the likelihood function recursively.

In the depth from defocus (DFD) method two defocused images of a scene are obtained by capturing the scene with different sets of camera parameters. An arbitrary selection of the camera settings can result in observed images whose relative blurring is insufficient to yield a good estimate of the depth. In this paper, we study the effect of the degree of relative blurring on the accuracy of the estimate of the depth by addressing the DFD problem in a maximum likelihood-based framework. We propose a criterion for optimal selection of camera parameters to obtain an improved estimate of the depth. The optimality criterion is based on the Cramer-Rao bound of the variance of the error in the estimate of blur. Simulations as well as experimental results on real images are presented for validation.

A limitation of the existing maximum likelihood (ML) based methods for blur identification is that the estimate of blur is poor when the blurring is severe. In this paper, we propose an ML-based method for blur identification from multiple observations of a scene. When the relations among the blurring functions of these observations are known, we show that the estimate of blur obtained by using the proposed method is very good. The improvement is particularly significant under severe blurring conditions. With an increase in the number of images, direct computation of the likelihood function, however, becomes difficult as it involves calculating the determinant and the inverse of the cross-correlation matrix. To tackle this problem, we propose an algorithm that computes the likelihood function recursively as more observations are added.

In this paper, we propose a regularized solution to the depth from defocus (DFD) problem using the space-frequency representation (SFR) framework. A smoothness constraint is imposed on the estimates of the blur parameter, and a variational approach to the DFD problem is developed. Among the numerous SFRs, we study the applicability of the complex spectrogram and the Wigner distribution, in particular, for depth recovery. The performance of the proposed variational method is tested on both synthetic and real images. The method yields good results, and the quality of the estimates is significantly better than that obtained without the smoothness constraint on the blur parameter.

The recovery of depth from defocus involves calculating the depth of various points in a scene by modeling the effect that the focal parameters of the camera have on images acquired with a small depth of field. We propose a method that, instead of analyzing an image region in isolation, uses a block shift-variant interactive blur model to account for the interaction among neighboring subimages. Simulation results are presented on the performance of the model.