Abstract High-spatio-temporal resolution (HSTR) video recording plays a crucial role in enhancing various imagery tasks that require fine-detailed information. State-of-the-art cameras provide this required high frame-rate and high spatial resolution together, albeit at a high cost. To alleviate this issue, this paper proposes a dual camera system for the generation of HSTR video using reference-based super-resolution (RefSR). One camera captures high spatial resolution low frame rate (HSLF) video while the other captures low spatial resolution high frame rate (LSHF) video simultaneously for the same scene. A novel deep learning architecture is proposed to fuse HSLF and LSHF video feeds and synthesize HSTR video frames. The proposed model combines optical flow estimation and (channel-wise and spatial) attention mechanisms to capture the fine motion and complex dependencies between frames of the two video feeds. Simulations show that the proposed model provides significant improvement over existing reference-based SR techniques in terms of PSNR and SSIM metrics. The method also exhibits sufficient frames per second (FPS) for aerial monitoring when deployed on a power-constrained drone equipped with dual cameras. The source code is publicly available at https://github.com/umutsuluhan/HSTRNet .

Wide area surveillance (WAS) requires high spatiotemporal resolution (HSTR) video for better precision. As an alternative to expensive WAS systems, low-cost hybrid imaging systems can be used. This paper presents the usage of multiple video feeds for the generation of HSTR video as an extension of reference based super resolution (RefSR). One feed captures video at high spatial resolution with low frame rate (HSLF) while the other captures low spatial resolution and high frame rate (LSHF) video simultaneously for the same scene. The main purpose is to create an HSTR video from the fusion of HSLF and LSHF videos. In this paper we propose an end-to-end trainable deep network that performs optical flow (OF) estimation and frame reconstruction by combining inputs from both video feeds. The proposed architecture provides significant improvement over existing video frame interpolation and RefSR techniques in terms of PSNR and SSIM metrics and can be deployed on drones with dual cameras.

It is expected that unmanned aerial vehicles (UAVs) will play a vital role in future communication systems. Optimum positioning of UAVs, serving as base stations, can be done through extensive field measurements or ray tracing simulations when the 3D model of the region of interest is available. In this paper, we present an alternative approach to optimize UAV base station altitude for a region. The approach is based on deep learning; specifically, a 2D satellite image of the target region is input to a deep neural network to predict path loss distributions for different UAV altitudes. The neural network is designed and trained to produce multiple path loss distributions in a single inference; thus, it is not necessary to train a separate network for each altitude.

Deep learning based methods have received a great deal of interest in recent years to solve the single image super-resolution (SISR) problem and their performance is proven to be superior when compared to classical SR techniques. Yet, most of these methods fail to generalize well on real life image datasets because they are trained on synthetic datasets with a small range of blur kernels. This makes data-driven approaches inherently weak when it comes to real images. Therefore, applying image super-resolution independently of the blur kernel is still a challenging task. In this paper we propose IKR-Net, Iterative Kernel Reconstruction network, for blind SISR. In the proposed approach, kernel estimation and high resolution image reconstruction are carried out iteratively using deep models. The iterative refinement provides significant improvement in both the reconstructed image and the estimated blur kernel. IKR-Net achieves state-of-the-art results in blind SISR, especially for images with motion blur.

Deep learning-based single image super-resolution (SR) consistently shows superior performance compared to the traditional SR methods. However, most of these methods assume that the blur kernel used to generate the low-resolution (LR) image is known and fixed (e.g. bicubic). Since blur kernels involved in real-life scenarios are complex and unknown, per-formance of these SR methods is greatly reduced for real blurry images. Reconstruction of high-resolution (HR) images from randomly blurred and noisy LR images remains a challenging task. Typical blind SR approaches involve two sequential stages: i) kernel estimation; ii) SR image reconstruction based on estimated kernel. However, due to the ill-posed nature of this problem, an iterative refinement could be beneficial for both kernel and SR image estimate. With this observation, in this paper, we propose an image SR method based on deep learning with iterative kernel estimation and image reconstruction. Simulation results show that the proposed method outperforms state-of-the-art in blind image SR and produces visually superior results as well.

In this paper, we propose a method to extract light field using a conventional camera from a single image capture. The method involves an offline calibration process, where point spread functions, relating different perspective images captured with a narrow aperture to a central image captured with a wide aperture, are estimated for different depths. During application, light field perspective images are recovered by de-convolving the input image with the set of point spread functions that were estimated in the offline calibration process.

Light field imaging extends the traditional photography by capturing both spatial and angular distribution of light, which enables new capabilities, including post-capture refocusing, post-capture aperture control, and depth estimation from a single shot. Micro-lens array (MLA) based light field cameras offer a cost-effective approach to capture light field. A major drawback of MLA based light field cameras is low spatial resolution, which is due to the fact that a single image sensor is shared to capture both spatial and angular information. In this paper, we present a learning based light field enhancement approach. Both spatial and angular resolution of captured light field is enhanced using convolutional neural networks. The proposed method is tested with real light field data captured with a Lytro light field camera, clearly demonstrating spatial and angular resolution improvement.

In this paper, we present a fast implementation of the bilateral filter with arbitrary range and domain kernels. It is based on the histogram-based fast bilateral filter approximation that uses uniform box as the domain kernel. Instead of using a single box kernel, multiple box kernels are used and optimally combined to approximate an arbitrary domain kernel. The method achieves better approximation of the bilateral filter compared to the single box kernel version with little increase in computational complexity. We also derive the optimal kernel size when a single box kernel is used.

In this paper, we present a fast implementation of the bilateral filter with arbitrary range and domain kernels. It is based on the fast bilateral filter approximation that uses uniform box domain kernel. Instead of using a single box kernel, multiple box kernels are used and combined optimally to approximate an arbitrary domain kernel. The method achieves better approximation of the bilateral filter compared to the single box kernel version with little increase in computational complexity.

This paper presents a method that estimates both the photometric mapping and the dense motion field in image sequences. The method extends the Horn-Schunck type dense variational optical flow estimation approach with the use of intensity mapping functions in an alternating optimization scheme. The intensity mapping functions are also updated through the iterations with the help of weighted histograms, where the weights reflect the visibility of pixels in one image from the other. We include experiments with both synthetic and real imagery to demonstrate the efficacy of the proposed method.

Spatial resolution of digital images are limited due to optical/sensor blurring and sensor site density. In single-chip digital cameras, the resolution is further degraded because such devices use a color filter array to capture only one spectral component at a pixel location. The process of estimating the missing two color values at each pixel location is known as demosaicking. Demosaicking methods usually exploit the correlation among color channels. When there are multiple images, it is possible not only to have better estimates of the missing color values but also to improve the spatial resolution further (using super-resolution reconstruction). In this paper, we propose a multi-frame spatial resolution enhancement algorithm based on the projections onto convex sets technique.

In this paper we propose a new method to reduce noise in digital images. The method is based on the bilateral filter. The bilateral filter is a nonlinear filter that does spatial averaging without smoothing edges. The spatial averaging aspect of the bilateral filter is very crucial; the bilateral filter has been shown to work better than wavelet thresholding in some recent papers. The proposed method improves the bilateral filter through decomposing a signal into its frequency components. In this way, noise in different frequency components can be eliminated. Experimental results with both simulated and real images are given. In addition to this new method, we also provide an empirical study of the optimal parameter selection for the bilateral filter.

Most interest point detection algorithms are highly sensitive to illumination variations. This paper presents a method to detect interest points robustly under large photometric changes. The method, which we call contrast invariant feature transform (CIFT), determines salient interest points in an image by calculating and processing contrast signatures. A contrast signature shows the response of an interest point detector with respect to a set of contrast stretching functions. The method is generic and can be used with most interest point detectors. In this paper, we demonstrate how CIFT improves the repeatability rate of the Harris corner detector.

The bilateral filter is a nonlinear filter that does spatial averaging without smoothing edges; it has shown to be an effective image denoising technique. An important issue with the application of the bilateral filter is the selection of the filter parameters, which affect the results significantly. There are two main contributions of this paper. The first contribution is an empirical study of the optimal bilateral filter parameter selection in image denoising applications. The second contribution is an extension of the bilateral filter: multiresolution bilateral filter, where bilateral filtering is applied to the approximation (low-frequency) subbands of a signal decomposed using a wavelet filter bank. The multiresolution bilateral filter is combined with wavelet thresholding to form a new image denoising framework, which turns out to be very effective in eliminating noise in real noisy images. Experimental results with both simulated and real data are provided.

Super resolution (SR) is a well-known technique to increase the quality of an image using multiple overlapping pictures of a scene. SR requires accurate registration of the images, both geometrically and photometrically. Most of the SR articles in the literature have considered geometric registration only, assuming that images are captured under the same photometric conditions. This is not necessarily true as external illumination conditions and/or camera parameters (such as exposure time, aperture size and white balancing) may vary for different input images. Therefore, photometric modeling is a necessary task for super resolution. In this paper, we investigate super-resolution image reconstruction when there is photometric variation among input images.

Finding geometric and photometric relation among images is crucial in many computer vision tasks such as panoramic imaging, high dynamic range imaging, stereo imaging, and change detection. Most photometric registration algorithms require accurate geometric registration of images. On the other hand, geometric registration may fail when images are not aligned photometrically. There are two contributions of this paper: (i) A contrast invariant feature detection algorithm is proposed. This would allow geometric registration of images without photometric registration, (ii) A photometric registration algorithm that can handle scene occlusions is presented.

Spatial resolution of digital images are limited due to optical/sensor blurring and sensor site density. In single-chip digital cameras, the resolution is further degraded because such devices use a color filter array to capture only one spectral component at a pixel location. The process of estimating the missing two color values at each pixel location is known as demosaicking. Demosaicking methods usually exploit the correlation among color channels. When there are multiple images, it is possible not only to have better estimates of the missing color values but also to improve the spatial resolution further (using super-resolution reconstruction). Previously, we have proposed a demosaicking algorithm based on the projection onto convex sets (POCS) technique. In this paper, we improve the results of that algorithm adding a new constraint set based on the spatio-intensity neighborhood. We extend the algorithm to image sequences for multi-frame demosaicking and super resolution.

In image retrieval applications one of the boundary-dependent approaches is matching contours with their polygonal representation. We introduce (1) a new polygonal shape representation, (2) an efficient algorithm to compute a unique representation of a polygon to handle orientation and (3) a matching method that is invariant to rigid and affine transformation. In the method, polygons are represented by a sequence of distance vectors ordered in a predefined cyclic way. Each vector is composed of two primitives which are radial distance from the centroid to a vertex and the following edge distance in a specified direction. Matching of polygons is achieved by bitwise comparison of their string code. The algorithm has a computational complexity of O(n log n), hence it has advantage for practical use.

Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods assume that exposure time is fixed for all observations, which is not necessarily true. In reality, cameras have limited dynamic range and nonlinear response to the quantity of light received, and exposure time might be adjusted automatically or manually to capture the desired portion of the scene's dynamic range. In this letter, we propose a Bayesian super-resolution algorithm based on an imaging model that includes camera response function, exposure time, sensor noise, and quantization error in addition to spatial blurring and sampling.

The process of reconstructing a high-resolution image from multiple low-resolution images is known as super-resolution reconstruction. One of the assumptions of current super-resolution reconstruction methods is that all images are captured with the same exposure time and aperture size. This is not necessarily true. In reality, cameras have limited dynamic range and a nonlinear response to the quantity of light received; also, camera settings might be adjusted automatically or manually to capture the desired portion of the scene's dynamic range. We propose a super-resolution algorithm based on an imaging model that includes a camera response function, exposure time, sensor noise and quantization error in addition to spatial blurring and sampling. The algorithm is based on Bayesian estimation. Initial experiments demonstrate the effectiveness of the algorithm.

Image registration is a necessary step in a variety of computer vision applications. One of the recent focus areas in image registration is extracting and matching features that are invariant to affine transformation. This is critical in various applications, including 3D reconstruction and object recognition. In this paper, we present a feature-based image registration method that is robust to scaling and rotation. This is achieved by extracting and matching features in the log-polar domain, where rotation and scale correspond to translation. Registration parameters are then estimated by applying the RANSAC technique to the feature correspondences. The RANSAC technique provides a robust estimation even when there are moving objects within the scene. Experimental results with synthetic and real images are provided.

Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods neglect camera response function, exposure time, white balancing, and external illumination changes. In this paper, we show how to extend traditional super-resolution reconstruction methods to handle these factors. We provide formulations for several different super-resolution reconstruction approaches. Experimental results are also included.

Considerable attention has been directed to the problem of producing high-resolution video and still images from multiple low-resolution images. This multiframe reconstruction, also known as super-resolution reconstruction, is beginning to be applied to compressed video. Super-resolution techniques that have been designed for raw (i.e., uncompressed) video may not be effective when applied to compressed video because they do not incorporate the compression process into their models. The compression process introduces quantization error, which is the dominant source of error in some cases. In this paper, we propose a stochastic framework where quantization information as well as other statistical information about additive noise and image prior can be utilized effectively.

An imaging system produces a degraded measurement of a real-valued quantity that varies temporally, spectrally, and spatially. When there are multiple measurements of the same scene, it is possible to combine the nonredundant information in those measurements and produce an improved image that has more information than any of the measurements does alone. This type of reconstruction is known as image fusion. Depending on the diversity of information in the measurements, it is possible to achieve temporal, spectral, spatial, and gray-scale improvements with an image fusion algorithm. In this paper, we are proposing an image fusion algorithm that produces an image of higher spatial and gray-scale information from multiple measurements. The algorithm estimates the spatial and illumination correlation between multiple measurements, and employs a set-theoretic reconstruction technique.

Face images that are captured by surveillance cameras usually have a very low resolution, which significantly limits the performance of face recognition systems. In the past, super-resolution techniques have been proposed to increase the resolution by combining information from multiple images. These techniques use super-resolution as a preprocessing step to obtain a high-resolution image that is later passed to a face recognition system. Considering that most state-of-the-art face recognition systems use an initial dimensionality reduction method, we propose to transfer the super-resolution reconstruction from pixel domain to a lower dimensional face space. Such an approach has the advantage of a significant decrease in the computational complexity of the super-resolution reconstruction. The reconstruction algorithm no longer tries to obtain a visually improved high-quality image, but instead constructs the information required by the recognition system directly in the low dimensional domain without any unnecessary overhead. In addition, we show that face-space super-resolution is more robust to registration errors and noise than pixel-domain super-resolution because of the addition of model-based constraints.

Most commercial digital cameras have three types of color sensors (for red, green, and blue channels) that are placed on a detector surface according to a specific pattern. At the location of each pixel only one color sample is taken, and the values of the other colors must be interpolated using neighboring samples. This color plane interpolation is known as demosaicing; it is one of the important tasks in a digital camera pipeline. If demosaicing is not performed appropriately, images suffer from highly visible color artifacts. In this paper we present a new demosaicing technique that uses inter-channel correlation effectively in an alternating-projections scheme. We have compared this technique with various state-of-the-art demosaicing techniques, and it outperforms all of them, both visually and in terms of mean square error.

Face images that are captured by surveillance cameras usually have a very low resolution, which significantly limits the performance of face recognition systems. In the past, super-resolution techniques have been proposed that attempt to increase the resolution by combining information from multiple images. These techniques use super-resolution as a preprocessing system to obtain a high resolution image that can later be passed to a face recognition system. Considering that most state-of-the-art face recognition systems use an initial dimensionality reduction method, we propose embedding the super-resolution algorithm into the face recognition system so that super-resolution is not performed in the pixel domain, but is instead performed in a reduced dimensional domain. The advantage of such an approach is a significant decrease in the computational complexity of the super-resolution algorithm because the algorithm no longer tries to construct a visually improved high quality image, but instead constructs the information required by the recognition algorithm directly in the lower dimensional domain without any unnecessary overhead.

Multiframe resolution enhancement ("superresolution") methods are becoming widely studied, but only a few procedures have been developed to work with compressed video, despite the fact that compression is a standard component of most image- and video-processing applications. One of these methods uses quantization-bound information to define convex sets and then employs a technique called "projections onto convex sets" (POCS) to estimate the original image. Another uses a discrete cosine transformation (DCT)-domain Bayesian estimator to enhance resolution in the presence of both quantization and additive noise. The latter approach is also capable of incorporating known source statistics and other reconstruction constraints to impose blocking artifact reduction and edge enhancement as part of the solution. We propose a spatial-domain Bayesian estimator that has advantages over both of these approaches.

A major drawback of block-based still-image or video-compression methods at low rates is the visible block boundaries that are also known as blocking artifacts. Several methods have been proposed in the literature to reduce these artifacts. Most are single image methods, which do not distinguish between video and still images. However, video has a temporal dimension that can lead to better reconstruction if utilized effectively. We show how to combine information from multiple frames to reduce blocking artifacts. We derive constraint sets using motion between neighboring frames and quantization information that is available in the video bit stream. These multiframe constraint sets can be used to reduce blocking artifacts in an alternating-projections scheme. They can also be included in existing set-theoretic algorithms to improve their performance by narrowing down the feasibility set. Experimental results show the effectiveness of using these multiframe constraint sets.

Most commercial digital cameras use color filter arrays to sample red, green, and blue colors according to a specific pattern. At the location of each pixel only one color sample is taken, and the values of the other colors must be interpolated using neighboring samples. This color plane interpolation is known as demosaicing; it is one of the important tasks in a digital camera pipeline. If demosaicing is not performed appropriately, images suffer from highly visible color artifacts. In this paper we present a new demosaicing technique that uses inter-channel correlation effectively in an alternating-projections scheme. We have compared this technique with six state-of-the-art demosaicing techniques, and it outperforms all of them, both visually and in terms of mean square error.

A major drawback of block-based still image or video compression methods at low rates are the visible block boundaries that are also known as blocking artifacts. Several methods have been proposed in the literature to reduce these artifacts for video sequences. However, most are simply adaptations of still image blocking artifact reduction methods, which do not exploit temporal information. We propose a novel multi-frame blocking artifact reduction method that incorporates temporal information effectively. This method uses the spatial correlations that exist between the successive frames to define constraint sets at multiple frames and provides a Projections Onto Convex Sets (POCS) solution. The proposed method operates solely on transform domain (DCT) data, and hence provides a solution that is compatible with the observed video. It does not need to make any spatial smoothness assumptions, which are typical with blocking artifact reduction algorithms for still images.

Resolution enhancement for video sequences has always been an attractive application in multimedia signal processing. "Superresolution" methods, that combine non-redundant information from a set of low-resolution images, are beginning to be applied to the most popular video compression standard, MPEG. Bayesian approaches, which are very successful for raw video, largely fail for MPEG video, since they do not incorporate the compression process into their models. This compression process introduces quantization noise, which is comparable to the additive noise that is used in the Bayesian models. We present an analytical derivation that combines the quantization and additive noises in a stochastic framework for MPEG-compressed video. This is a general framework in the sense that different video acquisition models, source statistics, implementation techniques can be used with it.

The number of bits assigned to represent the color intensity at image pixels is usually referred as the bit depth. When the bit depth is not sufficient, images suffer from ridge-like structures known as false contours. Bit-depth limitations become important when low-contrast details are required, as in medical imaging, aerial/satellite photography, and high-quality scanning: applications. In this paper, we investigate a method for increasing bit depth. Specifically, we show that when a sequence of video frames is available, then it is possible to achieve a higher bit depth through a projections onto convex sets (POCS) based reconstruction method.