We present a novel video stabilization algorithm (LSstab) that removes unwanted motions in real-time. LSstab is based on a novel least squares formulation of the smoothing cost function to alleviate the undesirable camera jitter. A recursive least square solver is derived to minimize the smoothing cost function with an O(N) computation complexity. LSstab is evaluated using a suite of publicly available videos against the state of the art video stabilization methods. Results show LSstab reaches comparable or better performance, achieving real-time processing speed when a GPU is used.

We present a real-time, multi-view video stereopsis (RTMVS) algorithm. This algorithm processes five synchronized video streams from cameras of a stationary camera array using a commodity laptop computer equipped with an Nvidia GPU. It provides 3D visualization of a dynamic scene from a chosen viewpoint at the video frame rate. In RTMVS, 3D surfaces are represented as a set of triangles anchored on a sparse set of 3D feature points. The computationally intensive Structure-from-Motion (SfM) algorithm is executed as an initial step. Feature points in each video stream are tracked using a KLT tracker. Triangles will be updated only when at least one vertex moves from its current position. The algorithm will redetect features every X frame. Epipolar geometry and Trifocal tensor are also exploited to accelerate sparse feature point matching and track filtering. Compared to a dense point cloud multi-view stereopsis baseline algorithm, RTMVS reduces the processing time per frame from 30s down to less than 44 ms.

A robust multi-view disparity estimation algorithm for noisy images is presented. The proposed algorithm constructs 3D focus image stacks (3DFIS) by projecting and stacking multi-view images and estimates a disparity map based on the 3DFIS. To make the algorithm robust to noise and occlusion, a texture-based view selection and patch size variation scheme based on texture map is proposed. Experiment results indicate that the proposed algorithm outperforms conventional stereo matching algorithms as well as previously reported multi-view disparity estimation algorithms under noisy conditions.

A novel patch-based multi-view image denoising algorithm is proposed. This method leverages the 3D focus image stacks structure to exploit self-similarity and image redundancy inherent in multiple view images. Then a depth-guided adaptive window and dynamic view selection criterion is developed to aid proper selection of most consistent patches for the multi-view image denoising. Extensive experiments have been performed. Comparing the outcomes against those of state of the art image denoising algorithms, our proposed algorithm demonstrates significant performance advantage.

We propose a novel method for capturing high-speed, high dynamic range video with a single low-speed camera using a coded sampling technique. Traditional video cameras use a constant full-frame exposure time, which makes temporal super-resolution difficult due to the ill-posed nature of inverting the sampling operation. Our method samples at the same rate as the traditional low-speed camera but uses random per-pixel exposure times and offsets. By exploiting temporal and spatial redundancy in the video, we can reconstruct a high-speed video from the coded input. Furthermore, the different exposure times used in our sampling scheme enable us to obtain a higher dynamic range than a traditional camera or other temporal superresolution methods. We validate our approach using simulation and provide a detailed discussion on how to make a hardware implementation. In particular, we believe that our approach can maintain a 100% light throughput similarly to existing cameras and can be implemented on a single chip, making it suitable for small form factors.

This paper extends the classical warping-based optical flow method to achieve accurate flow in the presence of spatially-varying motion blur. Our idea is to parameterize the appearance of each frame as a function of both the pixel motion and the motion-induced blur. We search for the flows that best match two consecutive frames, which amounts to finding the derivative of a blurred frame with respect to both the motion and the blur, where the blur itself is a function of the motion. We propose an efficient technique to calculate the derivatives using prefiltering. Our technique avoids performing spatially-varying filtering (which can be computationally expensive) during the optimization iterations. In the end, our derivative calculation technique can be easily incorporated with classical flow code to handle video with non-uniform motion blur with little performance penalty. Our method is evaluated on both synthetic and real videos and outperforms conventional flow methods in the presence of motion blur.