In this paper, we investigate effective neural network layers for optical flow estimation and in particular for omni-directional optical flow. Optical flow has many applications in computer graphics, augmented reality and in 3D modeling. We create a simple dataset that enables us to efficiently assess the effectiveness of different neural network layers for optical flow. Based on this small-sized diagnostic dataset, FlowCLEVR, we conclude that a deformable convolution layer is highly effective in reducing motion and occlusion boundary blur. Based on these results, we are able to design modifications to various existing network architectures improving their performance. We demonstrate improved performance on FlowCLEVR, on standard datasets for optical flow in planar images and on a novel omni-directional optical flow dataset. We also extend our work to omni-directional stereo.

In this paper we present an approach for 6 DoF panoramic videos from omni-directional stereo (ODS) images using convolutional neural networks (CNNs). More specifically, we use CNNs to generate panoramic depth maps from ODS images in real-time. These depth maps would then allow for re-projection of panoramic images thus providing 6 DoF to a viewer in virtual reality (VR). As the boundaries of a panoramic image must touch in order to envelope a viewer, we introduce a border weighted loss function as well as new error metrics specifically tailored for panoramic images. We show experimentally that training with our border weighted loss function improves performance by benchmarking a baseline skip-connected encoder-decoder style network as well as other state-of-the-art methods in depth map estimation from mono and stereo images. Finally, a practical application for VR using real world data is also demonstrated.

The number of video surveillance cameras has increased by a large amount in recent years. There is therefore a need to process the captured videos such that human operators can quickly review the activities recorded by a camera over a long period of time. We propose in this paper an approach for producing video summaries, an abbreviated video preserving the important elements of interest. We introduce a dataset as well as three evaluation metrics for quantifying the performance of a video summary with respect to compression length, the amount of activity retained and the amount of activity that is packed into each frame of the summary.