In this paper, we propose a joint and unified framework to compressively capture a light field in the consideration of both directional and positional multiplexing based on Kronecker compressed sensing (KCS). First of all, both of the 2D angular and 2D spatial correlations of the light field can be fully utilized in the compressive acquisition, and the multiplexing is more flexible and balanced during the acquisition. Secondly, other types of light field acquisition can be unified into our proposed framework. In the experiment, it is shown that more balanced allocation between directional and positional multiplexing achieves better reconstruction quality of light field given the same number of total acquisitions. Furthermore, the experimental result also illustrates that the proposed method can capture a light field with full resolution and achieve better reconstruction quality than other previous methods.

The coded aperture/mask technique enables us to capture light field data in a compressive way through a single camera. A pixel value recorded by such a camera is a summation of the light rays that pass though different positions on the coded aperture/mask. The target light field can be reconstructed from the recorded pixel values by using prior information of the light field signal. As prior information, a dictionary (light field atoms), which was learned from training datasets, was used in the current state of the art. Meanwhile, it was reported that general bases such as DCT were not suitable to efficiently represent prior information. In this work, however, we demonstrate that a 4D-DCT basis works surprisingly better if it is combined with a weighting scheme in which the amplitude difference in DCT coefficients is considered. Simulation results using 18 light field datasets are reported to show the superior performance of the weighted 4D-DCT basis to the learned dictionary.

We propose a method for super-resolution image synthesis that accurately handles the physical pixel arrangement of a light field (plenoptic) camera. We use a Lytro camera to obtain 4D light field data (a set of multi-viewpoint images) through a micro-lens array. The light field data are multiplexed on a single image sensor, and thus, the data is first de-multiplexed into a set of multi-viewpoint (sub-aperture) images. However, the de-multiplexing process usually involves interpolation of the original data such as demosaicing for a color filter array and pixel resampling for the non-square micro-lens arrangement. During this interpolation, some information is added or lost to/from the original data. In contrast, our method can preserve the originally captured data as they are, and directly use them for the super-resolution image synthesis, where the super-resolved image and the corresponding depth map are alternatively refined. We experimentally demonstrate that our method can achieve higher image quality than that with a standard Light Field Toolbox.

In this paper, a new type of 3D display, called a layered light-field display, was investigated. By using only a few light-attenuating layers located in front of a backlight, this display can present many views in different directions simultaneously without sacrificing the resolutions of each view. The essential factor for efficient layer-based representation, which has not been deeply analyzed in previous works, is redundancy- namely, a low rank structure-of the light-field data. Accordingly, to reveal the origin of the redundancy, a generative model, in which a textured surface located at a certain depth generates a light field, was formulated and evaluated in this paper. Our theoretical analysis shows that the redundancy depends on not only the texture complexity but also the depth of the surface from the light attenuating layers. The theoretical model was validated through experimental simulation of the display.

Our goal is to propose and evaluate a new streaming data format that can be adapted to the limited bandwidth and capable of free-viewpoint video streaming using super multi-view video plus depth (MVD). Additionally, we proposed a view synthesis method for our data format. Given a requested free-viewpoint, we use the two closest views and corresponding depth maps to perform free-viewpoint video synthesis. The new data format consists of all views and corresponding depth maps in a lowered resolution, and the two closest views to the requested viewpoint in the high resolution. When the requested viewpoint changes, the two closest viewpoints will change, but one or both views are transmitted only in the low resolution during periods of large round-trip delay time. Therefore, the resolution compensation is required before view synthesis. Experimental results show that our proposed framework achieves view synthesis quality close to view synthesis using high resolution multi-view video plus depth.

The existing virtual view synthesis methods generate the images with many artifacts that are annoying, especially for forward virtual viewpoint, and virtual viewpoint generated by reference views with large baseline, due to occlusions and the limited sampling density. In this paper, we propose a new view synthesis method, robust to the above-mentioned problem, consist of three steps, using stereo contents. Firstly, view plus depth data of each viewpoint is 3D warped to the virtual viewpoint. We determine which neighboring pixels should be connected or kept isolated. Polygons enclosed by the connected pixels, i.e. superpixel, are interpolated. Secondly, we blend those warped images by comparing each pixel's depth value to obtain the virtual view, in which non-occlusion holes have already been interpolated by the process in the first step. Thirdly, the remaining holes are filled by inpainting. Our experimental results and comparisons show that the proposed view synthesis method allows smoother view reconstruction, while holes due to occlusion and 3D warping are filled with less artifacts.

SECOND-MVD (Synthesis Error COmpeNsateD Multiview Video plus Depth) is an alternative 3D format that we introduce for representation of multiview video. In this data format, images at some viewpoint remain original, and the others are converted to a novel format. Residual based representation, such as layered depth video and free-viewpoint TV data unit were proposed. We propose hybrid image that not only consists of residual but also remainder pixels. Generation and reconstruction process of a hybrid image uses virtual image synthesized by the images that remained original in SECOND-MVD. In this paper, we investigate the compression performance of SECOND-MVD using hybrid image. Experiments demonstrate reduction in bit rate using hybrid image against residual image in SECOND-MVD framework.

We propose a novel 3D space representation for multi-view video, using epipolar plane depth images (EPDI). Multi-view video plus depth (MVD) is used as common data format for FTV (Free-viewpoint TV), which enables synthesizing virtual view images. Due to the large amount of data and complexity of the multi-view video coding (MVC), compression of MVD is a challenging issue. We address this problem and propose a new representation that is constructed from MVD using ray-space. MVD is converted into image and depth ray-spaces. The proposed representation is obtained by converting each of ray-spaces into a global depth map and a global view using EPDI. Experiments demonstrate the analysis of this representation.

Free-viewpoint TV (FTV) is cutting the frontier of audiovisual communications. FTV is an innovative media that enables us to view 3-D space by freely changing our viewpoints. It also allows us to listen at any listening point in the 3-D space. Since FTV transmits all audiovisual information of the 3-D space, it can reconstruct an audiovisual replica of the 3-D space anywhere and anytime over distance and time. For video, FTV captures a part of rays in 3-D space by using many cameras, and the other rays that are not captured are obtained by interpolating the captured rays. We constructed real-time FTV systems including the complete chain of operation from image capture to display. We also carried out FTV on a laptop computer and a mobile player. For audio, two kinds of free listening-point systems are demonstrated. MPEG regarded FTV as the most challenging 3-D media and has been conducting its international standardization activities. The first phase of FTV was multiview video coding (MVC) and the second phase of FTV is 3-D video (3DV). MVC enables the efficient coding of multiple camera views and was completed in 2009. MVC has been adopted by Blu-ray 3-D. 3DV is a standard that targets serving a variety of 3-D displays and its call for proposals was issued in March 2011.

We have developed a 3DTV system that captures and shows 3D images that covers 360-degree of viewing zone horizontally. 3D images are captured and reproduced as gathering of dense light rays. This system consists of a cylinder-shaped 3D display that allows viewers to see 3D images from 360-degree, a ray capturing unit, a realtime image correction unit, and data transferring system. The capturing unit acquires multiview images from all horizontal directions around an object with narrow view interval, which the display needs as light ray data. The image correction unit can corrects rotation and distortion that caused by optics of ray capturing unit in real time.

Traffic sign recognition systems can be applied to assist drivers and improve the traffic safety. High resolution image of traffic sign can improve the recognition result, especially for some complex traffic signs. In this paper, a dual-focal active camera system is proposed to obtain a high resolution image of traffic sign. In the system an active telephoto camera is equipped as an assistant of a wide angle camera. To make the proposed system correctly capture a high resolution image of traffic sign, the shape, color features and the relationship between continuous frames are used together in the traffic sign detection. The experiment results demonstrate that the proposed method makes the dual-focal active camera system effectively work when the system is installed on an automobile and moves on the road.

View synthesis using depth maps is an important application in 3D image processing. In this paper, a novel method is proposed for the plausible view synthesis of Free-viewpoint TV (FTV), using two input images and their depth maps. The depth estimation based on stereo matching is known to be error-prone, leading to noticeable artifacts in the synthesized new views. To produce high-quality view synthesis, we introduce a probabilistic framework which constrains the reliability of each pixel of new view by Maximizing Likelihood (ML). The spatial adaptive reliability is provided by incorporating Gamma hyper-prior and the synthesis error approximation. Furthermore, we generate the virtual view by solving a Maximum a Posterior (MAP) problem using graph cuts. We compare the proposed method with other depth based view synthesis approaches on MPEG test sequences. The results show the outperformance of our method both at subjective artifacts reduction and objective PSNR improvement.

In this paper, we introduce a new vision system for traffic sign recognition. The new vision system is a hybrid camera system, including a wide angle camera, a narrow angle camera and control part. The traffic sign candidates are detected in the image of the wide angle camera, based on shape and local feature. The control part of the vision system is adjusted to the correct positions, which makes the narrow angle camera of this system to get a high resolution image of each detected candidate. This high resolution image is used for traffic sign classification. In the new vision system, the traffic sign detection and classification are processed in different resolution images separately. This advantage of the system makes traffic sign recognition at long distance possible.

As one of ITS technique, a new visible light road-to-vehicle communication system at intersections is proposed. In this system, the communication between a vehicle and an LED traffic light is conducted using an LED traffic light as a transmitter, and an on-vehicle high-speed camera as a receiver. The LEDs in the transmitter emit light in high frequency and those emitting LEDs are captured by the high-speed camera for making communication. Here, the luminance value of LEDs in the transmitter should be captured in consecutive frames to achieve effective communication. For this purpose, first the transmitter should be found, then it should be tracked in consecutive frames by processing the images from the high-speed camera. In this paper, we propose new effective algorithms for finding and tracking the transmitter, which result in a increased communication speed, compared to the previous methods. Experiments using appropriate images showed the effectiveness of the proposals.

The availability of multi-view images of a scene makes new and exciting applications possible, including Free-Viewpoint TV (FTV). FTV allows us to change viewpoint freely in a 3D world, where the virtual viewpoint images are synthesized by Image-Based Rendering (IBR). In this paper, we introduce a FTV depth estimation method for forward virtual viewpoints. Moreover, we introduce a view generation method by using a zoom camera in our camera setup to improve virtual viewpoint-ts' image quality. Simulation results confirm reduced error during depth estimation using our proposed method in comparison with conventional stereo matching scheme. We have demonstrated the improvement in image resolution of virtually moved forward camera using a zoom camera setup.

We propose a novel 3D space representation for multi-view video, using epipolar plane depth images (EPDI). Multi-view video plus depth (MVD) is used as common data format for FTV(Free-viewpoint TV), which enables synthesizing virtual view images. Due to large amount of data and complexity of the multi-view video coding (MVC), compression of MVD is a challenging issue. We address this problem and propose a new representation that is constructed from MVD using rayspace. MVD is converted into image and depth ray-spaces. The proposed representation is obtained by converting each of ray-spaces into a global depth map and a texture map using EPDI. Experiments demonstrate the analysis of this representation, and its efficiency.

In this paper, we propose a parallel processing method to generate free viewpoint image in realtime. It is impossible to arrange the cameras in a high density realistically though it is necessary to capture images of the scene from innumerable cameras to express the free viewpoint image. Therefore, it is necessary to interpolate the image of arbitrary viewpoint from limited captured images. However, this process has the relation of the trade-off between the image quality and the computing time. In proposed method, it aimed to generate the high-quality free viewpoint image in realtime by applying the parallel processing to time-consuming interpolation part.

3D scene information can be represented in several ways. In applications based on a (N-)view plus (N-)depth representation, both view and depth data is compressed. In this paper we present a depth compression method containing an depth up-sample filter which uses the color view as prior. Our method of depth down-/up-sampling is able to maintain clear object boundaries in the reconstructed depth maps. Our experimental results show that the proposed depth re-sampling filter, used in combination with a standard state-of-the art video encoder, can increase both the coding efficiency and rendering quality.

View synthesis using depth maps is a well-known technique for exploiting the redundancy between multi-view videos. In this paper, we deal with the bitrates of view synthesis at the decoder side of FTV that would use compressed depth maps and views. Both inherent depth estimation error and coding distortion would degrade synthesis quality. The focus is to reduce bitrates required for generating the high-quality virtual view. We employ a reliable view synthesis method which is compared with standard MPEG view synthesis software. The experimental results show that the bitrates required for synthesizing high-quality virtual view could be reduced by utilizing our enhanced view synthesis technique to improve the PSNR at medium bitrates.

In this paper, we propose a semi-automatic depth estimation algorithm whereby the user defines object depth boundaries and disparity initialization. Automatic depth estimation methods generally have difficulty to obtain good depth results around object edges and in areas with low texture. The goal of our method is to improve the depth in these areas and reduce view synthesis artifacts in Depth Image Based Rendering. Good view synthesis quality is very important in applications such as 3DTV and Free-viewpoint Television (FTV). In our proposed method, initial disparity values for smooth areas can be input through a so-called manual disparity map, and depth boundaries are defined by a manually created edge map which can be supplied for one or multiple frames. For evaluation we used MPEG multi-view videos and we demonstrate our algorithm can significantly improve the depth maps and reduce view synthesis artifacts.

This paper presents a novel method to represent a real 3D world using IBR (Image-based Rendering) technology. The major achievement is realization of "walk-through" experience, in which audiences can see the scene of sport games as if they were the player, while the conventional IBR method cannot provide such an experience. The key idea is a newly introduced framework "locally divided ray space" to capture the ray inside the scene. Consequently, the proposed algorithm can generate a virtual image inside the scene using multiple real images captured from outside. In this paper, the outline of the proposed method as well as the possible demonstration at the conference venue is described.

In this paper, we propose two novel methods for arbitrary listening-point generation for 3D audio-video (3DAV) integration in a large-scale multipoint cameras and microphones system with abilities to process, and display information of any recorded 3D scene in realtime. With this system, users are able to control their own viewpoint/listening-point position, freely. Arbitrary listening-point can be generated by either (i) ray-space representation of sound wave field (i.e. source sound independent) for multi frequency layers, or (ii) acoustic transfer function estimation (i.e. source sound dependent) and blind separation of sources of sounds. Arbitrary viewpoint generation is based on ray-space method, which is enhanced by using multipass dynamic programming for geometry compensation. Integration is done by either (i) ray-space representation of sound wave and image together, or (ii) integrating each camera video signal and acoustic transfer function of the same location as integrated 3DAV data. The prototype system of integrated audio-visual viewer achieves both good image and sound qualities with 15 frames/second.

This paper proposes arbitrary listening-point generation of sound without source localization, and a theory based on the ray-space representation of light rays using sub-band signal processing. An array of beam-formed dynamic microphone-arrays (MAs), are set and each MA generates a multi frequency layer sound-image (SImage) by scanning the viewing range of a camera. Each layer is captured by active microphones in MA for a given frequency range which makes the correct interval to capture that frequency layer, and a sound wave ray-space. Arbitrary listening-point generation of each layer is done by geometry compensation of corresponding images in the location of each MA or SImages or their combination. Each layer sound of an SIamge is generated by averaging the sound wave in each pixel or group of pixel. The listening-point is generated after combining each layer sound in frequency domain and inverse transformation from frequency to time domain

This paper addresses the problem of 3D sound representation without sound source localization and proposes a theory based on the ray-space representation of light rays, which is independent of object's specifications. An array of beam-formed microphone-arrays (MAs), are set and each MA generates a sound-image (SImage) by scanning the viewing range of a camera in the same location. SImage has the same size of an image and contains of blocks of sound wave with duration of one image-frame. Captured SImages with the array of MAs generate the sound wave ray-space. To make a dense SImage ray-space, we propose to use the geometry compensation of corresponding images in the location of each MA. By a dense sound ray-space, any virtual SImage, which corresponds to an arbitrary listening-point, can be generated. The listening-point sound is generated by averaging the sound wave in each pixel or group of pixel of the virtual SImage.

Considering nodes energy and channel bandwidth limitations in a multiview images network, avoiding inter-node communication in the coding scheme is necessary and makes the communication efficient, in comparison with conventional coding methods with internode communication. In our system, we consider a multiview images network as an array of nodes on a line with the same distance to each other. Each node in this system includes a camera to capture with a limited processing and communication abilities. We propose a multiview images coding without requiring inter-node communication, to gain the advantage of correlation at decoder side for two network configurations, but the main concept of the proposed coding for both network configurations is the same. The two network configurations are distinguished based on presence of parent node in each cluster, which sends full information to central node (i.e. joint decoder). The other nodes called children nodes and send their partial information to central node. In this paper, a coding scheme is proposed for multiview images network with parent node based on the number of parent nodes (i.e. “one parent node cluster” and “two parent node cluster”) in each cluster. Not that, because we are involved with multiview images, the decoding procedure is searching for correspondence between partially sent multiview images data. In coding scheme with parent node, if each parent node fails the decoding task of children nodes will be failed or the error caused by corresponding search during the decoding of children nodes is increased if network uses other parent nodes to decode the cluster without parent node. To make the network robust in case of node failure, we avoid parent nodes, and developed a network configuration where all nodes are children nodes (i.e. "without parent node cluster"). For such a network, we also propose a coding algorithm for multiview images that the joint decoder at central node is able to decode the partial information of each children node with side information of other children node. So, in case of losing any viewpoint, joint decoder is still able to decode other received partial information of children nodes. Finally, we have compared coding scheme of network with and without inter-node communication for different cluster configuration mentioned herein, and “all parent node cluster” configuration, considering communication rate in the network, symmetry in communication and decoding quality of children node, and their robustness in case of nodes failure.