FTV (Free-viewpoint Television) is the ultimate 3DTV that captures all rays in 3D space and reproduce the same rays at different place and time, thus it allows users to view a 3D scene by freely changing the viewpoints. We proposed the concept of FTV and realized FTV by developing various types of ray capture, processing and display technologies. At present, FTV is available on a laptop PC or a mobile player, and FTV with free listening-point audio is also realized. The international standardization of FTV has been conducted in MPEG. The first phase of FTV was MVC (Multi-view Video Coding) and the second phase is 3DV (3D Video).

We propose a novel 3D space representation for multi-view video, using epipolar plane depth images (EPDI). Multi-view video plus depth (MVD) is used as common data format for FTV (Free-viewpoint TV), which enables synthesizing virtual view images. Due to the large amount of data and complexity of the multi-view video coding (MVC), compression of MVD is a challenging issue. We address this problem and propose a new representation that is constructed from MVD using ray-space. MVD is converted into image and depth ray-spaces. The proposed representation is obtained by converting each of ray-spaces into a global depth map and a global view using EPDI. Experiments demonstrate the analysis of this representation.

Free-viewpoint TV (FTV) is cutting the frontier of audiovisual communications. FTV is an innovative media that enables us to view 3-D space by freely changing our viewpoints. It also allows us to listen at any listening point in the 3-D space. Since FTV transmits all audiovisual information of the 3-D space, it can reconstruct an audiovisual replica of the 3-D space anywhere and anytime over distance and time. For video, FTV captures a part of rays in 3-D space by using many cameras, and the other rays that are not captured are obtained by interpolating the captured rays. We constructed real-time FTV systems including the complete chain of operation from image capture to display. We also carried out FTV on a laptop computer and a mobile player. For audio, two kinds of free listening-point systems are demonstrated. MPEG regarded FTV as the most challenging 3-D media and has been conducting its international standardization activities. The first phase of FTV was multiview video coding (MVC) and the second phase of FTV is 3-D video (3DV). MVC enables the efficient coding of multiple camera views and was completed in 2009. MVC has been adopted by Blu-ray 3-D. 3DV is a standard that targets serving a variety of 3-D displays and its call for proposals was issued in March 2011.

We have developed a 3DTV system that captures and shows 3D images that covers 360-degree of viewing zone horizontally. 3D images are captured and reproduced as gathering of dense light rays. This system consists of a cylinder-shaped 3D display that allows viewers to see 3D images from 360-degree, a ray capturing unit, a realtime image correction unit, and data transferring system. The capturing unit acquires multiview images from all horizontal directions around an object with narrow view interval, which the display needs as light ray data. The image correction unit can corrects rotation and distortion that caused by optics of ray capturing unit in real time.

Traffic sign recognition systems can be applied to assist drivers and improve the traffic safety. High resolution image of traffic sign can improve the recognition result, especially for some complex traffic signs. In this paper, a dual-focal active camera system is proposed to obtain a high resolution image of traffic sign. In the system an active telephoto camera is equipped as an assistant of a wide angle camera. To make the proposed system correctly capture a high resolution image of traffic sign, the shape, color features and the relationship between continuous frames are used together in the traffic sign detection. The experiment results demonstrate that the proposed method makes the dual-focal active camera system effectively work when the system is installed on an automobile and moves on the road.

FTV (Free-viewpoint TV) is the ultimate 3DTV that delivers the replica of a 3D scene and enables us to view the scene by freely changing our viewpoints. We realized FTV by developing various types of ray capture, processing and display technologies. We have also realized an all-around ray-reproducing 3DTV. The international standardization of FTV has been conducted in MPEG. The first phase of FTV was MVC (Multi-view Video Coding) and the second phase is 3DV (3D Video).

We have developed a new type of television named FTV (Free-viewpoint TV). FTV is the ultimate 3DTV that enables us to view a 3D scene by freely changing our viewpoints. At present, FTV is available on a single PC or a mobile player. The international standardization of FTV has been conducted in MPEG. The first phase of FTV was MVC (Multi-view Video Coding) and the second phase is 3DV (3D Video). FDU (FTV Data Unit) is proposed as a data format for 3DV. FTU can compensate errors of the synthesized views caused by depth error.

View synthesis using depth maps is an important application in 3D image processing. In this paper, a novel method is proposed for the plausible view synthesis of Free-viewpoint TV (FTV), using two input images and their depth maps. The depth estimation based on stereo matching is known to be error-prone, leading to noticeable artifacts in the synthesized new views. To produce high-quality view synthesis, we introduce a probabilistic framework which constrains the reliability of each pixel of new view by Maximizing Likelihood (ML). The spatial adaptive reliability is provided by incorporating Gamma hyper-prior and the synthesis error approximation. Furthermore, we generate the virtual view by solving a Maximum a Posterior (MAP) problem using graph cuts. We compare the proposed method with other depth based view synthesis approaches on MPEG test sequences. The results show the outperformance of our method both at subjective artifacts reduction and objective PSNR improvement.

In this paper, we introduce a new vision system for traffic sign recognition. The new vision system is a hybrid camera system, including a wide angle camera, a narrow angle camera and control part. The traffic sign candidates are detected in the image of the wide angle camera, based on shape and local feature. The control part of the vision system is adjusted to the correct positions, which makes the narrow angle camera of this system to get a high resolution image of each detected candidate. This high resolution image is used for traffic sign classification. In the new vision system, the traffic sign detection and classification are processed in different resolution images separately. This advantage of the system makes traffic sign recognition at long distance possible.

As one of ITS technique, a new visible light road-to-vehicle communication system at intersections is proposed. In this system, the communication between a vehicle and an LED traffic light is conducted using an LED traffic light as a transmitter, and an on-vehicle high-speed camera as a receiver. The LEDs in the transmitter emit light in high frequency and those emitting LEDs are captured by the high-speed camera for making communication. Here, the luminance value of LEDs in the transmitter should be captured in consecutive frames to achieve effective communication. For this purpose, first the transmitter should be found, then it should be tracked in consecutive frames by processing the images from the high-speed camera. In this paper, we propose new effective algorithms for finding and tracking the transmitter, which result in a increased communication speed, compared to the previous methods. Experiments using appropriate images showed the effectiveness of the proposals.

The availability of multi-view images of a scene makes new and exciting applications possible, including Free-Viewpoint TV (FTV). FTV allows us to change viewpoint freely in a 3D world, where the virtual viewpoint images are synthesized by Image-Based Rendering (IBR). In this paper, we introduce a FTV depth estimation method for forward virtual viewpoints. Moreover, we introduce a view generation method by using a zoom camera in our camera setup to improve virtual viewpoint-ts' image quality. Simulation results confirm reduced error during depth estimation using our proposed method in comparison with conventional stereo matching scheme. We have demonstrated the improvement in image resolution of virtually moved forward camera using a zoom camera setup.

We propose a novel 3D space representation for multi-view video, using epipolar plane depth images (EPDI). Multi-view video plus depth (MVD) is used as common data format for FTV(Free-viewpoint TV), which enables synthesizing virtual view images. Due to large amount of data and complexity of the multi-view video coding (MVC), compression of MVD is a challenging issue. We address this problem and propose a new representation that is constructed from MVD using rayspace. MVD is converted into image and depth ray-spaces. The proposed representation is obtained by converting each of ray-spaces into a global depth map and a texture map using EPDI. Experiments demonstrate the analysis of this representation, and its efficiency.

In this paper, we propose a parallel processing method to generate free viewpoint image in realtime. It is impossible to arrange the cameras in a high density realistically though it is necessary to capture images of the scene from innumerable cameras to express the free viewpoint image. Therefore, it is necessary to interpolate the image of arbitrary viewpoint from limited captured images. However, this process has the relation of the trade-off between the image quality and the computing time. In proposed method, it aimed to generate the high-quality free viewpoint image in realtime by applying the parallel processing to time-consuming interpolation part.

3D scene information can be represented in several ways. In applications based on a (N-)view plus (N-)depth representation, both view and depth data is compressed. In this paper we present a depth compression method containing an depth up-sample filter which uses the color view as prior. Our method of depth down-/up-sampling is able to maintain clear object boundaries in the reconstructed depth maps. Our experimental results show that the proposed depth re-sampling filter, used in combination with a standard state-of-the art video encoder, can increase both the coding efficiency and rendering quality.

View synthesis using depth maps is a well-known technique for exploiting the redundancy between multi-view videos. In this paper, we deal with the bitrates of view synthesis at the decoder side of FTV that would use compressed depth maps and views. Both inherent depth estimation error and coding distortion would degrade synthesis quality. The focus is to reduce bitrates required for generating the high-quality virtual view. We employ a reliable view synthesis method which is compared with standard MPEG view synthesis software. The experimental results show that the bitrates required for synthesizing high-quality virtual view could be reduced by utilizing our enhanced view synthesis technique to improve the PSNR at medium bitrates.

We have developed a new type of television named FTV (Free-viewpoint TV). FTV is the ultimate 3DTV that enables us to view a 3D scene by freely changing our viewpoints. We proposed the concept of FTV and constructed the world's first real-time system including the complete chain of operation from image capture to display. FTV is based on the rayspace method that represents one ray in real space with one point in the ray-space. We have developed ray capture, processing and display technologies for FTV. FTV can be carried out today in real time on a single PC or on a mobile player. We also realized FTV with free listening-point audio. The international standardization of FTV has been conducted in MPEG. The first phase of FTV was MVC (Multi-view Video Coding) and the second phase is 3DV (3D Video). MVC was completed in May 2009. The Blu-ray 3D specification has adopted MVC for compression. 3DV is a standard that targets serving a variety of 3D displays. The view generation function of FTV is used to decouple capture and display in 3DV. FDU (FTV Data Unit) is proposed as a data format for 3DV. FTU can compensate errors of the synthesized views caused by depth error.

In this paper, we propose a semi-automatic depth estimation algorithm whereby the user defines object depth boundaries and disparity initialization. Automatic depth estimation methods generally have difficulty to obtain good depth results around object edges and in areas with low texture. The goal of our method is to improve the depth in these areas and reduce view synthesis artifacts in Depth Image Based Rendering. Good view synthesis quality is very important in applications such as 3DTV and Free-viewpoint Television (FTV). In our proposed method, initial disparity values for smooth areas can be input through a so-called manual disparity map, and depth boundaries are defined by a manually created edge map which can be supplied for one or multiple frames. For evaluation we used MPEG multi-view videos and we demonstrate our algorithm can significantly improve the depth maps and reduce view synthesis artifacts.

We have developed a new type of television named FTV (Free-viewpoint TV). FTV is an innovative visual media that enables us to view a 3D scene by freely changing our viewpoints. We constructed the world's first real-time FTV system including the complete chain from capturing to display. We have also developed new type of ray capture and display technologies such as a 360-degree mirror-scan ray capturing system and a 360 degree ray-reproducing display. MPEG has been conducting the standardization activities of FTV. The first phase of FTV was multi-view video coding and the second phase is 3D video.

We have developed a new type of television named FTV (Free viewpoint TV). FTV is an innovative media that enables us to view a 3D scene by freely changing our viewpoints. FTV will be widely used since it is an ultimate 3DTV, a natural interface between human and environment, and an innovative tool to create new types of content and art. FTV has been contributing to the standardization of MVC (Multi-view Video Coding) and 3DV (3D Video) at MPEG. MVC is the first phase of FTV and 3DV is the second phase of FTV. An FTV system can be constructed in various ways, but typically includes multiple viewpoint video capture, image correction, depth estimation, coding/decoding and view interpolation. Coding is related to view generation in FTV. In this paper, 4 frameworks for FTV coding are discussed and an FTV data format as a combination of FTV data units is proposed.

In this paper, we propose two novel methods for arbitrary listening-point generation for 3D audio-video (3DAV) integration in a large-scale multipoint cameras and microphones system with abilities to process, and display information of any recorded 3D scene in realtime. With this system, users are able to control their own viewpoint/listening-point position, freely. Arbitrary listening-point can be generated by either (i) ray-space representation of sound wave field (i.e. source sound independent) for multi frequency layers, or (ii) acoustic transfer function estimation (i.e. source sound dependent) and blind separation of sources of sounds. Arbitrary viewpoint generation is based on ray-space method, which is enhanced by using multipass dynamic programming for geometry compensation. Integration is done by either (i) ray-space representation of sound wave and image together, or (ii) integrating each camera video signal and acoustic transfer function of the same location as integrated 3DAV data. The prototype system of integrated audio-visual viewer achieves both good image and sound qualities with 15 frames/second.

LEDs are expected as lighting sources for next generation, and data transmission system using LEDs attract attention. In this paper, we present hierarchical coding scheme using LED traffic lights and high-speed camera for intelligent transport systems (ITS) application. Further, if each of LEDs in traffic lights is individually modulated, parallel data transmissions are possible using a camera as a reception device. Such parallel LED-camera channel can be modeled as spatial low-pass filtered channel of which the cut-off frequency varies according to the distance. To overcome, we propose hierarchical coding scheme based on 2D fast Haar wavelet transform. As results, the proposed hierarchical transmission schemes outperform the conventional on-off keying and the reception of high priority data is guaranteed even LED-camera distance is further.

The 11 papers in this special section focus on multiview video coding.

Neighboring views must be highly correlated in multiview video systems. We should therefore use various neighboring views to efficiently compress videos. There are many approaches to doing this. However, most of these treat pictures of other views in the same way as they treat pictures of the current view, i.e., pictures of other views are used as reference pictures (inter-view prediction). We introduce two approaches to improving compression efficiency in this paper. The first is by synthesizing pictures at a given time and a given position by using view interpolation and using them as reference pictures (view-interpolation prediction). In other words, we tried to compensate for geometry to obtain precise predictions. The second approach is to correct the luminance and chrominance of other views by using lookup tables to compensate for photoelectric variations in individual cameras. We implemented these ideas in H.264/AVC with inter-view prediction and confirmed that they worked well. The experimental results revealed that these ideas can reduce the number of generated bits by approximately 15% without loss of PSNR.

We developed a novel multipoint measurement system capable of acquiring video and sound at more than 100 points in a "synchronized" manner. In this paper, we first describe the specification of the system and how the system works in detail. Then we report some experimental results that confirm the performance of the system. We also describe test data set we provided for MPEG (moving picture experts group) multi-viewpoint video coding activities. Using this system, we are planning to conduct projects to measure humans and their activities, collect a large volume of real-world data of video and sound, and release them to the public

We propose a new multi-view video coding method using adaptive selection of motion/disparity compensation based on H.264/AVC. One of the key points of the proposed method is the use of view interpolation as a tool for disparity compensation by assigning reference picture indices to interpolated images. Experimental results show that significant gains can be obtained compared to the conventional approach that was often used

We have been developing ray-based 3D information systems that consist of ray acquisition, ray processing, and ray display. FTV (free viewpoint TV) based on the ray-space method is a typical example. FTV will bring an epoch-making change in the history of television because it enables us to view a distant 3D world freely by changing our viewpoints as if we were there. We constructed a real-time FTV including the complete chain from capturing to display. A new algorithm was developed to generate free viewpoint images. We need a new user interface for FTV to make full use of 3D information of FTV. Two types of user interface were made using 2D and automultiscopic displays with a head tracking system. We are creating "ray-based image engineering" through these works.

This paper addresses the problem of 3D sound representation without sound source localization and proposes a theory based on the ray-space representation of light rays, which is independent of object's specifications. An array of beam-formed microphone-arrays (MAs), are set and each MA generates a sound-image (SImage) by scanning the viewing range of a camera in the same location. SImage has the same size of an image and contains of blocks of sound wave with duration of one image-frame. Captured SImages with the array of MAs generate the sound wave ray-space. To make a dense SImage ray-space, we propose to use the geometry compensation of corresponding images in the location of each MA. By a dense sound ray-space, any virtual SImage, which corresponds to an arbitrary listening-point, can be generated. The listening-point sound is generated by averaging the sound wave in each pixel or group of pixel of the virtual SImage.

Considering nodes energy and channel bandwidth limitations in a multiview images network, avoiding inter-node communication in the coding scheme is necessary and makes the communication efficient, in comparison with conventional coding methods with internode communication. In our system, we consider a multiview images network as an array of nodes on a line with the same distance to each other. Each node in this system includes a camera to capture with a limited processing and communication abilities. We propose a multiview images coding without requiring inter-node communication, to gain the advantage of correlation at decoder side for two network configurations, but the main concept of the proposed coding for both network configurations is the same. The two network configurations are distinguished based on presence of parent node in each cluster, which sends full information to central node (i.e. joint decoder). The other nodes called children nodes and send their partial information to central node. In this paper, a coding scheme is proposed for multiview images network with parent node based on the number of parent nodes (i.e. “one parent node cluster” and “two parent node cluster”) in each cluster. Not that, because we are involved with multiview images, the decoding procedure is searching for correspondence between partially sent multiview images data. In coding scheme with parent node, if each parent node fails the decoding task of children nodes will be failed or the error caused by corresponding search during the decoding of children nodes is increased if network uses other parent nodes to decode the cluster without parent node. To make the network robust in case of node failure, we avoid parent nodes, and developed a network configuration where all nodes are children nodes (i.e. "without parent node cluster"). For such a network, we also propose a coding algorithm for multiview images that the joint decoder at central node is able to decode the partial information of each children node with side information of other children node. So, in case of losing any viewpoint, joint decoder is still able to decode other received partial information of children nodes. Finally, we have compared coding scheme of network with and without inter-node communication for different cluster configuration mentioned herein, and “all parent node cluster” configuration, considering communication rate in the network, symmetry in communication and decoding quality of children node, and their robustness in case of nodes failure.

Conventional ray-space acquisition systems require very precise mechanisms to control the small movement of cameras or objects. Most of them adopt camera with a gantry or a turntable. Although they are good for acquiring the ray-space of small objects, they are not suitable for ray-space acquisition of very large structures, such as a building, tower, etc. This paper proposes a new ray-space acquisition system which consists of a camera and a 3-D position and orientation sensor. It is not only a compact and easy-to-handle system, but is also free from limitations of size or shape, in principle. It can obtain any ray-space data as far as the camera is located within the coverage of the 3-D sensor. This paper describes our system and its specifications. Experimental results are also presented.

In the fractal coding schemes proposed so far based on iterated function systems, an image is represented by a self-affine set of square blocks. We propose a scheme of fractal image coding with blocks of variable shapes. In this scheme, the range blocks are determined by the so-called splitting-and-merging method. Also, we define two kinds of range blocks, shade blocks and non-shade blocks, according to the variance of the brightness level inside blocks. By this method, range blocks both of various size and of various shapes are extracted from an input image. The larger range blocks result in the a greater reduction of the total number of blocks and, hence, the amount of encoded bits. Because of the specified order used in merging of primitive blocks, only a few more bits are added for each range block to encode its shape. The results of computer simulation with a test gray-scale image show that compared with the conventional scheme using square range blocks, the proposed coding scheme reduces about 0.2 bits/pel to achieve almost the same reproduced quality.

The optimal analysis/synthesis filters giving the maximum coding gain are derived in subband schemes. The optimal analysis filters consist of the emphasis of the picture signal and ideal band-splitting. The characteristics of the emphasis is determined by the spectrum of the picture signal. A large improvement of coding gain is achieved by the subband scheme with the optimal subband filters obtained here. Approximated emphasis characteristic determined from a spectrum model of picture signals can be used and the ideal band-splitting filters can be replaced by conventional subband filters since the degradation of coding gain due to these approximations is small. Computer simulation of super HD image coding by the proposed scheme is performed. SN ratio of the reconstructed image is increased and edges are reconstructed very well compared to the conventional subband scheme. The proposed scheme is very suited to super HD image coding since the improvement of SN ratio is large for images with high correlation between the neighboring pixels.