Collaborative Intelligence is a new paradigm that splits a deep neural network (DNN) into an edge and cloud for deploying a DNN-based image recognition application. In this paradigm, deep features, which are the outputs of the edge DNN, are compressed and transmitted to the cloud DNN. Because the deep features have a number of responses that are similar to each other, for efficient compression, previous methods spatially arrange and compress the deep features as an image to utilize the similarity as a spatial correlation. However, if the deep features are arranged in not only spatial but also temporal directions like those in a video, it may be possible to compress them more efficiently by increasing a temporal correlation. To explore this possibility, we propose a “spatio-temporal arrangement”. This method spatially arranges the deep features as images and temporally arranges them as a video with a novel ordering search algorithm. Our method effectively increases the spatial and temporal correlations hidden in the deep features and achieves high compression efficiency compared with the previous methods. Experimental results demonstrate the compression efficiency of our method is better than that of the previous methods (1.50% to 4.98% on BD-Rate evaluation in a lossy setting). Our analysis shows that our method effectively increases the correlation when the input is an image with rich edges and textures.

Anti-aliased convolutional neural networks (CNNs) introduce blur filters to intermediate representations in CNNs to achieve high accuracy. A promising way to build a new antialiased CNN is to fine-tune a pre-trained CNN, which can easily be found online, with blur filters. However, blur filters drastically degrade the pre-trained representation, so the fine-tuning needs to rebuild the representation by using massive training data. Therefore, if the training data is limited, the fine-tuning cannot work well because it induces overfitting to the limited training data. To tackle this problem, this paper proposes “knowledge transferred fine-tuning”. On the basis of the idea of knowledge transfer, our method transfers the knowledge from intermediate representations in the pre-trained CNN to the anti-aliased CNN while fine-tuning. We transfer only essential knowledge using a pixel-level loss that transfers detailed knowledge and a global-level loss that transfers coarse knowledge. Experimental results demonstrate that our method significantly outperforms the simple fine-tuning method.

Compressive spectral imaging (CSI) is an emerging technique for acquiring hyperspectral images based on compressed sensing theory. By combining powerful data-driven methods with traditional iterative convex optimization algorithms, we propose a multi-staged neural network architecture, which is inspired by ADMM, to achieve fast and accurate reconstruction for CSI. All parameters of ADMM and deep hyperspectral priors are jointly learned. The proposed ADMM-inspired network was experimentally shown to outperform an existing state-of-the-art reconstruction method in terms of both speed and accuracy.

Collaborative Intelligence is a new paradigm that splits a deep neural network (DNN) into an edge DNN and a cloud DNN. In this paradigm, deep features, which are the outputs of the edge DNN, are compressed and transmitted to the cloud DNN. Since the deep features have a few responses that are similar to each other, previous studies have proposed compressing them as an image with a spatial arrangement to utilize spatial correlation between the deep features. However, this method may not sufficiently consider the similarity because only the spatial correlation is utilized. In this work, we propose a “spatio-temporal arranging” that considers the similarity of deep features in both the spatial and temporal directions. This method arranges the deep features spatiotemporally and compresses them as a video to utilize the spatio-temporal correlation. We perform spatial arranging as images to increase the spatial correlation and temporal arranging as a video with a novel ordering search to increase the temporal correlation. Experimental results demonstrate that our method performs better than previous methods.

Video magnification methods can magnify and reveal subtle changes invisible to the naked eye. However, in such subtle changes, meaningful ones caused by physical and natural phenomena are mixed with non-meaningful ones caused by photographic noise. Therefore, current methods often produce noisy and misleading magnification outputs due to the non-meaningful subtle changes. For detecting only meaningful subtle changes, several methods have been proposed but require human manipulations, additional resources, or input video scene limitations. In this paper, we present a novel method using fractional anisotropy (FA) to detect only meaningful subtle changes without the aforementioned requirements. FA has been used in neuroscience to evaluate anisotropic diffusion of water molecules in the body. On the basis of our observation that temporal distribution of meaningful subtle changes more clearly indicates anisotropic diffusion than that of non-meaningful ones, we used FA to design a fractional anisotropic filter that passes only meaningful subtle changes. Using the filter enables our method to obtain better and more impressive magnification results than those obtained with state-of-the-art methods.

A new full parallax light field 3D display that optically interpolates the rays emitted is proposed. Using a barrier with special aperture structure, we achieve interpolation by optical linear blending not only for the horizontal direction, but also for the vertical direction. Linear blending produces visually equivalent intermediate view point images equivalent to the real views if the disparity between the two images being blended are small enough. Therefore, a high feel of existence can be produced as the interpolation yields smooth motion parallax. Since interpolation reduces the number of viewpoints, number of pixels in each direction can be increased yielding high resolution. In this demonstration, we reproduce CG objects and scenes of a ball game on a table top.

This paper proposes a learning-based quality evaluation framework for inpainted results that does not require any subjectively annotated training data. Image inpainting, which removes and restores unwanted regions in images, is widely acknowledged as a task whose results are quite difficult to evaluate objectively. Thus, existing learning-based image quality assessment (IQA) methods for inpainting require subjectively annotated data for training. However, subjective annotation requires huge cost and subjects’ judgment occasionally differs from person to person in accordance with the judgment criteria. To overcome these difficulties, the proposed framework generates and uses simulated failure results of inpainted images whose subjective qualities are controlled as the training data. We also propose a masking method for generating training data towards fully automated training data generation. These approaches make it possible to successfully estimate better inpainted images, even though the task is quite subjective. To demonstrate the effectiveness of our approach, we test our algorithm with various datasets and show it outperforms existing IQA methods for inpainting.

This paper proposes an image quality assessment (IQA) method for image inpainting, aiming at selecting the best one from a plurality of results. It is known that inpainting results vary largely with the method used for inpainting and the parameters set. Thus, in a typical use case, users need to manually select the inpainting method and the parameters that yield the best result. This manual selection takes a great deal of time and thus there is a great need for a way to automatically estimate the best result. Unlike existing IQA methods for inpainting, our method solves this problem as a learning-based ordering task between inpainted images. This approach makes it possible to introduce auto-generated training sets for more effective learning, which has been difficult for existing methods because judging inpainting quality is quite subjective. Our method focuses on the following three points: (1) the problem can be divided into a set of “pairwise preference order estimation” elemental problems, (2) this pairwise ordering approach enables a training set to be generated automatically, and (3) effective feature design is enabled by investigating actually measured human gazes for order estimation.

Video magnification reveals subtle changes invisible to the naked eye, but such tiny yet meaningful changes are often hidden under large motions: small deformation of the muscles in doing sports, or tiny vibrations of strings in ukulele playing. For magnifying subtle changes under large motions, video acceleration magnification method has recently been proposed. This method magnifies subtle acceleration changes and ignores slow large motions. However, quick large motions severely distort this method. In this paper, we present a novel use of jerk to make the acceleration method robust to quick large motions. Jerk has been used to assess smoothness of time series data in the neuroscience and mechanical engineering fields. On the basis of our observation that subtle changes are smoother than quick large motions at temporal scale, we used jerk-based smoothness to design a jerk-aware filter that passes subtle changes only under quick large motions. By applying our filter to the acceleration method, we obtain impressive magnification results better than those obtained with state-of-the-art.

Navigation and mobility mechanics for virtual environments aim to be realistic or fun, but rarely prioritize the accuracy of movement. We propose PinchMove, a highly accurate navigation mechanic utilizing pinch gestures and manipulation of the viewport for confined environments that prefers accurate movement. We ran a pilot study to first determine the degree of simulator sickness caused by this mechanic, and a comprehensive user study to evaluate its accuracy in a virtual environment. We found that utilizing an 80° tunneling effect at a maximum speed of 15.18° per second was deemed suitable for PinchMove in reducing motion sickness. We also found our system to be at average, more accurate in enclosed virtual environments when compared to conventional methods. This paper makes the following three contributions: 1) We propose a navigation solution in near-field virtual environments for accurate movement, 2) we determined the appropriate tunneling effect for our method to minimize motion sickness, and 3) We validated our proposed solution by comparing it with conventional navigation solutions in terms of accuracy of movement. We also propose several use- case scenarios where accuracy in movement is desirable and further discuss the effectiveness of PinchMove.

This study aims at ascertaining the applicability of a virtual reality environment (VRE) to sports training. Hitting an incoming object is one of the most common actions in various ball games, in which players are required to move to a suitable position and hit the object in a split second; this is a complicated task requiring spatio-temporal reaction to the object. Due to this complexity, how a VRE can serve as a training environment still remains an open question. In the work reported in this paper, we investigated the idea of substituting a VRE for an actual environment for training on the task of hitting a baseball. By focusing on the batter's temporal behavior with real and virtual environments, we clarified factors that contribute to the batter's reaction. This helped us understand how training VREs can be effectively utilized and the VRE requirements needed for sports training.

View synthesis prediction has been studied as an efficient inter-view prediction scheme. Existing view synthesis prediction schemes fall into two types according to the pixel warping direction. While backward warping based view synthesis prediction enables block-based processing, forward warping based view synthesis prediction can handle occlusions properly. This paper proposes a two-step warping based view synthesis prediction; a virtual depth map is first generated by forward warping, and then prediction signals are generated by block-based backward warping using the virtual depth map. The technique of backward-warping-aware depth inpainting is also proposed. Experiments show that the proposed VSP scheme can achieve the decoder runtime reductions of about 37% on average with slight bitrate reductions relative to the conventional forward warping based VSP. Compared to the conventional backward warping based VSP, the proposed method reduces the bitrate for the synthesized views by up to 2.9% and about 2.2% on average.

Video with depth is attracting attention as a promising approach to future 3D video. Unlike conventional video coding technologies, video with depth should take the correlations between video and depth into account. This paper proposes a novel bi-prediction scheme for video coding that exploits the structure similarity between video and depth. In the proposed scheme, a prediction image is generated by the pixel-wise weighted average of motion compensated reference images, and weights are decided by considering pixel similarities in the depth domain. The proposed pixel-wise weighted bi-prediction offers implicit depth-based block partitioning with consideration of the pixel correlations between predicted and reference blocks. Compared to using explicit, arbitrary shaped block partitioning, the proposed method offers better prediction for non-rigid objects, and requires no increase in motion information. Experiments show that the proposed method can achieve up to 17% bitrate reduction for bi-predictive frames and about 8% on average for 6 sequences relative to the existing biprediction scheme.

Pixel domain methods for content-aware image retargeting have a performance limitation in that they distort important objects when an image is shrunken too much. To address this problem, we have designed and developed a novel 3D space domain image retargeting which rearranges objects with occlusions and deforms objects. With this method image formation is modeled as a quasi-3D space. The method also reduces image width by shrinking a part of the image space. Instead of directly optimizing the position and shape of objects, it closes objects by shrinking the gap space between them and deforms objects by shrinking spaces that include parts of them. It is capable of being extended to higher dimension problems such as multi-view image retargeting because it needs to optimize only one quasi-3D space whether the input is ordinary images or more complex ones such as video and stereo images. We have also designed a simple algorithm that makes use of dynamic programming. An experiment we conducted demonstrated the algorithm's effectiveness.

We propose a new interactive panorama video delivery system that enables users not only to select favorite parts of an event site, but also to watch repeatedly with stable and the user-selected camera trajectory. In this paper, we present a system that makes a favorable and stable camera trajectory as a scenario from user's manipulations of a tablet device in a format that can be easily shared with others. We also evaluate the validity of the camera trajectory results obtained our stabilization technique.

We propose a scenario-driven interactive panorama video delivery system that allows users to repeatedly watch the enjoyable parts of an event and share them with others. It provides functions for interactively watching panorama video parts, as well as for scenario making (with user-selected camera trajectory), and delivery that allows users to share their panorama watching experiences. In this technical demos, we demonstrate the system that can make a favorable and stable camera trajectories as a scenario from intuitive user manipulations of a tablet device in a format that can be easily shared with others.

This paper proposes a novel prediction scheme for depth map coding. We utilize the fact that depth values are largely dependent on objects and one small block consists of only a small number of objects at most. The proposed method approximates each block with one palette and one object shape map. The palette consists of two representative depth values for foreground object and background object in the target block. The object shape map expresses which object exists at each pixel. These data are encoded only at the blocks where the proposed method is used. In order to reduce the bitrates required to transfer palettes and object shape maps, the proposed method enables their prediction by utilizing spatial and temporal correlations. Experiments are conducted on three kinds of depth maps; estimated from multiview images, captured by special sensors, and generated by computer. The results show that the proposed method reduces the bitrate by up to 40% and about 20% on average for 13 sequences relative to the MPEG-4 AVC/H.264.

Many new applications are being created around the panoramic video service. The typical system divides the high resolution panoramic video into tiles and the sender transmits a set of tiles, the partial panoramic video. Coding each tile at a uniform bitrate yields poor video quality because each tile has different visual characteristics. This paper proposes a new data format and tile adaptive rate control to achieve high quality partial panoramic video transmission, even over restricted bandwidth networks. The proposed data format, based on the MVC standard, has two types of video stream and meta data. Each tile is encoded at multiple bitrates and multiplexed synchronously. The meta data has quality values of each tile at the multi-bitrates, and is used to determine the view_ids associated with the bitrates and user's desired view. Our tile-adaptive rate control proposal maximizes the partial panoramic video quality even in restricted bandwidth networks. An experiment shows that the proposed method can achieve higher video quality. The method ensures that the facial elements in the user's view, which often exhibit the greatest motion and to which we are most sensitive, have high quality.

We propose a system that allows multiple groups of users at remote locations to naturally share a 3D image of real objects. All users can interactively observe a 3D stereoscopic image without distortion from their own viewpoints. The system basically consists of a combination of systems: imaging and display. The imaging system captures the real object's images from certain number of viewpoints which are used to generate stereoscopic images from arbitrary viewpoints based on an image-based rendering technique implemented on GPU. The display system interactively generates a 3D virtual image of the real object based on the IllusionHole technique. Users at one place just put the real object on their imaging system to capture a set of its images from sparse viewpoints around it; other groups of multiple users at remote places connected by networks observe its virtual image from arbitrary viewpoints on their display systems.

We propose a system that allows multiple groups of users at remote locations to naturally share a 3D image of real objects. All users can interactively observe a 3D stereoscopic image without distortion from their own viewpoints. The system basically consists of a combination of subsystems: imaging and display. The imaging subsystem generates images of the real object observed from arbitrary viewpoints based on image-based rendering technique implemented on GPU. The display system generates a 3D virtual image of the real object to be interactively observed by multiple people around the tabletop display. People at one place just put the real object on their imaging system to capture a set of its images from sparse viewpoints around it; other groups of multiple persons at remote places connected by networks observe its virtual image from arbitrary viewpoints, as if there is a copy of the real object, on their display systems. This paper describes details of the system configurations and algorithms. Then discussions are made based on experimental results.

View synthesis prediction has been studied to achieve efficient inter-view prediction. Existing view synthesis prediction methods synthesize predicted pictures by using decoded pictures from the other views and geometric information of the scene. However, these conventional methods have no ability to compensate the inter-view difference in image signals caused by individual camera characteristics and the nonLambert reflection of objects. This paper proposes adaptive appearance compensated view synthesis prediction; it uses adaptive filters to compensate not only the differences in color and brightness but also the differences in the spatial frequency of images. The object-dependent local differences are also compensated efficiently by estimating the optimal compensation parameters block by block at the decoder. Experiments show that the proposed method reduces the bitrate by up to 40% relative to H.264/AVC multiview video coding, and 26% relative to the conventional view synthesis prediction.

In order to represent 3D space, we have proposed to use the representation that consists of multi-view video plus a single view depth map. This format is backward compatible with the MPEG-C Part 3 (a.k.a. ISO/IEC 23002-3). This paper proposes a coding scheme on this 3D space representation that is efficient even if low delay decoding functionality is required. We apply the residual prediction framework to the view synthesis prediction errors. Experiments show that the proposed scheme achieves up to about 7.7% bitrate reduction compared to multi-view video coding with disparity compensation, even if the depth map video is added. Furthermore, the proposed scheme doesn’t require any syntax changes to the conventional video coding standard and it needs few modifications on the circuits of conventional video codecs; it might be possible to reuse almost all of the components. The backward compatibility and reusability achieved by the proposed scheme are quite important to reduce manufacturing costs and time to market.

A real-time MVC viewer for free-viewpoint navigation is demonstrated. MVC, a video coding standard for multi-view video, is an extension of the state-of-the-art H.264/AVC. Realizing parallel decoding and view synthesis is essential to develop real-time viewers. A new decoder architecture is introduced for the proposed viewer. In addition, a file format for MVC is introduced. View synthesis is performed entirely within GPU. It is demonstrated that the MVC viewer can decode a MVC bitstream and generate a virtual camera image in real-time.

Multiview video coding demands high compression rates as well as view scalability, which enables the video to be displayed on a multitude of different terminals. In order to achieve view scalability, it is necessary to limit the inter-view prediction structure. In this paper, we propose a new multiview video coding scheme that can improve the compression efficiency under such a limited inter-view prediction structure. All views are divided into two groups in the proposed scheme: base view and enhancement views. The proposed scheme first estimates a view-dependent geometry of the base view. It then uses a video encoder to encode the video of base view. The view-dependent geometry is also encoded by the video encoder. The scheme then generates prediction images of enhancement views from the decoded video and the view-dependent geometry by using image-based rendering techniques, and it makes residual signals for each enhancement view. Finally, it encodes residual signals by the conventional video encoder as if they were regular video signals. We implement one encoder that employs this scheme by using a depth map as the view-dependent geometry and 3-D warping as the view generation method. In order to increase the coding efficiency, we adopt the following three modifications: (1) object-based interpolation on 3-D warping; (2) depth estimation with consideration of rate-distortion costs; and (3) quarter-pel accuracy depth representation. Experiments show that the proposed scheme offers about 30% higher compression efficiency than the conventional scheme, even though one depth map video is added to the original multiview video.

Neighboring views must be highly correlated in multiview video systems. We should therefore use various neighboring views to efficiently compress videos. There are many approaches to doing this. However, most of these treat pictures of other views in the same way as they treat pictures of the current view, i.e., pictures of other views are used as reference pictures (inter-view prediction). We introduce two approaches to improving compression efficiency in this paper. The first is by synthesizing pictures at a given time and a given position by using view interpolation and using them as reference pictures (view-interpolation prediction). In other words, we tried to compensate for geometry to obtain precise predictions. The second approach is to correct the luminance and chrominance of other views by using lookup tables to compensate for photoelectric variations in individual cameras. We implemented these ideas in H.264/AVC with inter-view prediction and confirmed that they worked well. The experimental results revealed that these ideas can reduce the number of generated bits by approximately 15% without loss of PSNR.

We propose a new multi-view video coding method using adaptive selection of motion/disparity compensation based on H.264/AVC. One of the key points of the proposed method is the use of view interpolation as a tool for disparity compensation by assigning reference picture indices to interpolated images. Experimental results show that significant gains can be obtained compared to the conventional approach that was often used

Temporal scalability is effective to adapt the bitstream adaptation for various capabilities of the video terminals and the delivery network, e.g. the processing speed of video terminals and the transmission rate, respectively. This technique uses the reference picture selection method on both the current layer and lower layers. The conventional scalability selects only from the last previous picture of the current layer and that of the first lower layer. This paper proposes a novel prediction scheme, the HRPS (hierarchical reference picture selection) method in which the reference picture is selected from more previous pictures in more layers, in order to improve coding efficiency, keeping the temporal scalability functionality. The proposed method is developed with a modification of the H.264. This paper demonstrates the effectiveness of the HRPS compared with the conventional temporal scalable methods.

Fractional pel motion compensation generally improves coding efficiency due to more precise motion accuracy and low path filtering effect in generating an image at fractional pel positions. In H.264, quarter pel motion compensation is applied, where the image at half pel position is generated by a 6 tap Wiener filter. And the adaptive interpolation filter technique, which adaptively changes filter characteristics for half pel positions has been proposed. That technique also changes the image at quarter pel positions, so it can be exploited to extend motion accuracy to be more precise. In this paper, a 3D motion vector coding (3DMVC) technique with block base adaptive interpolation filter (BAIF) is proposed. This paper also demonstrates the proposed method ensures filter data is successfully integrated into motion vector coding and outperforms the normal H.264.

Fractional pel motion compensation generally improves coding efficiency due to more precise motion accuracy and low path filtering effect in generating image at fractional pel positions. In H.264, quarter pel motion compensation is applied, where image at half pel position is generated by 6 tap Wiener filter. And the adaptive interpolation filter technique, which adaptively changes filter characteristics for half pel positions have been proposed. That technique also changes image at quarter pel positions, so it can be exploited to extent motion accuracy to be more precise. In this paper, 3D motion vector coding (3DMVC) technique with block base adaptive interpolation filter (BAIF) is proposed. This paper also demonstrates the proposed method achieves filter data is successfully integrated into motion vector coding and outperforms the normal H.264.

Post filtering is an effective tool for reducing coding artifacts in video coding. Most post-filtering tools are based on lowpass filtering, and an edge preserving filtering is useful because edges have an important effect on visual quality. An edge preserving filtering has an edge detection step, region detection step, and filtering step. The effectiveness of such filtering is much affected by the edge detection feature. Therefore the edge detection threshold should be carefully determined. We created a new method of determining the edge detection threshold of a post filter, that is based on the prefiltering features and encoding conditions. Computer simulations show that this method achieves the best visual quality for given prefiltering and encoding conditions.

The increasing processing speed of computers has made software-based video coding possible. Several reports have been published on increasing the processing speed of such video coding through improved software development techniques, and on increasing video coding speed by efficiently omitting the video coding process itself. However, the actual speed of software-based video coding depends on the implementation environments. Accordingly, under some environments of lower processing speed, the processor can not encode video data in real time, even if real-time coding is necessary. This has created a need to control a video coding process with depending on the processing speed. To address this need, we propose a new scheme for controlling motion estimation. This scheme, which we call “time adaptive motion estimation” (TAME), is based on the Successive Elimination Algorithm (SEA) method. It enables motion estimation to be completed within a pre-determined target time. Computer simulation results demonstrate that the proposed method achieves nearly the same quality as in the case where coding time is unlimited.