Eulerian video magnification (EVM) has progressed to magnify subtle motions with a target frequency even under the presence of large motions of objects. However, existing EVM methods often fail to produce desirable results in real videos due to (1) misextracting subtle motions with a non-target frequency and (2) collapsing results when large de/acceleration motions occur (e.g., objects suddenly start, stop, or change direction). To enhance EVM performance on real videos, this paper proposes a bilateral video magnification filter (BVMF) that offers simple yet robust temporal filtering. BVMF has two kernels; (I) one kernel performs temporal bandpass filtering via a Laplacian of Gaussian whose passband peaks at the target frequency with unity gain and (II) the other kernel excludes large motions outside the magnitude of interest by Gaussian filtering on the intensity of the input signal via the Fourier shift theorem. Thus, BVMF extracts only subtle motions with the target frequency while excluding large motions outside the magnitude of interest, regardless of motion dynamics. In addition, BVMF runs the two kernels in the temporal and intensity domains simultaneously like the bilateral filter does in the spatial and intensity domains. This simplifies implementation and, as a secondary effect, keeps the memory usage low. Experiments conducted on synthetic and real videos show that BVMF outperforms state-of-the-art methods.

In recent years, the human Digital Twin has been discussed as new technology. When we discuss a world in which one’s self-avatar autonomously performs social activities in cyberspace, the questions arise whether or not the behavior of the avatars feels like one’s own, and whether or not we can approve of the self-avatars’ social activities on behalf of ourselves. We define such feeling as the sense of self-individuality. In this study, we focused on the situation in which self-avatars perform presentations on behalf of ourselves to investigate the effect of the modification experience on the presentation motions by self-avatars on the sense of self-individuality. We conducted VR-based experiments in which the motion modification intervention was performed on self-avatars over eight weeks by 24 experiment participants. As a result, we found that the sense of self-individuality was improved as the number of modifications and interventions increased. However, we found that the intensity of motion modification did not correlate with the improvement of the sense of self-individuality in this experiment condition. We also found that the sense of self-individuality was reduced when others intervened in the motion. From these results, we clarified that the experience of motion modification on self-avatars is significant when designing the behavior of avatars acting on behalf of ourselves in human Digital Twin. Further investigation is required to clarify the effect of the long-term intervention on behavior to distinguish between the mere exposure effect.

What we wear (our clothes and wearable accessories) can represent our mood at the moment. We developed Emolleia to explore how to make aesthetic wears more expressive to become a novel form of non-verbal communication to express our emotional feelings. Emolleia is an open wearable kinetic display in form of three 3D printed flowers that can dynamically open and close at different speeds. With our open-source platform, users can define their own animated motions. In this paper, we described the prototype design, hardware considerations, and user surveys (n=50) to evaluate the expressiveness of 8 pre-defined animated motions of Emolleia. Our initial results showed animated motions are feasible to communicate different emotional feelings especially at the valence and arousal dimensions. Based on the findings, we mapped eight pre-defined animated motions to the reported, user-perceived valence, arousal and dominance and discussed possible directions for future work.

We propose an action quality assessment (AQA) method that can specifically assess target action quality with ignoring scene context, which is a feature unrelated to the target action. Existing AQA methods have tried to extract spatiotemporal features related to the target action by applying 3D convolution to the video. However, since their models are not explicitly designed to extract the features of the target action, they mis-extract scene context and thus cannot assess the target action quality correctly. To overcome this problem, we impose two losses to an existing AQA model: scene adversarial loss and our newly proposed human-masked regression loss. The scene adversarial loss encourages the model to ignore scene context by adversarial training. Our human-masked regression loss does so by making the correlation between score outputs by an AQA model and human referees undefinable when the target action is not visible. These two losses lead the model to specifically assess the target action quality with ignoring scene context. We evaluated our method on a diving dataset commonly used for AQA and found that it outperformed current state-of-the-art methods. This result shows that our method is effective in ignoring scene context while assessing the target action quality.

The tactile and kinesthetic sensation of pushing a button is usually lost when interacting with modern devices like touchscreens and/or virtual reality platforms. We present FingerFlex, a standalone glove wearable actuating the metacarpophalangeal joint (MCP) of each finger via shape memory alloy (SMA). SMA actuation is subtle, silent, and light, making it ideal for actuation of the fingers which we use to simulate the sensation of pressing a button.

This paper presents a new generalized (or ray-pixel, raxel) camera calibration algorithm for camera systems involving distortions by unknown refraction and reflection processes. The key idea is use of intersections of calibration planes, while conventional methods utilized collinearity constraints of points on the planes. We show that intersections of calibration planes can realize a simple linear algorithm, and that our method can be applied to any ray-distributions while conventional methods require knowing the ray-distribution class in advance. Evaluations using synthesized and real datasets demonstrate the performance of our method quantitatively and qualitatively.

View synthesis prediction is critical for efficient compression of 3D video, which consists of multiview video and depth maps. However, its performance is limited in practical situations since it is necessary to use erroneous depth information and to perform block-based compensation instead of pixel-based warping. This paper proposes a novel view synthesis prediction scheme where motion field is synthesized by utilizing coarse disparity filed derived from erroneous depth information. As part of the proposed depth-based motion field synthesis, occlusion-aware backward mapping and 3D motion field warping are performed. In order to improve prediction performance with block-based compensation, an adaptive prediction sample generation that utilizes both temporal and inter-view correlations is also proposed. Experiments show that the proposed scheme achieves average bitrate reductions of 1.38% and 1.19% for coded views and synthesized views. The maximum gain is 11.57% for dependent view.

Inter-view residual prediction and advanced residual prediction (ARP) are efficient tools for coding dependent views of 3D video. It can predict the residue of motion compensated prediction (MCP) using the additional derived disparity vector. However, its coding performance depends on the accuracy of disparity vector derivation. In this paper, we propose the improved disparity vector derivation scheme for ARP. In the proposed scheme, the disparity vector is derived from the corresponding block in the reference picture. And that corresponding block is pointed by the same as MCP. Moreover, the disparity vector can be derived not only from the current reference block, but also from the blocks on the all other reference pictures included in the current reference picture lists. In addition, the disparity vector can be derived from both of the blocks predicted by disparity compensated prediction (DCP) and the blocks predicted by MCP and ARP because the derived disparity vector for ARP is stored in additional disparity vector field. Experimental results show that 0.2% bitrate reduction of synthesized views, and up to 0.5% for each dependent views in the reference software of 3D-HEVC.

Advanced multiview video systems are able to generate intermediate viewpoints of a 3-D scene. To enable low-complexity free view generation, texture and its associated depth are used as input data for each viewpoint. To improve the coding efficiency of such content, view synthesis prediction (VSP) is proposed to further reduce interview redundancy in addition to traditional disparity compensated prediction. This paper describes and analyzes rate-distortion optimized VSP designs, which were adopted in the 3-D extensions of both Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC). In particular, we propose a novel backward-VSP scheme using a derived disparity vector, as well as efficient signalling methods in the context of AVC and HEVC. In addition, we put forward a novel depth-assisted motion vector prediction method to optimize the coding efficiency. A thorough analysis of coding performance is provided using different VSP schemes and configurations. Experimental results demonstrate average bit rate reductions of 2.5% and 1.2% in AVC and HEVC coding frameworks, respectively, with up to 23.1% bit rate reduction for dependent views.

View synthesis prediction has been studied as an efficient inter-view prediction scheme. Existing view synthesis prediction schemes fall into two types according to the pixel warping direction. While backward warping based view synthesis prediction enables block-based processing, forward warping based view synthesis prediction can handle occlusions properly. This paper proposes a two-step warping based view synthesis prediction; a virtual depth map is first generated by forward warping, and then prediction signals are generated by block-based backward warping using the virtual depth map. The technique of backward-warping-aware depth inpainting is also proposed. Experiments show that the proposed VSP scheme can achieve the decoder runtime reductions of about 37% on average with slight bitrate reductions relative to the conventional forward warping based VSP. Compared to the conventional backward warping based VSP, the proposed method reduces the bitrate for the synthesized views by up to 2.9% and about 2.2% on average.

Video with depth is attracting attention as a promising approach to future 3D video. Unlike conventional video coding technologies, video with depth should take the correlations between video and depth into account. This paper proposes a novel bi-prediction scheme for video coding that exploits the structure similarity between video and depth. In the proposed scheme, a prediction image is generated by the pixel-wise weighted average of motion compensated reference images, and weights are decided by considering pixel similarities in the depth domain. The proposed pixel-wise weighted bi-prediction offers implicit depth-based block partitioning with consideration of the pixel correlations between predicted and reference blocks. Compared to using explicit, arbitrary shaped block partitioning, the proposed method offers better prediction for non-rigid objects, and requires no increase in motion information. Experiments show that the proposed method can achieve up to 17% bitrate reduction for bi-predictive frames and about 8% on average for 6 sequences relative to the existing biprediction scheme.

Pixel domain methods for content-aware image retargeting have a performance limitation in that they distort important objects when an image is shrunken too much. To address this problem, we have designed and developed a novel 3D space domain image retargeting which rearranges objects with occlusions and deforms objects. With this method image formation is modeled as a quasi-3D space. The method also reduces image width by shrinking a part of the image space. Instead of directly optimizing the position and shape of objects, it closes objects by shrinking the gap space between them and deforms objects by shrinking spaces that include parts of them. It is capable of being extended to higher dimension problems such as multi-view image retargeting because it needs to optimize only one quasi-3D space whether the input is ordinary images or more complex ones such as video and stereo images. We have also designed a simple algorithm that makes use of dynamic programming. An experiment we conducted demonstrated the algorithm's effectiveness.

In Japanese, every content word has its own mora-based H/L pitch pattern when it is uttered in isolation, called accent type. When reading out a written sentence, however, this lexical H/L pattern is often changed according to the context, known as word accent sandhi. In our previous work, an accent sandhi predictor was developed using CRF [1], and in this paper, the predictor is improved through feature engineering especially fo-cusing on phrases including numerals and those including loan-words. This is because our previous work showed that the pre-diction performance was relatively low for those phrases. To optimize the features used for CRF, it is critical to take into ac-count the mechanism of word accent sandhi. We review linguis-tic and technical literature that attempted to characterize accent sandhi in the phrases including numerals and loanwords and, by reflecting these characteristics, the features are re-designed. Experiments show that the proposed predictor improved the per-formance relatively by 37 % and 41%, respectively. Index Terms: word accent sandhi, accent nucleus, text-to-speech, Japanese education, rule-based, corpus-based, CRF

This paper proposes a novel prediction scheme for depth map coding. We utilize the fact that depth values are largely dependent on objects and one small block consists of only a small number of objects at most. The proposed method approximates each block with one palette and one object shape map. The palette consists of two representative depth values for foreground object and background object in the target block. The object shape map expresses which object exists at each pixel. These data are encoded only at the blocks where the proposed method is used. In order to reduce the bitrates required to transfer palettes and object shape maps, the proposed method enables their prediction by utilizing spatial and temporal correlations. Experiments are conducted on three kinds of depth maps; estimated from multiview images, captured by special sensors, and generated by computer. The results show that the proposed method reduces the bitrate by up to 40% and about 20% on average for 13 sequences relative to the MPEG-4 AVC/H.264.

View synthesis prediction has been studied to achieve efficient inter-view prediction. Existing view synthesis prediction methods synthesize predicted pictures by using decoded pictures from the other views and geometric information of the scene. However, these conventional methods have no ability to compensate the inter-view difference in image signals caused by individual camera characteristics and the nonLambert reflection of objects. This paper proposes adaptive appearance compensated view synthesis prediction; it uses adaptive filters to compensate not only the differences in color and brightness but also the differences in the spatial frequency of images. The object-dependent local differences are also compensated efficiently by estimating the optimal compensation parameters block by block at the decoder. Experiments show that the proposed method reduces the bitrate by up to 40% relative to H.264/AVC multiview video coding, and 26% relative to the conventional view synthesis prediction.

The robot leg for rehabilitation education requires the characteristics of the output force and the stiffness of the human leg to be mimicked correctly. There have been several efforts to mimic the motion characteristic of living organism by taking into account the arrangement of muscles in the mechanism of the robot. The present paper introduces a robot leg with a human-like two-joint link mechanism that is operated with the muscle coordinate system. The force and stiffness characteristics will enable the proposed robot leg to be used as a training robot for rehabilitation education.

In order to represent 3D space, we have proposed to use the representation that consists of multi-view video plus a single view depth map. This format is backward compatible with the MPEG-C Part 3 (a.k.a. ISO/IEC 23002-3). This paper proposes a coding scheme on this 3D space representation that is efficient even if low delay decoding functionality is required. We apply the residual prediction framework to the view synthesis prediction errors. Experiments show that the proposed scheme achieves up to about 7.7% bitrate reduction compared to multi-view video coding with disparity compensation, even if the depth map video is added. Furthermore, the proposed scheme doesn’t require any syntax changes to the conventional video coding standard and it needs few modifications on the circuits of conventional video codecs; it might be possible to reuse almost all of the components. The backward compatibility and reusability achieved by the proposed scheme are quite important to reduce manufacturing costs and time to market.

A real-time MVC viewer for free-viewpoint navigation is demonstrated. MVC, a video coding standard for multi-view video, is an extension of the state-of-the-art H.264/AVC. Realizing parallel decoding and view synthesis is essential to develop real-time viewers. A new decoder architecture is introduced for the proposed viewer. In addition, a file format for MVC is introduced. View synthesis is performed entirely within GPU. It is demonstrated that the MVC viewer can decode a MVC bitstream and generate a virtual camera image in real-time.

Multiview video coding demands high compression rates as well as view scalability, which enables the video to be displayed on a multitude of different terminals. In order to achieve view scalability, it is necessary to limit the inter-view prediction structure. In this paper, we propose a new multiview video coding scheme that can improve the compression efficiency under such a limited inter-view prediction structure. All views are divided into two groups in the proposed scheme: base view and enhancement views. The proposed scheme first estimates a view-dependent geometry of the base view. It then uses a video encoder to encode the video of base view. The view-dependent geometry is also encoded by the video encoder. The scheme then generates prediction images of enhancement views from the decoded video and the view-dependent geometry by using image-based rendering techniques, and it makes residual signals for each enhancement view. Finally, it encodes residual signals by the conventional video encoder as if they were regular video signals. We implement one encoder that employs this scheme by using a depth map as the view-dependent geometry and 3-D warping as the view generation method. In order to increase the coding efficiency, we adopt the following three modifications: (1) object-based interpolation on 3-D warping; (2) depth estimation with consideration of rate-distortion costs; and (3) quarter-pel accuracy depth representation. Experiments show that the proposed scheme offers about 30% higher compression efficiency than the conventional scheme, even though one depth map video is added to the original multiview video.

Neighboring views must be highly correlated in multiview video systems. We should therefore use various neighboring views to efficiently compress videos. There are many approaches to doing this. However, most of these treat pictures of other views in the same way as they treat pictures of the current view, i.e., pictures of other views are used as reference pictures (inter-view prediction). We introduce two approaches to improving compression efficiency in this paper. The first is by synthesizing pictures at a given time and a given position by using view interpolation and using them as reference pictures (view-interpolation prediction). In other words, we tried to compensate for geometry to obtain precise predictions. The second approach is to correct the luminance and chrominance of other views by using lookup tables to compensate for photoelectric variations in individual cameras. We implemented these ideas in H.264/AVC with inter-view prediction and confirmed that they worked well. The experimental results revealed that these ideas can reduce the number of generated bits by approximately 15% without loss of PSNR.

We propose a new multi-view video coding method using adaptive selection of motion/disparity compensation based on H.264/AVC. One of the key points of the proposed method is the use of view interpolation as a tool for disparity compensation by assigning reference picture indices to interpolated images. Experimental results show that significant gains can be obtained compared to the conventional approach that was often used