Abstract Video summarisation can relieve the pressure on video storage, transmission, archiving, and retrieval caused by the explosive growth of online videos in recent years. Most existing supervised video summarisation methods use convolutional neural network (CNN) or recurrent neural network (RNN) to model the temporal dependencies between video frames or video shots. CNN mainly focuses on local information, and RNN loses long‐term information when the input sequence is long, both of which have limited ability to obtain long‐range memory in the video. Therefore, a recurrent unit augmented memory network (RUAMN) for video summarisation is proposed, which effectively utilises the long‐term memory extraction ability of the end‐to‐end memory network (MemN2N) and solves the problem that MemN2N is insensitive to temporal sequence information. At the same time, the proposed RUAMN enhances the process of memory update between multiple computational steps (hops), and finally generates a meaningful video summarisation result. Specifically, the proposed RUAMN is mainly composed of the input module, the global‐and‐local sampling, the memory module and the output module. The input module uses a bidirectional GRU to obtain the forward and backward information of each video frame. Then the global‐and‐local sampling performs global sampling and local sampling on the output sequence of the input module respectively to obtain several shorter sequences, so that the memory modules can capture the fine‐grained relationship features between video frames more effectively. The memory module extracts the long‐term memory information in the feature sequence, and finally the frame‐level importance scores are predicted by the output module. Extensive experiments on benchmark datasets, that is, TVSum and SumMe, demonstrate the superiority of our method over several state‐of‐the‐art supervised video summarisation methods.

Product placement, also called advertisement embedding, is to place some specific products in an image or a video, which may attract consumers to buy their products. However, adding advertisement objects in images is difficult, because where to add the product and how to fuse the background must be concerned. In this paper, to overcome this issue, we present a novel hierarchical framework with conditional generative adversarial network to add advertisement object in all kinds of scene images. The key point of our framework is leaning the relation between surrounding and products. To generate the products more realistic and to make the detail information such characters and logo more clear, we propose local-global discriminator. Our experiment demonstrates the effectiveness of our proposed method on different scene images. Detail codes and results are available in GitHub. https://github.com/happyyuwei/ProductPlacement.

This paper proposes a modified just noticeable depth difference (JNDD) (JNDiD) model in perceived depth space. The JNDiD model improves the accuracy of current JNDD models by also taking into account the blurriness caused by the change in accommodation when a 3-D object is perceived far from the screen. This change in accommodation is a result of the convergence-accommodation conflict, in which convergence plays a leading role. In the JNDiD model described in this paper, the JNDD threshold is the addition between a base threshold and an additional threshold. Adapted to different blurring effects in three depth regions, the additional threshold is defined as a three-piecewise linear function. The proposed model also attempts to separate the JNDD modeling from the display modeling and viewing conditions in perceived depth space, making it applicable for different types of displays given their specific model parameters. With the help of model parameter transfer functions, the JNDiD model in perceived depth space and the corresponding one in any specific stimulated depth space can be easily transformed to each other. Experimental results demonstrate the effectiveness and superiority of the JNDiD model in perceived depth space.

In this paper, we propose a no-reference (NR) stereo image quality assessment metric by learning gradient dictionary-based color visual characteristics. To be specific, firstly, since human eyes are highly sensitive to the structure of images, the gradient magnitude (GM) and gradient orientation (GO) are extracted from left and right views of stereo image, meanwhile, the difference map is obtained. Considering the influence of color distortion, images are decomposed into RGB channels to be processed respectively, and we get the local gradient of the color image by adding up the RGB gradient vectors. Constructively, the gradient dictionary is generated, which is different from traditional image dictionary. All quality-aware features are extracted by joint sparse representation. Afterwards, to avoid over-fitting, the principal component analysis (PCA) is applied to optimize the quality-aware features. Finally, all features are fed into the trained support vector regression (SVR) model to predict the objective score. The experimental results show that the proposed metric always achieves high consistency with human subjective assessment for both symmetric and asymmetric distortions.

3D-high efficiency video coding (HEVC) is developed for the compression of the multi-view video plus depth format, which is based on the latest generation of video coding standard, HEVC. It further adopts several new intra prediction modes, depth-modeling modes (DMMs) in intra candidate modes for a better representation of edges in depth maps, which introduces a drastic increase in the computational complexity. The procedure of depth intra mode decision together with DMMs and existing intra modes is a very time consuming part due to huge complexity of full rate distortion (RD) cost calculation. In this paper, a low complexity intra mode selection algorithm is proposed to reduce complexity of depth intra prediction in both intra-frames and inter-frames. An experimental analysis is first performed to study the inter-view correlation and the inter-component (texture video and its associated depth) correlation in intra coding information such as the intra mode and RD cost. All intra modes available in 3D-HEVC are classified into three activity classes assigned with different mode-weight factors, and the coding mode complexity of a coding unit (CU) is defined according to the intra mode information from available spatiotemporal, inter-view, and inter-component neighboring coded CUs. The coding mode complexity analysis is utilized to assign different candidate intra modes for different types of CUs. The optimal intra prediction mode and the RD cost value in current CU depth level are further used to skip unnecessary intra prediction sizes. Experimental results show that the proposed fast depth intra coding algorithm achieves 61% complexity reduction on intra prediction, while incurring a 0.2% Bjontegaard metric increase for coded and synthesized views compared to the test model of 3D-HEVC.

In this paper, we propose a novel full reference stereoscopic image quality assessment (FR-SIQA) metric by utilizing SSIM-based binocular perceptual model. The goal is to predict the perceptual quality of a stereoscopic image via jointly considering the qualities of cyclopean image and the difference image. Specifically, we first apply the contrast sensitivity filtering to both the reference and distorted stereo pairs. Constructively, a new cyclopean image is generated by considering binocular perceptual model and binocular rivalry simultaneously. Finally, the overall quality score of a testing stereoscopic image is predicted by combining the qualities of its cyclopean image and difference image. Experimental results show that the proposed metric achieves high consistency with human subjective assessment and outperforms several the state-of-the-art FR-SIQA methods.

Based on the related psychological and physiological advancement, a just noticeable disparity difference (JNDiD) model for 3D displays under the assumption that eyes converge at the virtual object is presented in this paper. Specifically, according to whether the surface of the virtual object is perceived to be blurred, the perceived depth space is divided into three regions. Furthermore, a three-phase linear function considering the accommodation convergence mismatch in different depth regions is employed to build the JNDiD model. Experimental results demonstrate the effectiveness and superiority of our method compared with the state-of-the-art just noticeable depth difference (JNDD) model for 3D displays.

This paper presents a parallax-aware image stitching approach under large occlusion/disocclusion. Different from previous research, we explore the image stitching issue in a parallax-aware perspective via local alignment. Specifically, we first label each feature point with a probability of being large parallax by developing a graph-based optimization framework. Afterwards, an integer programming model is built to pick out a group of feature matches free from parallax in a local region. Finally, by enforcing a stitching seam passing through such a locally aligned area, we are able to generate a high-quality stitching result under large parallax. Experimental results demonstrate the effectiveness and superiority of the proposed parallax-aware approach.

Abstract It is a challenging task for ordinary users to capture selfies with a good scene composition, given the limited freedom to position the camera. Creative hardware (e.g., selfie sticks) and software (e.g., panoramic selfie apps) solutions have been proposed to extend the background coverage of a selife, but to achieve a perfect composition on the spot when the selfie is captured remains to be difficult. In this paper, we propose a system that allows the user to shoot a selfie video by rotating the body first, then produce a final panoramic selfie image with user‐guided scene composition as postprocessing. Our key technical contribution is a fully Automatic, robust multi‐frame segmentation and stitching framework that is tailored towards the special characteristics of selfie images. We analyze the sparse feature points and employ a spatial‐temporal optimization for bilayer feature segmentation, which leads to more reliable background alignment than previous image stitching techniques. The sparse classification is then propagated to all pixels to create dense foreground masks for person‐background composition. Finally, based on a user‐selected foreground position, our system uses content‐preserving warping to produce a panoramic seflie with minimal distortion to the face region. Experimental results show that our approach can reliably generate high quality panoramic selfies, while a simple combination of previous image stitching and segmentation approaches often fails.

We present a fully automatic system for extracting the semantic structure of a typical academic presentation video, which captures the whole presentation stage with abundant camera motions such as panning, tilting, and zooming. Our system automatically detects and tracks both the projection screen and the presenter whenever they are visible in the video. By analyzing the image content of the tracked screen region, our system is able to detect slide progressions and extract a high-quality, non-occluded, geometrically-compensated image for each slide, resulting in a list of representative images that reconstruct the main presentation structure. Afterwards, our system recognizes text content and extracts keywords from the slides, which can be used for keyword-based video retrieval and browsing. Experimental results show that our system is able to generate more stable and accurate screen localization results than commonly-used object tracking methods. Our system also extracts more accurate presentation structures than general video summarization methods, for this specific type of video.

This paper presents an automatic and efficient system for extracting dynamic objects of interest from videos. We take advantage of a saliency map and an optimization-based segmentation algorithm to extract the foreground objects automatically in some key frames. Then, the segmentation results in those key frames are propagated to other frames via an error map-based propagation scheme. Finally, a Bayesian matting-based refinement approach is employed to to handle the topology changes. Experiments show that our system is able to generate high quality results at a low computation cost.

Depth map, as important auxiliary information in 3D procession, is used to synthesize virtual view rather than exhibition. Inspired by this, a synthesis-guided depth super resolution (SGDSR) algorithm is proposed. Employing the synthesis error between virtual view and corresponding original one as the criteria, the best super-resolved result is selected among numerous candidate super resolution (SR) results. To fully exploit varying property within different regions of an image, a patch-based SGDSR is further devised in this paper. Experimental results demonstrate the effectiveness of our method subjectively and objectively on both single view and two views platform based on depth-image-based rendering (DIBR).

This paper presents a data-driven approach for facial expression retargeting in video, i.e., synthesizing a face video of a target subject that mimics the expressions of a source subject in the input video. Our approach takes advantage of a pre-existing facial expression database of the target subject to achieve realistic synthesis. First, for each frame of the input video, a new facial expression similarity metric is proposed for querying the expression database of the target person to select multiple candidate images that are most similar to the input. The similarity metric is developed using a metric learning approach to reliably handle appearance difference between different subjects. Secondly, we employ an optimization approach to choose the best candidate image for each frame, resulting in a retrieved sequence that is temporally coherent. Finally, a spatio-temporal expression mapping method is employed to further improve the synthesized sequence. Experimental results show that our system is capable of generating high quality facial expression videos that match well with the input sequences, even when the source and target subjects have big identity difference. In addition, extensive evaluations demonstrate the high accuracy of the learned expression similarity metric and the effectiveness of our retrieval strategy.

This paper presents a method to synthesize a realistic facial animation of a target person, driven by a facial performance video of another person. Different from traditional facial animation approaches, our system takes advantage of an existing facial performance database of the target person, and generates the final video by retrieving frames from the database that have similar expressions to the input ones. To achieve this we develop an expression similarity metric for accurately measuring the expression difference between two video frames. To enforce temporal coherence, our system employs a shortest path algorithm to choose the optimal image for each frame from a set of candidate frames determined by the similarity metric. Finally, our system adopts an expression mapping method to further minimize the expression difference between the input and retrieved frames. Experimental results show that our system can generate high quality facial animation using the proposed data-driven approach.