Personalized TTS is an exciting and highly desired application that allows users to train their TTS voice using only a few recordings. However, TTS training typically requires many hours of recording and a large model, making it unsuitable for deployment on mobile devices. To overcome this limitation, related works typically require fine-tuning a pre-trained TTS model to preserve its ability to generate high-quality audio samples while adapting to the target speaker’s voice. This process is commonly referred to as "voice cloning." Although related works have achieved significant success in changing the TTS model’s voice, they are still required to fine-tune from a large pre-trained model, resulting in a significant size for the voice-cloned model. In this paper, we propose applying trainable structured pruning to voice cloning. By training the structured pruning masks with voice-cloning data, we can produce a unique pruned model for each target speaker. Our experiments demonstrate that using learnable structured pruning, we can compress the model size to 7 times smaller while achieving comparable voice-cloning performance.

User-defined keyword spotting is a task to detect new spoken terms defined by users. This can be viewed as a few-shot learning problem since it is unreasonable for users to define their desired keywords by providing many examples. To solve this problem, previous works try to incorporate self-supervised learning models or apply meta-learning algorithms. But it is unclear whether self-supervised learning and meta-learning are complementary and which combination of the two types of approaches is most effective for few-shot keyword discovery. In this work, we systematically study these questions by utilizing various self-supervised learning models and combining them with a wide variety of meta-learning algorithms. Our result shows that HuBERT combined with Matching network achieves the best result and is robust to the changes of few-shot examples.

For showing 2D video contents on 3DTVs, 2D-to-3D conversion is required to convert 2D contents to 3D ones. In general, the conversion process consists of depth map generation, which estimates the 3D geometry of the scene, and rendering, which produces output stereo images. We propose a depth map refinement algorithm which uses adaptive decimation and guided interpolation to refine the depth map. The proposed approach eliminates unnecessary textures and keeps object boundaries on the depth map. In the refined depth map, the transition of depth values within an object is smooth, the object boundaries on the depth map are aligned to those on the input image, and the foreground can be clearly separated from the background. These characteristics on the depth map help to achieve better 3D visual perception. Compared to the state-of-the-art approaches, the blurriness and object boundary alignment are easier to be adjusted by the proposed two-stage operations, and the computational complexity of the proposed algorithm is much lower.

High Efficiency Video Coding (HEVC) is a new international video coding standard that has been developed by the Joint Collaborative Team on Video Coding (JCT-VC). In this paper, an overview of the motion vector coding techniques for HEVC is presented. Our three proposed coding tools for the motion vector predictor (MVP) in the Inter, Skip and Merge modes of HEVC are also presented, which includes a new location of the temporal MVP, a priority-based derivation method of spatial MVPs, and a derivation method of temporal MVPs. A combination of these three tools can achieve on average 1.3%, 1.8%, 1.2% and 2.2% bit rate reductions for high efficiency random access, low complexity random access, high efficiency low delay, and low complexity low delay, respectively.

Hand tracking is a fundamental task in a gesture recognition system. Most previous works tracked the hand position on color images and relied heavily on skin color information. However, color information is very vulnerable to lighting variations and skin color varies across difference human races. Furthermore, one can not effectively discriminate faces or other skin-color-like objects from hands when using skin color detection. In this paper, we propose a hand tracking algorithm that uses depth images only, and also a hand click detection method to initialize the hand tracking automatically. We show that depth images suffice and are advantageous to real-time hand tracking. A region growing technique is applied to segment the hand region on depth images. Then a mean-shift based algorithm accurately locates the hand center in the segmented hand region. The experimental results show that the proposed tracking algorithm runs at 300+ FPS, and the average error of the tracked 3D hand positions is less than 1 centimeter. The proposed method enables a plethora of potential applications to natural Human-Computer Interaction (HCI), and is adequate for embedded systems of consumer electronics because of its low complexity and low bandwidth requirement.

In surveillance applications, keyframes are usually used to summarize important video content. However, traditional keyframe extracting methods select video frames as keyframes directly from the input video, and the information in these selected video frames is scattered and the users cannot perceive how the events happened in the original video easily. In this paper, we propose a new type of keyframe (called an “augmented 3-D keyframe”) that is a more meaningful and compact keyframe, augmented with representative objects, important contents (human faces, license plates, and so on), motion information (including trajectories and simple movement situations), and some marks of the moving objects extracted from a surveillance video clip captured by a static camera. The innovation is that we include simple 3-D geometric information to generate this augmented 3-D keyframe without reconstructing a complete 3-D scene. Testing results of the proposed augmented 3-D keyframe are presented and compared with results obtained by the traditional keyframe approach.

Image-based rendering has been successfully used to display 3-D objects for many applications. A well-known example is the object movie, which is an image-based 3-D object composed of a collection of 2-D images taken from many different viewpoints of a 3-D object. In order to integrate image-based 3-D objects into a chosen scene (e.g., a panorama), one has to meet a hard challenge--to efficiently and effectively remove the background from the foreground object. This problem is referred to as multiview images (MVIs) segmentation. Another task requires MVI segmentation is image-based 3-D reconstruction using multiview images. In this paper, we propose a new method for segmenting MVI, which integrates some useful algorithms, including the well-known graph-cut image segmentation and volumetric graph-cut. The main idea is to incorporate the shape prior into the image segmentation process. The shape prior introduced into every image of the MVI is extracted from the 3-D model reconstructed by using the volumetric graph cuts algorithm. Here, the constraint obtained from the discrete medial axis is adopted to improve the reconstruction algorithm. The proposed MVI segmentation process requires only a small amount of user intervention, which is to select a subset of acceptable segmentations of the MVI after the initial segmentation process. According to our experiments, the proposed method can provide not only good MVI segmentation, but also provide acceptable 3-D reconstructed models for certain less-demanding applications.

Research on creating a large, high-resolution, low-cost display system has become increasingly important due to the growing desire in many fields for bigger and better displays. The goal of this research is to build a seamless large-scale display system by tiling multiple projectors with the help of a pan-tilt-zoom camera. In order to achieve this goal, we took two tasks into consideration: geometric calibration and photometric calibration. Compared to the previous work, our method for geometric calibration is more accurate, thanks to the much higher resolution images acquired by combining several zoom-in images and doing lens correction for the camera at first. For photometric calibration, we achieve color uniformity among projectors by utilizing the same PTZ camera, which is calibrated once with a colorimeter in a factory. Furthermore, we adopted the technique of producing a high dynamic range image from several images of different exposures to increase the measurement accuracy of the camera. In our experiments, the average error of photometric measurement achieved by our method is less than the mean perceptibility tolerance. Our method has great potential for many applications that require large and high-resolution displays

Watershed segmentation has recently become a popular tool for image segmentation. There are two approaches to implementing watershed segmentation: immersion approach and toboggan simulation. Conceptually, the immersion approach can be viewed as an approach that starts from low altitude to high altitude and the toboggan approach as an approach that starts from high altitude to low altitude. The former seemed to be more popular recently (e.g., Vincent and Soille), but the latter had its own supporters (e.g., Mortensen and Barrett). It was not clear whether the two approaches could lead to exactly the same segmentation result and which approach was more efficient. In this paper, we present two "order-invariant" algorithms for watershed segmentation, one based on the immersion approach and the other on the toboggan approach. By introducing a special RIDGE label to achieve the property of order-invariance, we find that the two conceptually opposite approaches can indeed obtain the same segmentation result. When running on a Pentium-III PC, both of our algorithms require only less than 1/30 s for a 256 x 256 image and 1/5 s for a 512 x 512 image, on average. What is more surprising is that the toboggan algorithm, which is less well known in the computer vision community, turns out to run faster than the immersion algorithm for almost all the test images we have used, especially when the image is large, say, 512 x 512 or larger. This paper also gives some explanation as to why the toboggan algorithm can be more efficient in most cases.

Abstract Our goal is to augment a panorama with object movies in a visually 3D‐consistent way. Notice that a panorama is recorded as one single 2D image and an object movie (OM) is composed of a set of 2D images taken around a 3D object. The challenge is how to integrate the above two sources of 2D images in a 3D‐consistent way so that the user can easily manipulate object movies in a panorama. To solve this problem, we adopt a purely image‐based approach that does not have to reconstruct the geometric models of the 3D objects to be inserted in the panorama. A critical issue of this method is how to generate the novel views required for showing an OM in different places of a panorama, and we have proposed a view morphing technique, called t‐DBVM, to solve this problem. Our experiments have shown that this purely image‐based approach can effectively generate visually convincing OM‐augmented panoramas. This method has great potential for many applications that require integration of panoramas and object movies, such as virtual malls, virtual museum, and interior design. Copyright © 2002 John Wiley & Sons, Ltd.