Significant progress has been made with deep neural networks recently. Sharing trained models of deep neural networks has been a very important in the rapid progress of research and development of these systems. At the same time, it is necessary to protect the rights to shared trained models. To this end, we propose to use digital watermarking technology to protect intellectual property and detect intellectual property infringement in the use of trained models. First, we formulate a new problem: embedding watermarks into deep neural networks. Second, we propose a general framework for embedding a watermark in model parameters, using a parameter regularizer. Our approach does not impair the performance of networks into which a watermark is placed because the watermark is embedded while training the host network. Finally, we perform comprehensive experiments to reveal the potential of watermarking deep neural networks as the basis of this new research effort. We show that our framework can embed a watermark during the training of a deep neural network from scratch, and during fine-tuning and distilling, without impairing its performance. The embedded watermark does not disappear even after fine-tuning or parameter pruning; the watermark remains complete even after 65% of parameters are pruned.

Although telepresence robots that act as proxies for telecommuters are commercially available, few of them are designed for informal communication, which requires great social skills. In this work, we develop a novel robot system that automatically tracks a passing by person(called walker) in the office to simulate basic social skills for triggering informal communication at a distance. Moreover, we conduct an evaluation experiment with 70 participants in an unconstrained setting (a company's lounge) and confirm that automatic tracking of a walker makes him/her more aware of the telecommuter's mediated presence and feel more watched in comparison to traditional systems. However, we also observe that the interaction rate has not been improved, which infers that additional social skills are required such as an appropriate response from the robot.

Geometric verification with epipolar geometry often results in a high score for an incorrect image pair due to ambiguity in its geometric constraints. The ambiguity is caused by a high degree of freedom in the epipolar geometry and a weak constraint from the fitting between a point and a line. In order to mitigate the ambiguity, we propose to filter geometrically inconsistent components, namely correspondences, a sample, a model, and inliers in a RANSAC-based geometric verification. For the filtering, we introduce novel semi-2D constraints whose geometric constraint is weaker than full-2D constraint, but stronger than pure-epipolar constraint. Additionally, an advantage of the proposed approach is that it requires only an image pair instead of neither additional information nor prior learning. Experiments on the public dataset containing 3D object images show that the proposed approach improves the true positive rate when the false positive rate is low, and greatly reduces computational time for the geometric verification of both a correct image pair and an incorrect image pair.

In this paper, we propose an online 3D person tracking method and an attribute estimation method with pedestrian dead reckoning (PDR). For person tracking, we employ a structured prediction approach, which extends the Struck algorithm. Although the main stream of visual object tracking, including Struck, utilizes only 2D information in image coordinates, it is difficult to track object correctly because of changes in the scale and angle of the target. In contrast, our classifier adaptively learns structural relationship in world coordinates and in image coordinates using Structured SVM. Furthermore, we combine visual tracking results and sensor trajectories based on PDR. Our method estimates a person attribute whether insider like a sales staff, or outsider like a customer. According to experimental results, the proposed method outperforms the existing methods regarding the quality of localization. In addition, experimental results show that our method can estimate the attribute at a ratio of 0.84.

Multimodal representation of conversational agents requires accurate synchronization of gesture and speech. For this purpose, we investigate the important issues in synchronization as a practical guideline for our algorithm design through a precedent case study and propose a two-step synchronization approach. Our case study reveals that two issues (i.e. duration and timing) play an important role in the manual synchronizing of gesture with speech. Considering the synchronization problem as a motion synthesis problem instead of a behavior scheduling problem used in the conventional methods, we use a motion graph technique with constraints on gesture structure for coarse synchronization in a first step and refine this further by shifting and scaling the motion in a second step. This approach can successfully synchronize gesture and speech with respect to both duration and timing. We have confirmed that our system makes the creation of attractive content easier than manual creation of equal quality. In addition, subjective evaluation has demonstrated that the proposed approach achieves more accurate synchronization and higher motion quality than the state-of-the-art method.

In the field of affect recognition, most researchers have focused on using multimodal fusion while temporal fusion techniques have yet to be adequately explored. This paper demonstrates that a powerful temporal fusion approach can significantly improve the performance of affect recognition. As a typical approach in state-of-the-art methods, the segment-based affect recognition technique from body movements is used as a baseline, whereby a body movement is parsed into a sequence of motion clips (called segments) and all segments are treated as being the same using a majority voting strategy. Our basic idea is that different types of segments have different influences/weights in recognizing the affective state. To verify this idea, an entropy-based method is proposed to estimate the segment weights in supervised and unsupervised learning. Furthermore, the recognition results from all segments are fused with the segment weights using the sum rule. Our experimental results on a public data set demonstrate that the segment weights can greatly improve the percentage of correctness (accuracy) from the baseline.

The mood of an image is typically conveyed by the combination of colors. Such a combination of colors is called a color theme. In this paper, we propose a color transformation method based on color theme that can i) control the ratios of assigned colors while ii) maintaining the spatial coherence of regions. In our color transformation method, one color in a color theme is assigned to the pixels in an image and the pixels are transformed toward the corresponding color. We consider the color assignment problem as a labeling problem. To ensure that i) and ii) above can be achieved, we introduce offsets that are added to labeling costs in the label optimization process. The offsets are determined according to the concept of maximally stable extremal regions (MSER) in order to preserve spatial coherence. Experimental results show that our approach is effective for the control of the color ratios and preserving the coherence of the regions.

In this paper, a new reranking approach is proposed to refine the results obtained with a bag-of-visual words (BoVW) image retrieval method. First, a simple but effective criterion to reject unreliable feature matches is proposed, where the information of nearest neighbors from a large dataset is used to accurately estimate feature density. Second, by adopting a product quantization-based nearest neighbor method in both the voting and reranking steps, it becomes possible to reuse the information obtained in the BoVW method in the reranking step. Finally, a density ratio-based scoring method is naturally integrated to calculate a new score from inliers.

An affective style of human motion is essential for human computer interaction using embodied conversational agents. Motion synthesis for affective agents generates a skeletal motion in a particular affective style (briefly called affective motion in this paper) from an input neutral motion. This appeals to the user but is very challenging due to the well-known fact that a skeletal motion is a high-dimensional and non-linear signal. We solve this problem by using regression analysis to estimate the relationship between neutral motions and affective motions, adopting principal component regression (PCR) to deal with the high-dimensional motion signal for the first time. Furthermore, we propose a novel method called piecewise principal component regression (PPCR) to deal with the non-linear problem, in which the motion signal is automatically divided into several segments and PCR is performed on each segment. Our experimental results demonstrate that the proposed PPCR method is successful in generating affective motion within high quality.

In this paper, we propose a fast, accurate content-based video copy detection scheme based on bag-of-global visual features, which is characterized by (1) utilizing an efficient DCT-sign-based feature for fast detection; (2) performing multiple assignment in the temporal domain, in addition to the feature and spatial domain to ensure repeatability in segment-level matching; and (3) adopting an inverse document frequency weighting and temporal burstiness-aware scoring to emphasize distinctive visual words. Despite detection 95 times faster than real-time, the proposed system achieves a false negative rate of 0.2% against queries that are altered by non-geometric transformations without any false positives.

We propose a new voting strategy referred to as ratio voting to improve bag-of-visual words-based image retrieval. It limits the number of votes in proportion to the number of features in visual words, while conventional schemes use (estimated) distances or rank information as a filtering criterion. Ratio voting realizes adaptive thresholding that captures the density of feature vectors. In experiments, we adopt two different distance estimation methods in the post-filtering step and show that ratio voting achieves a considerable improvement in spite of its simplicity in both cases. Furthermore, we perform exhaustive experiments in combining ratio voting with multiple assignment approaches and show that choosing a multiple assignment approach also has a remarkable impact on accuracy.

Music and dance are two major forms of entertainment in our daily life. Moreover, the fact that people dance to music suggests the possibility of synchronizing human motion with music. In this paper, we present a novel system to automatically synthesize human motion that is synchronized with streaming music using both rhythm and intensity features. In our system, a motion capture database is re-organized into a novel graph-based representation with metadata (called metadata motion graphs) beforehand, which is specially designed for the streaming application. When receiving a certain amount of music data as a segment, our system will search a best path for the segment on a metadata motion graph. This approach, whose effectiveness is demonstrated in a user study, can compose motions segment by segment, which (1) are synchronized with the music at a beat level in a short enough period, (2) are connected seamlessly with the previous segment, and (3) have the necessary synchronization capacity for the remaining music no matter how long it is.

We describe an accurate content-based copy detection system that uses both local and global visual features to ensure robustness. Our system advances state-of-the-art techniques in four key directions. (1) Multiple-codebook-based product quantization: conventional product quantization methods encode feature vectors using a single codebook, resulting in large quantization error. We propose a novel codebook generation method for an arbitrary number of codebooks. (2) Handling of temporal burstiness: for a stationary scene, once a query feature matches incorrectly, the match continues in successive frames, resulting in a high false-alarm rate. We present a temporal-burstiness-aware scoring method that reduces the impact from similar features, thereby reducing false alarms. (3) Densely sampled SIFT descriptors: conventional global features suffer from a lack of distinctiveness and invariance to non-photometric transformations. Our densely sampled global SIFT features are more discriminative and robust against logo or pattern insertions. (4) Bigram- and multiple-assignment-based indexing for global features: we extract two SIFT descriptors from each location, which makes them more distinctive. To improve recall, we propose multiple assignments on both the query and reference sides. Performance evaluation on the TRECVID 2009 dataset indicates that both local and global approaches outperform conventional schemes. Furthermore, the integration of these two approaches achieves a three-fold reduction in the error rate when compared with the best performance reported in the TRECVID 2009 workshop.

In this paper, we present a practical and automatic comic-like image layout system for event photos or event movies. We focus on the comic-like layout with cropping. The comic-like layouts allow viewers to track events in time sequential order like comic books, in which event images are cropped and arranged in panels from top left to bottom right on a screen so that the resultant layout has variety of image shapes and gives a pleasant impression. The main drawback of the conventional method is that an important region such as a face or a salient object area is sometimes cropped. In order to avoid excessive cropping of important regions while maintaining the image order, our system achieves adaptive layout to input images by generating all possible templates suitable for comic and deciding appropriate templates based on our proposed layout evaluation. In the demo, we will show that our full automatic system generates comic-like layouts immediately. Furthermore, the system allows a user to select his/her favorite layout from the multiple ones with different looks.

Motion-compensated (MC) prediction is one of the most effective coding tools for high-compression video coding. In former works, adaptive interpolation filter (AIF) technology, which improves the coding performance of the fractional-pel MC, has been proposed. As for the real-time codec system, the implementation of AIF based on the Wiener filter algorithm is difficult because of its high computational complexity. To overcome this problem, in this paper, we propose a low-complexity AIF scheme, which can maintain the coding performance achieved by the conventional AIF. The key technology is the efficient filter design process based on the amplitude characteristic analysis with the precise estimation of MC error signal. The experimental result confirmed that almost the same coding performance was maintained at only 30% of the complexity compared to the conventional AIF.

TV logo recognition is one of the suggested solutions for preventing unauthorized duplication and redistribution. The major problem of the previous logo recognition is that the matching process against the reference logos requires much time. Since millions of minutes of TV programs are produced and broadcasted every day, an efficient algorithm is necessary. In this paper, we propose lightweight logo recognition using an image moment. An image moment represents a geometrical feature by a numerical low dimensional vector and thus matching can be achieved by a simple distance metric. Since an image moment itself is not so robust to noise, we introduce conspicuousness of a transparent logo according to visual properties of its background and surrounding regions. A validation experiment using the actual TV programs achieves much faster processing than the existing method with only a small degradation in accuracy. In the proposed method, computational complexity is independent of the number of the reference logos and the size of the logo region.

We propose a robust background subtraction method for multi-view images, which is essential for realizing free viewpoint video where an accurate 3D model is required. Most of the conventional methods determine background using only visual information from a single camera image, and the precise silhouette cannot be obtained. Our method employs an approach of integrating multi-view images taken by multiple cameras, in which the background region is determined using a 3D model generated by multi-view images. We apply the likelihood of background to each pixel of camera images, and derive an integrated likelihood for each voxel in a 3D model. Then, the background region is determined based on the minimization of energy functions of the voxel likelihood. Furthermore, the proposed method also applies a robust refining process, where a foreground region obtained by a projection of a 3D model is improved according to geometric information as well as visual information. A 3D model is finally reconstructed using the improved foreground silhouettes. Experimental results show the effectiveness of the proposed method compared with conventional works.

This paper presents an efficient video-on-demand (VOD) scheme for free viewpoint television (FTV), and proposes a data format and its data generation method to provide a walkthrough experience. We employ a hybrid rendering approach to describe a 3D scene using 3D model data for objects and textures. However, conventional hybrid rendering methods such as multi-texturing include excessive redundancy in texture data and demand a great deal of bandwidth to transmit. In this paper we propose an efficient texture data format, which removes the redundancy due to occlusion of objects by employing an orthogonal projection image for each object. The additional advantage of the data format is great simplification at the server to choose the transmitted images that correspond to the requested viewpoint. Experiments using multiview real video sequences confirm that the proposed scheme can reduce the transmission of texture data by as much as 42% compared to the conventional scheme.

To improve high resolution video coding efficiency under low bit-rate condition, an appropriate coding mode is required from an R-D optimization (RDO) perspective, although a coding mode defined within the H.264 standard is not always optimal for RDO criteria. With this in mind, we previously proposed extended SKIP modes with close-to-optimal R-D characteristics. However, the additional modes did not always satisfy the optimal R-D characteristics, especially for low bit-rate coding. In this paper, we propose an enhanced coding mode capable of providing a candidate corresponding to the minimum R-D cost by controlling the residual signal associated with the extended SKIP mode. The experimental result showed that the PSNR improvement against H.264 and our previous approach reached 0.42 dB and 0.24 dB in the maximum case, respectively.

In this paper, the authors propose a novel method to measure the perceived picture quality of H.264 coded video based on hybrid no reference framework. The latter term means that the proposed model uses only receiver-side information for objective video quality assessment, but analyzes both the compressed bitstream and baseband signals of the decoded picture to improve the estimation accuracy of the subjective quality. The proposed method extracts quantizer-scale information from the bit-stream along with two spatiotemporal image features from the baseband signal, which are integrated to express the overall quality using the weighted Minkowski metric. A computer simulation shows the proposed method can estimate the subjective quality at a correlation coefficient of 0.909 whereas the PSNR metric, which is referred to as a benchmark, correlates the subjective quality at a correlation coefficient of 0.773.

In this paper, we present a pioneering study on genre classification for home video. Analyzing home video is a challenging problem because it is generally unstructured and of low production quality in audio and video. Our approach is to define a set of genres referring to those in the actual video sharing site, and to extract salient low level features from MPEG compressed data which are robust to low production quality content. Experimental results based on ensemble learning show that our proposed method achieves around 0.7 to 0.8 F-measure values with 37.5 times faster processing than real-time playback for QVGA resolution home video.

This paper presents a novel method to represent a real 3D world using IBR (Image-based Rendering) technology. The major achievement is realization of "walk-through" experience, in which audiences can see the scene of sport games as if they were the player, while the conventional IBR method cannot provide such an experience. The key idea is a newly introduced framework "locally divided ray space" to capture the ray inside the scene. Consequently, the proposed algorithm can generate a virtual image inside the scene using multiple real images captured from outside. In this paper, the outline of the proposed method as well as the possible demonstration at the conference venue is described.

In this paper, we propose two novel methods for arbitrary listening-point generation for 3D audio-video (3DAV) integration in a large-scale multipoint cameras and microphones system with abilities to process, and display information of any recorded 3D scene in realtime. With this system, users are able to control their own viewpoint/listening-point position, freely. Arbitrary listening-point can be generated by either (i) ray-space representation of sound wave field (i.e. source sound independent) for multi frequency layers, or (ii) acoustic transfer function estimation (i.e. source sound dependent) and blind separation of sources of sounds. Arbitrary viewpoint generation is based on ray-space method, which is enhanced by using multipass dynamic programming for geometry compensation. Integration is done by either (i) ray-space representation of sound wave and image together, or (ii) integrating each camera video signal and acoustic transfer function of the same location as integrated 3DAV data. The prototype system of integrated audio-visual viewer achieves both good image and sound qualities with 15 frames/second.

In this paper, visual cryptography for halftone images is proposed, which uses threshold arrays generated by a void-and-cluster method. The proposed halftone visual cryptography (HFC) using threshold arrays shows great flexibility for a combination of share images in order to decode a secret image. Moreover, it also achieves high quality share images, high speed processing and extensibility to (k,n) the visual cryptography of three share image categories, such as (2,3) or (3,3). Simulation results show that all the above features are achieved, while a secret image is clearly decoded, and the secret image cannot be seen on each share image.

A novel video watermarking method is proposed, which can directly embed a watermark into a bit stream generated by an H.264 video encoder. Watermarking to encoded data provides a content provider of flexible capability to attach secret information; however it brings a drift problem. A drift compensation method has been proposed, however it requires many computations such as DCT and motion compensation. Therefore a new drift compensation method with spatiotemporal watermarking is proposed that only cancels the effect of the embedding on blocks with little motion. It achieves great reduction of computation and realizes fast and high picture quality watermarking. The computer simulation shows that degradation of picture quality is negligible compared to noises due to H.264 compression, and that the proposed method is highly resilient to frequency domain attacks

In this paper, we propose an efficient algorithm for conversion of an MP3 stream into AAC. Generally, this kind of conversion, transcoding, requires full decoding and re-encoding. However, the re-encoding based transcoding process may result in degradation of quality and take longer than encoding from a PCM signal. This paper proposes a method to inherit the frame structure and quantization scale from MP3 to AAC. This enables a reduction in the iteration process, which requires the most time in the AAC encoding process, without incurring degradation in quality. Experimental results show that the proposed method can perform bitstream domain transcoding at high speed while maintaining a high level of audio quality

This paper proposes an MP3 watermarking method that is applicable to a mobile terminal with limited computational resources. Considering that the embedded information is copyright information and metadata, which should be extracted before playing back, the watermark detection process should be executed quickly. However, conventional methods cannot detect a digital watermark at high speed. Thus, this paper proposes that scalefactor values in MP3 data be altered so as not to spoil audio quality. Evaluation tests show that the proposed method is capable of embedding 3bits/frame information and detecting at very high speed without degrading audio quality. Finally, this paper describes an application example of our proposal for authentication with a digital signature.

Multimedia contents transmission and management system "VAST-web" is proposed. It employs store-and-forward transmission technique to maintain the contents quality. It also has a central management function of contents stored at remote locations so that contents query and contents pull model as well as contents push model are supported. A novel transmission protocol "SVFTP" is also introduced, which is a core technique in VAST-web. Its throughput performance is evaluated theoretically and experimentally to show it is well-suited to large volume multimedia contents transmission. Besides, its pipeline transmission capability provides VAST-web system with great flexibility for media playback.