Clustering is essential in data analysis since many real-world datasets are unlabeled and are expensive to label. Density-based clustering algorithms are known for their capability of identifying clusters of non-spherical shapes and have been widely studied over recent decades. Among these algorithms, Density Peaks (DP) clustering is an outstanding one that is particularly robust to changes in the distance metric. However, the performance of DP clustering highly depends on the choice of its hyperparameters: the cutoff distance and the number of clusters. As a result, we developed a learning-based approach for selecting appropriate values of the hyperparameters for DP clustering in one-shot. We address the challenging issue of effective one-shot hyperparameter selection by proposing two novel embeddings: the HINT (Histogram of Neighborhood Transform) embedding and the Gamma embedding. The HINT embedding calculates the histogram of the$m$-th neighborhood distances for each node. The neighborhood distances histogram captures practical characteristics of the density property for a given dataset. Meanwhile, the Gamma embedding condenses the information of a decision graph while still providing crucial clues for determining the number of clusters. Therefore, we achieved effective one-shot hyperparameter selection by the proposed novel embeddings. As compared with an exhaustive grid search method, our method is 169 times faster, while its relative performance ratio is up to 89.6%, which demonstrates its effectiveness. As there has been a shortage of research devoted to hyperparameter selection for DP clustering, we expected our promising result to inspire more studies toward the important research topic.

The success of modern deep learning algorithms requires large amounts of training data, which leads to high computational and storage costs. Dataset Distillation (DD) is a rising research field that resolves this issue by synthesizing a compact training dataset from a large one. Recent gradient matching DD methods have achieved remarkable results. However, these methods typically utilize weak models for DD performance improvement, while well-trained models are often considered inferior choices due to their lower performance. Conversely, our study provides new insights into the role of well-trained models in DD, particularly under high-storage budget scenarios. We identify a previously overlooked design principle—a positive correlation between model capability and storage budget. Based on this principle, we propose Drop2Sparse, an approach that randomly sparsifies well-trained models to create efficient models for various storage budget scenarios. Drop2Sparse concurrently infuses significant model diversity and regularization effects into DD, outperforming previous state-of-the-art methods by up to 3.8% on CIFAR and 3.6% on ImageNet-subset. Moreover, our method exhibits remarkable cross-architecture generalization and achieves promising results even under challenging scenarios, such as using an extremely reduced model pool or highly accelerated training.

Deepfake is becoming a major security threat nowadays. The development of robust deepfake detection algorithm is booming in recent years. However, the existing works require many face images for model training and the frontal face images collection would have privacy concerns. Therefore, this privacy issue inspires our research work that a deepfake detection model is trained from photos with face mask. We found that training a deepfake detection model from photos with face mask is rarely discussed in the literature. In order to resolve the challenges from face mask, we incorporate sub-regions (e.g., eye, nose, jaw) in a face during the training process. In addition to the extension by including facial sub-regions, we found that the associate training strategy is another important design factor for performance improvement. Our method improved the result significantly and the AUC of FaceForensics++ (FF++) test dataset evaluation is increased from 87.67% to 98.93%. As a result, we can develop a promising deepfake detection model even from the photos with face mask. We expected this work can be a stepping stone to inspire more research works with privacy considerations.

Online users’ feedback has numerous text comments to enrich the review quality on mainstream platforms, such as Yelp and Google Maps. Reading through numerous review comments to speculate the important aspects is tedious and time-consuming. Apparently, there is a huge gap between the numerous commentary text and the crucial aspects for users’ preferences. In this study, we proposed a weakly supervised framework called Sequence to Commentary Aspect Score Estimation (Seq2CASE) to estimate the vital aspect scores from the review comments, since the ground truth of the aspect score is seldom available. The aspect score estimation from Seq2CASE is close to the actual aspect scoring; precisely, the average Mean Absolute Error (MAE) is less than 0.4 for a 5-point grading scale. The performance of Seq2CASE is comparable to or even better than the state-of-the-art supervised approaches in recommendation tasks. We expect this work to be a stepping stone that can inspire more unsupervised studies working on this important but relatively underexploited research.

In this paper we propose the macroblock scaling (MBS) algorithm, which can be applied to various CNN architectures to reduce their model size. MBS adaptively reduces each CNN macroblock depending on its information redundancy measured by our proposed effective flops. Empirical studies conducted with ImageNet and CIFAR-10 attest that MBS can reduce the model size of some already compact CNN models, e.g., MobileNetV2 (25.03% further reduction) and ShuffleNet (20.74%), and even ultra-deep ones such as ResNet-101 (51.67%) and ResNet-1202 (72.71%) with negligible accuracy degradation. MBS also performs better reduction at a much lower cost than the state-of-the-art optimization-based methods do. MBS's simplicity and efficiency, its flexibility to work with any CNN model, and its scalability to work with models of any depth make it an attractive choice for CNN model size reduction.

This paper introduces a scalable algorithm for rendering translucent materials with complex lighting. We represent the light transport with a diffusion approximation by a dual-matrix representation with the Light-to-Surface and Surface-to-Camera matrices. By exploiting the structures within the matrices, the proposed method can locate surface samples with little contribution by using only subsampled matrices and avoid wasting computation on these samples. The decoupled estimation of irradiance and diffuse BSSRDFs also allows us to have a tight error bound, making the adaptive diffusion approximation more efficient and accurate. Experiments show that our method outperforms previous methods for translucent material rendering, especially in large scenes with massive translucent surfaces shaded by complex illumination.

3D HEVC is one of the extensions of HEVC (High Efficiency Video Coding), which is the latest video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). The inherent high computing complexity of 3D HEVC handicaps its usage in practical applications. HEVC replaces the macroblock (MB) of H.264 with the largest coding unit (LCU), in which coding units (CUs) of sizes 64 × 64, 32 × 32, 16 × 16, or 8 × 8 pixels are used to build a coding tree unit (CTU). HEVC encoder will traverse the coding modes from the root (the coding level is 0) to the leaves (the coding level is 3) of the CTU. As a result, how to accelerate the encoding process of 3D HEVC with negligible loss of coding efficiency is a hot research topic in the field of video coding. Figure 1 shows the coding level distributions of I04 and I14, where Ikj represents the j-th frame in the k-th GOP. The POC's (picture order counts) of the two frame are P(I04) = 4 and P(I14) = 12 when GOP size is 8. From Figure 1 we observed that, in B-slices, the distributions of "optimal coding level = zero" for adjacent GOP's are almost the same. We found that the required coding level will be similar when the two CUs have similar depth values. We utilize this correlation to limit the required coding level visiting of a given CU which accelerates the encoding process. In this paper, a fast mode decision algorithm for 3D HEVC, based on the depth information related coding mode similarity, is proposed. According to the experimental results, the proposed algorithm can reduce up to 72% executing time of the overall encoding process, while the loss in coding efficiency is negligible.

The objective approaches of 3D image quality assessment play a key role in the development of compression standards and various 3D multimedia applications. The quality assessment of 3D images faces many new challenges, e.g. asymmetric stereo compression, depth perception, and virtual view synthesis, as compared with its 2D counterparts. Moreover, the widely used 2D image quality metric (e.g. PSNR) cannot be directly applied to deal with these newly introduced challenges. This statement can be verified by the low correlation between the computed objective measures and the subjectively measured mean opinion scores (MOS), when 3D images are the tested targets. In order to meet these challenges, in this work, besides traditional 2D image metrics, two binocular behaviors - the binocular combination and the Binocular Frequency Integration (BFI), are utilized as the bases for measuring the quality of stereoscopic 3D images. The effectiveness of BFI-based metrics is verified by conducting subjective evaluations on a publicly available stereo image dataset. Experimental results show that significant consistency could be reached between the measured MOS and the BFI-based metrics, in which the correlation coefficient between them can go up to 0.89 even if the stereo images have been asymmetrically HEVC compressed. Based on the proposed quality metric, we find that asymmetric-stereo compression schemes outperform the corresponding symmetric ones in high bit rate scenarios, which opens up a new research direction for 3D stereo image/video compression studies.

Color-plus-Depth 3D images are booming up with the advance of depth-sensing camera (e.g., Kinect). This new 3D visual content imposes new challenges on image resizing since we have to resize both the color and the depth images simultaneously. In order to resolve these newly introduced challenges of color-plus-depth 3D images, we propose a new energy function with salient depth cues consideration for seam carving operation. We further incorporate super-pixel over-segmentation and depth remapping for achieving an object-based and 3D viewing comfort zone aware resizing framework. Experimental results show the proposed framework can effectively generate the resized images by maintaining the salient regions and providing a comfortable 3D visual perception, at the same time.

3D scanning is an important technology since one can apply the scanning results of objects to numerous 3D applications (e.g. Artwork sculpture preserving, 3D animation and 3D printing). However, 3D laser scanners are too expensive to be adopted in daily usage. Therefore, building a 3D scanner by a low cost RGB-D camera (e.g. Kinect) is an emerging trend. For a 3D scanner, a specular object is one of the challenging 3D scanning targets where the specular surface will compromise the scanning (laser) lights and lead to scanning failure. In order to acquire the ground truth for 3D scanning of specular objects, we have to perform a non-specular painting process even a 3D laser scanner is used. In order to meet this challenge for an RGB-D camera based 3D scanner, we integrate the response of visual cues reflection and the depth scattering characteristics of specular surfaces to resolve the artifacts of 3D scanning results. Experimental results show our proposed system outperforms the traditional RGB-D 3D scanners in reconstruction quality while keeping the specular object intact (i.e., We need not to perform the pre-described non-specular painting process on the object before scanning).

The objective approaches of 3D image quality assessment play a key role for the development of compression standards and various 3D multimedia applications. The quality assessment of 3D images faces more new challenges, such as asymmetric stereo compression, depth perception, and virtual view synthesis, than its 2D counterparts. In addition, the widely used 2D image quality metrics (e.g., PSNR and SSIM) cannot be directly applied to deal with these newly introduced challenges. This statement can be verified by the low correlation between the computed objective measures and the subjectively measured mean opinion scores (MOSs), when 3D images are the tested targets. In order to meet these newly introduced challenges, in this paper, besides traditional 2D image metrics, the binocular integration behaviors-the binocular combination and the binocular frequency integration, are utilized as the bases for measuring the quality of stereoscopic 3D images. The effectiveness of the proposed metrics is verified by conducting subjective evaluations on publicly available stereoscopic image databases. Experimental results show that significant consistency could be reached between the measured MOS and the proposed metrics, in which the correlation coefficient between them can go up to 0.88. Furthermore, we found that the proposed metrics can also address the quality assessment of the synthesized color-plus-depth 3D images well. Therefore, it is our belief that the binocular integration behaviors are important factors in the development of objective quality assessment for 3D images.

This work addresses the data representation and the compression issues of angular disparity map following the way of HVS to perceive depth information. The continued fraction is utilized to represent the angular disparity map which enables the use of the state-of-the-art video codec (e.g. HEVC) to compress the data directly and maintains quality scalability properties. We observe that there is a non-monotonic phenomenon of the RD curves by applying HEVC compression to angular disparity map directly. This implies that the correlations among inter-layer (i.e., the neighboring integers in (2)) do not follow the traditional models of normal 2D video codecs. Of course, the detailed relationship between the sensitivities and the quantization errors of the newly proposed representation needs in depth further derivations. There are many interesting research issues may be introduced by the proposed data format (e.g., the sensitivities to quantization errors of θ and the rate-distortion optimization scheme for θ) which will, of course, be the research topics of our future work. We expect this work can be a bridge to connect the 3D perception and the 3D compression research fields.

Location-based service (LBS) is booming up in recent years with the rapid growth of mobile devices and the emerging of cloud computing paradigm. Among the challenges to establish LBS, the user privacy issue becomes the most important concern. A successful privacy-preserving LBS must be secure and provide accurate query [e.g., -nearest neighbor (NN)] results. In this work, we propose a private circular query protocol (PCQP) to deal with the privacy and the accuracy issues of privacy-preserving LBS. The protocol consists of a space filling curve and a public-key homomorphic cryptosystem. First, we connect the points of interest (POIs) on a map to form a circular structure with the aid of a Moore curve. And then the homomorphism of Paillier cryptosystem is used to perform secret circular shifts of POI-related information (POI-info), stored on the server side. Since the POI-info after shifting and the amount of shifts are encrypted, LBS providers (e.g., servers) have no knowledge about the user's location during the query process. The protocol can resist correlation attack and support a multiuser scenario as long as the predescribed secret circular shift is performed before each query; in other words, the robustness of the proposed protocol is the same as that of a one-time pad encryption scheme. As a result, the security level of the proposed protocol is close to perfect secrecy without the aid of a trusted third party and simulation results show that the k-NN query accuracy rate of the proposed protocol is higher than 90% even when is large.

Quick Response (QR) code is a widely used matrix bar code with the increasing population of smart phones. However, QR code usually consists of random textures which are not suitable for incorporating with other visual designs (e.g. name card and business advertisement poster). In order to overcome the shortcomings of noise-like looks of QR codes, we propose a systematic QR code beautification framework where the visual appearance of QR code is composed of visually meaningful patterns selected by users, and more importantly, the correctness of message decoding is kept intact. Our work makes QR code from machine decodable only (i.e. standardized random texture) to a personalized form with human visual pleasing appearance. We expect the proposed QR code beautifier can inspire more visual-pleasant mobile multimedia applications.

With the rapid emergence of 3D displays, we can enrich the user's viewing experiences by adding depth information to the widely existing 2D contents. However, effectively inferring the associated depth from a single 2D image is still a challenging problem. By taking benefits from the recently appeared image descriptors, we proposed the use of an SVM based framework for addressing the single image depth estimation. One advantage is its direct extension to incorporate the recent researches of large scale classification via SVM to meet the upcoming cloud computing paradigm. Our experimental results showed that the proposed framework outperforms the state-of-the-art approaches in performance, even the ones using more complex graphical models like MRF. Also, we made a brief investigation on the individual effectiveness of a set of commonly used image descriptors and found that spatial descriptors (e.g. texture) would be more effective than frequency ones (e.g. DCT coefficients).

Unseen visible watermarking (UVW) is a novel data hiding scheme that imitates real-world watermarks and maintains advantages of both visible and invisible watermarking. One important feature of UVW is that specific extraction module is not required during watermarking decoding. The UVW for 2D image is studied based on imaging functions, e.g. gamma-correction in LCD monitors. On the other hand, due to the great success of 3D movies and low-priced 3D display devices, 3D contents are booming up in recent years. In this work, we propose a UVW for color plus depth map 3D images in which the watermark extraction is realized by changing of the rendering conditions. Under normal rendering conditions, the watermarked cover can be perceived exactly the same as the original one. Limitations and future extensions of the proposed 3D UVW are also addressed in this work.

3D multimedia attracts people by the vivid and the fascinating depth perception. The movie industry is in the Renaissance phase with the aid of 3D multimedia since the 3D experience will be amplified in the movie theater. Although we are living in a 3D world, the development of 3D multimedia just started in recent years. Therefore, we want to propose a 3D multimedia signal processing framework to deal with the various challenges coming from this newly appeared media. We have addressed the research issues regarding content creation, data compression and content protection with which the traditional 2D multimedia based approaches will not performed efficiently and properly. Furthermore, we are studying the core issue of 3D multimedia which is the quality assessment of stereopsis. We expect the research of 3D multimedia signal processing can further popularize the 3D contents and brings new thoughts of multimedia research to enable new paradigms of 3D multimedia applications.

Summary form only given. In this work, we experimented on the compression efficiency of rendering lossless compression of depth images and found that the compression ratios can go up to 20.51 and 40 for Interview and Breakdancer test images, respectively, even if the parameter setting is in the worst case (i.e., set fdensity(Znear, Zfar) to its maximum value). This work is our first step toward exploring the performance of rendering lossless compression. It is our belief that, besides the rendering lossless quantization, there are a lot of different issues of depth image compression which are worthy of further exploitation.