Diffusion Transformers (DiTs) have exhibited robust capabilities in image generation tasks. However, accurate text-guided image editing for multimodal DiTs (MM-DiTs) still poses a significant challenge. Unlike UNet-based structures that could utilize self/cross-attention maps for semantic editing, MM-DiTs inherently lack support for explicit and consistent incorporated text guidance, resulting in semantic misalignment between the edited results and texts. In this study, we disclose the sensitivity of different attention heads to different image semantics within MM-DiTs and introduce HeadRouter , a training-free image editing framework that edits the source image by adaptively routing the text guidance to different attention heads in MM-DiTs. Furthermore, we propose a dual-token refinement module to refine text/image token representations for precise semantic guidance and accurate region expression. Experiments on multiple benchmarks demonstrate HeadRouter’s performance in terms of editing fidelity and image quality. The code is available at https://github.com/ICTMCG/HeadRouter .

Despite the remarkable process of talking-head-based avatar-creating solutions, directly generating anchor-style videos with full-body motions remains challenging. In this study, we propose Make-Your-Anchor+, a novel system necessitating only a one-minute video clip of an individual for training, subsequently enabling the automatic generation of anchor-style videos with precise torso and hand movements. Specifically, we finetune a proposed structure-guided diffusion model on input video to render 3D mesh conditions into human appearances. We adopt a two-stage training strategy for the diffusion model, effectively mapping movements with specific appearances to create digital avatars for online streamers, live shopping hosts, and other applications. To produce arbitrary long temporal video, we extract human motion information from video diffusion prior by adapting the frame-wise diffusion model to pretrained video diffusion weights with lower cost, and a simple yet effective batch-overlapped temporal denoising module is proposed to bypass the constraints on video length during inference. Finally, a novel identity-specific face enhancement module is introduced to improve the visual quality of facial regions in the output videos. Comparative experiments demonstrate the system's effectiveness and superiority in visual quality, temporal coherence, and identity preservation, outperforming SOTA diffusion/non-diffusion methods.

Recent advances in diffusion models have enhanced multimodal-guided visual generation, enabling customized subject insertion that seamlessly “brushes” user-specified objects into a given image guided by textual prompts. However, existing methods often struggle to insert customized subjects with high fidelity and align results with the user’s intent through textual prompts. In this work, we propose In-Context Brush, a zero-shot framework for customized subject insertion by reformulating the task within the paradigm of in-context learning. Without loss of generality, we formulate the object image and the textual prompts as cross-modal demonstrations, and the target image with the masked region as the query. The goal is to inpaint the target image with the subject aligning textual prompts without model tuning. Building upon a pretrained MMDiT-based inpainting network, we perform test-time enhancement via dual-level latent space manipulation: intra-head latent feature shifting within each attention head that dynamically shifts attention outputs to reflect the desired subject semantics and inter-head attention reweighting across different heads that amplifies prompt controllability through differential attention prioritization. Extensive experiments and applications demonstrate that our approach achieves superior identity preservation, text alignment, and image quality compared to existing state-of-the-art methods, without requiring dedicated training or additional data collection. Project page: https://yuci-gpt.github.io/In-Context-Brush/.

Personalized generation paradigms empower designers to customize visual intellectual property with the help of textual descriptions by adapting pre-trained text-to-image models on a few images. Recent studies focus on simultaneously customizing content and detailed visual style in images but often struggle with entangling the two. In this study, we reconsider the customization of content and style concepts from the perspective of parameter space construction. Unlike existing methods that utilize a shared parameter space for content and style learning, we propose a novel framework that separates the parameter space to facilitate individual learning of content and style by introducing “partly learnable projection” (PLP) matrices to separate the original adapters into divided sub-parameter spaces. A “ break-for-make ” customization learning pipeline based on PLP is proposed: we first break the original adapters into “up projection” and “down projection” for content and style concept under orthogonal prior and then make the entity parameter space by reconstructing the content and style PLP matrices by using Riemannian preconditioning to adaptively balance content and style learning. Experiments on various styles, including textures, materials, and artistic style, show that our method outperforms state-of-the-art single/multiple concept learning pipelines regarding content-style-prompt alignment. Code is available at https://github.com/ICTMCG/Break-for-make .

Neural implicit function based on signed distance field (SDF) has achieved impressive progress in reconstructing 3D models with high fidelity. However, such approaches can only represent closed surfaces. Recent works based on unsigned distance function (UDF) are proposed to handle both watertight and single-layered open surfaces. Nonetheless, as UDF is signless, its direct output is limited to the point cloud, which imposes an additional challenge on extracting high-quality meshes from discrete points. To address this challenge, we present a novel neural implicit representation coded HSDF, which is a hybrid of signed and unsigned distance fields. In particular, HSDF is able to represent arbitrary topologies containing both closed and open surfaces while being compatible with existing iso-surface extraction techniques for easy field-to-mesh conversion. In addition to predicting a UDF, we propose to learn an additional sign field. Unlike traditional SDF, HSDF is able to locate the surface of interest before level surface extraction by generating surface points following NDF (Chibane et al. 2020). We are then able to obtain open surfaces via an adaptive meshing approach that only instantiates regions containing surfaces into a polygon mesh. HSDF benefits downstream tasks like neural rendering, as it enables the rendering of back-faces of open surfaces. We also propose HSDF-Net, a dedicated learning framework that factorizes the learning of HSDF into two easier sub-problems. Experiments and evaluations show that HSDF outperforms the state-of-the-art techniques both qualitatively and quantitatively on some of the used datasets.

Nowadays, the proliferation of portraits or photographs containing human faces on the internet has created significant risks of illegal privacy collection and analysis by intelligent systems. Previous attempts to protect against unauthorized identification by face recognition models have primarily involved manipulating or adding adversarial perturbations to photos. However, it remains a challenge to balance privacy protection effectiveness and maintaining image visual quality. That is, to successfully attack real-world black-box face recognition models, significant manipulation is required for the source image, which will obviously damage the image visual quality. To address these issues, we propose an attribute-guided face identity protection (AG-FIP) approach that can protect facial privacy effectively without introducing meaningless or conspicuous artifacts into the source image. The proposed method involves mapping the images to latent space and subsequently implementing an adversarial attack through attribute editing. An attribute selection module followed by an attribute adversarially editing module is proposed to enhance the efficiency and effectiveness of adversarial attacks. Experimental results demonstrate that our approach outperforms SOTAs in terms of confusing black-box face recognition models, commercial face recognition APIs, and image visual quality.

Fake news detection plays a crucial role in protecting social media users and maintaining a healthy news ecosystem. Among existing works, comment-based fake news detection methods are empirically shown as promising because comments could reflect users' opinions, stances, and emotions and deepen models' understanding of fake news. Unfortunately, due to exposure bias and users' different willingness to comment, it is not easy to obtain diverse comments in reality, especially for early detection scenarios. Without obtaining the comments from the "silent'' users, the perceived opinions may be incomplete, subsequently affecting news veracity judgment. In this paper, we explore the possibility of finding an alternative source of comments to guarantee the availability of diverse comments, especially those from silent users. Specifically, we propose to adopt large language models (LLMs) as a user simulator and comment generator, and design GenFEND, a generated feedback-enhanced detection framework, which generates comments by prompting LLMs with diverse user profiles and aggregating generated comments from multiple subpopulation groups. Experiments demonstrate the effectiveness of GenFEND and further analysis shows that the generated comments cover more diverse users and could even be more effective than actual comments.

Deep neural networks can be vulnerable to adversarial attacks, even for the mainstream Transformer-based models. Although several robustness enhancement approaches have been proposed, they usually focus on some certain type of perturbation. As the types of attack can be various and unpredictable in practical scenarios, a general and strong defense method is urgently in require. We notice that most well-trained models can be weakly robust in the perturbation space, i.e., only a small ratio of adversarial examples exist. Inspired by the weak robust property, this paper presents a novel ensemble method for enhancing robustness. We propose a lightweight framework PAD to save computational resources in realizing an ensemble. Instead of training multiple models, a plugin module is designed to perturb the parameters of a base model which can achieve the effect of multiple models. Then, to diversify adversarial example distributions among different models, we promote each model to have different attention patterns via optimizing a diversity measure we defined. Experiments on various widely-used datasets and target models show that PAD can consistently improve the defense ability against many types of adversarial attacks while maintaining accuracy on clean data. Besides, PAD also presents good interpretability via visualizing diverse attention patterns.

Despite the remarkable process of talking-head-based avatar-creating solutions, directly generating anchor-style videos with full-body motions remains challenging. In this study, we propose Make-Your-Anchor, a novel system necessitating only a one-minute video clip of an individual for training, subsequently enabling the automatic generation of anchor-style videos with precise torso and hand movements. Specifically, we finetune a proposed structure-guided diffusion model on input video to render 3D mesh conditions into human appearances. We adopt a two-stage training strategy for the diffusion model, effectively binding movements with specific appearances. To produce arbitrary long temporal video, we extend the 2D U-Net in the frame-wise diffusion model to a 3D style without additional training cost, and a simple yet effective batch-overlapped temporal denoising module is proposed to bypass the constraints on video length during inference. Finally, a novel identity-specific face enhancement module is introduced to improve the visual quality of facial regions in the output videos. Comparative experiments demonstrate the effectiveness and superiority of the system in terms of visual quality, temporal coherence, and identity preservation, outperforming SOTA diffusion/non-diffusion methods. Project page: https://github.com/ICTMCG/Make-Your-Anchor.

Concept personalization methods enable large text-to-image models to learn specific subjects (e.g., ob-jects/poses/3D models) and synthesize renditions in new contexts. Given that the image references are highly biased towards visual attributes, state-of-the-art personalization models tend to overfit the whole subject and cannot disentangle visual characteristics in pixel space. In this study, we proposed a more challenging setting, namely fine-grained visual appearance personalization. Different from existing methods, we allow users to provide a sentence describing the desired attributes. A novel decoupled self-augmentation strategy is proposed to generate target-related and non-target samples to learn user-specified visual attributes. These augmented data allow for refining the model's understanding of the target attribute while mitigating the impact of unrelated attributes. At the inference stage, adjustments are conducted on semantic space through the learned tar-get and non-target embeddings to further enhance the dis-entanglement of target attributes. Extensive experiments on various kinds of visual attributes with SOTA personalization methods show the ability of the proposed method to mimic target visual appearance in novel contexts, thus improving the controllability and flexibility of personalization. Project page: https://github.com/ICTMCG/U-VAP.

As short-form video-sharing platforms become a significant channel for news consumption, fake news in short videos has emerged as a serious threat in the online information ecosystem, making developing detection methods for this new scenario an urgent need. Compared with that in text and image formats, fake news on short video platforms contains rich but heterogeneous information in various modalities, posing a challenge to effective feature utilization. Unlike existing works mostly focusing on analyzing what is presented, we introduce a novel perspective that considers how it might be created. Through the lens of the creative process behind news video production, our empirical analysis uncovers the unique characteristics of fake news videos in material selection and editing. Based on the obtained insights, we design FakingRecipe, a creative process-aware model for detecting fake news short videos. It captures the fake news preferences in material selection from sentimental and semantic aspects and considers the traits of material editing from spatial and temporal aspects. To improve evaluation comprehensiveness, we first construct FakeTT, an English dataset for this task, and conduct experiments on both FakeTT and the existing Chinese FakeSV dataset. The results show FakingRecipe's superiority in detecting fake news on short video platforms.

Due to the high mobility of vehicles, service migration is inevitable in vehicular edge computing (VEC) networks. Frequent service migrations incur prohibitive migration cost including the computing cost (e.g., increased computing delay) and communication cost (e.g., occupied backhaul bandwidth). Yet existing service migration schemes are usually designed without considering the impact of the computing cost. This paper considers the impact of computing and communication cost jointly, and proposes a computing and communication cost-aware service migration scheme for VEC networks (i.e., CA-migration). Taking the service delay as a QoS metric for VEC networks, this paper formulates a migration optimization problem aiming to maximize the services' satisfaction degree of delay (i.e., the probability that the service delay is smaller than the service delay requirement), where both the communication cost and computing cost affect the services' satisfaction degree. Since the optimization problem is a constrained non-linear integer programming problem, it is difficult to solve. Moreover, the VEC networks are highly dynamic. Thus, a fast transfer reinforcement learning (fast-TRL) method combining transfer learning and reinforcement learning is proposed to provide an adaptive service migration scheme in dynamic VEC networks. Simulation results show that compared with existing schemes, the proposed CA-migration scheme can increase the satisfaction degree by up to 30%, and needs 25% less training time to obtain the optimal service migration policy.

In this study, we revisit the fundamental setting of face-swapping models and reveal that only using implicit supervision for training leads to the difficulty of advanced methods to preserve the source identity. We propose a novel reverse pseudo-input generation approach to offer supplemental data for training face-swapping models, which addresses the aforementioned issue. Unlike the traditional pseudo-label-based training strategy, we assume that arbitrary real facial images could serve as the ground-truth outputs for the face-swapping network and try to generate corresponding input pair data. Specifically, we involve a source-creating surrogate that alters the attributes of the real image while keeping the identity, and a target-creating surrogate intends to synthesize attribute-preserved target images with different identities. Our framework, which utilizes proxy-paired data as explicit supervision to direct the face-swapping training process, partially fulfills a credible and effective optimization direction to boost the identity-preserving capability. We design explicit and implicit adaption strategies to better approximate the explicit supervision for face swapping. Quantitative and qualitative experiments on FF++, FFHQ, and wild images show that our framework could improve the performance of various face-swapping pipelines in terms of visual fidelity and ID preserving. Furthermore, we display applications with our method on re-aging, swappable attribute customization, cross-domain, and video face swapping. Code is available under https://github.com/ ICTMCG/CSCS.

The daily practice of online image sharing enriches our lives, but also raises a severe issue of privacy leakage. To mitigate the privacy risks during image sharing, some researchers modify the sensitive elements in images with visual obfuscation methods including traditional ones like blurring and pixelating, as well as generative ones based on deep learning. However, images processed by such methods may be recovered or recognized by models, which cannot guarantee privacy. Further, traditional methods make the images very unnatural with low image quality. Although generative methods produce better images, most of them suffer from insufficiency in the frequency domain, which influences image quality. Therefore, we propose the AdvERsArial Sensitive Element Remover (ERASER) to guarantee both image privacy and image quality. 1) To preserve image privacy, for the regions containing sensitive elements, ERASER guarantees enough difference after being modified in an adversarial way. Specifically, we take both the region and global content into consideration with a Prior Transformer and obtain the corresponding region prior and global prior. Based on the priors, ERASER is trained with an adversarial Difference Loss to make the content in the regions different. As a result, ERASER can reserve the main structure and change the texture of the target regions for image privacy preservation. 2) To guarantee the image quality, ERASER improves the frequency insufficiency of current generative methods. Specifically, the region prior and global prior are processed with Fast Fourier Convolution to capture characteristics and achieve consistency in both pixel and frequency domains. Quantitative analyses demonstrate that the proposed ERASER achieves a balance between image quality and image privacy preservation, while qualitative analyses demonstrate that ERASER indeed reduces the privacy risk from the visual perception aspect.

With information consumption via online video streaming becoming increasingly popular, misinformation video poses a new threat to the health of the online information ecosystem. Though previous studies have made much progress in detecting misinformation in text and image formats, video-based misinformation brings new and unique challenges to automatic detection systems: 1) high information heterogeneity brought by various modalities, 2) blurred distinction between misleading video manipulation and nonmalicious artistic video editing, and 3) new patterns of misinformation propagation due to the dominant role of recommendation systems on online video platforms. To facilitate research on this challenging task, we conduct this survey to present advances in misinformation video detection. We first analyze and characterize the misinformation video from three levels including signal, semantic, and intent. Based on the characterization, we systematically review existing works for detection from features of various modalities to techniques for clue integration. We also introduce existing resources including representative datasets and useful tools. Besides summarizing existing studies, we discuss related areas and outline open issues and future directions to encourage and guide more research on misinformation video detection. The corresponding repository is at https://github.com/ICTMCG/Awesome-Misinfo-Video-Detection.

Agricultural multi-robot task allocation is an important direction of intelligent agricultural robots. In this paper, to ensure the shortest total movement distance of robots and balance the workload, we transform the task allocation of agricultural multi-robot into a multi objective multiple traveling salesman problem (MO-MTSP). A processing framework is also designed to adjust the allocation plan in real-time by monitoring the status of robots and tasks. In this processing framework, an improved Non-dominated genetic algorithm (INSGA-II) is utilized to achieve better performance of task allocation. The improvement includes: 1) An objective function is designed to ensure a balanced task workload distribution for each robot considering the actual conditions of the farmland; 2) A neighborhood search algorithm is used to accelerate the convergence of the algorithm by adjusting the initial population plan; 3) An adaptive adjustment mechanism for crossover and mutation probabilities is proposed to enhance the flexibility of the task allocation plan searching. Finally, the proposed algorithm is evaluated on both public datasets and actual agricultural plot datasets. Experimental results show that the improved algorithm can achieve better allocation results, demonstrating the practical applicability of the algorithm in task allocation of multiple robots employed in agriculture scenarios.

Dialogue State Tracking (DST) module is an essential component of task-oriented dialog systems to understand users’ goals and needs. Collecting dialogue state labels including slots and values can be costly, requiring experts to annotate all (slot, value) information for each turn in dialogues. It is also difficult to define all possible slots and values in advance, especially with the wide application of dialogue systems in more and more new-rising applications. In this paper, we focus on improving DST module to generate dialogue states in circumstances with limited annotations and knowledge about slot ontology. To this end, we design a dual prompt learning framework for few-shot DST. The dual framework aims to explore how to utilize the language understanding and generation capabilities of pre-trained language models for DST efficiently. Specifically, we consider the learning of slot generation and value generation as dual tasks, and two kinds of prompts are designed based on this dual structure to incorporate task-related knowledge of these two tasks respectively. In this way, the DST task can be formulated as a language modeling task efficiently under few-shot settings. To evaluate the proposed framework, we conduct experiments on two task-oriented dialogue datasets. The results demonstrate that the proposed method not only outperforms existing state-of-the-art few-shot methods, but also can generate unseen slots. It indicates that DST-related knowledge can be probed from pre-trained language models and utilized to address low-resource DST efficiently with the help of prompt learning.

Federated learning (FL) over mobile networks has attracted intensive attention recently. User mobility is a fundamental feature of mobile networks, which leads to dynamic network topology and wireless connectivity losses. As such, user mobility is usually considered a “trouble maker” and a great challenge to FL over mobile networks. Interestingly, we found that small user mobility can positively contribute to improving FL performance. This is because the total dataset size and the data diversity that the FL can utilize are increased by user mobility. Based on this observation, we aim to tame and exploit mobility instead of treating it as a hostile “trouble maker”. To this end, we first investigate how the FL performance changes with user mobility theoretically by jointly taking into account the positive and negative aspects of mobility. Specifically, a closed-form expression to quantify the impact of mobility on the FL loss is derived, which explains when negative or positive aspects of mobility dominate the FL performance. Next, a joint FL and communication optimization problem is formulated based on theoretical analyses to minimize the FL loss function by optimizing wireless resource allocation. Finally, we propose a two-step optimization algorithm to solve the formulated problem. The simulation results verify the theoretical analyses. It is also shown that the proposed method can significantly enhance learning performance considering users with high mobility. When the average velocity is larger than 150 km/h, the proposed method achieves more than 80% accuracy in the MNIST dataset, while the existing methods may fail during training.

The daily practice of sharing images on social media raises a severe issue about privacy leakage. To address the issue, privacy-leaking image detection is studied recently, with the goal to automatically identify images that may leak privacy. Recent advance on this task benefits from focusing on crucial objects via pretrained object detectors and modeling their correlation. However, these methods have two limitations: 1) they neglect other important elements like scenes, textures, and objects beyond the capacity of pretrained object detectors. 2) the correlation among objects is fixed, but a fixed correlation is not appropriate for all the images. To overcome the limitations, we propose the Dynamic Region-Aware Graph Convolutional Network (DRAG) that dynamically finds out crucial regions including objects and other important elements, and model their correlation adaptively for each input image. To find out crucial regions, we cluster spatially-correlated feature channels into several region-aware feature maps. Furthermore, we dynamically model the correlation with the self-attention mechanism and explore the interaction among the regions with a graph convolutional network. The DRAG achieved an accuracy of 87% on the largest dataset for privacy-leaking image detection, which is 10 percentage points higher than the state of the art. The further case study demonstrates that it found out crucial regions containing not only objects but other important elements like textures. The code and more details are in https://github.com/guang-yanng/DRAG.

Social media spreads both real news and fake news in various domains including politics, health, entertainment, etc. It is crucial to automatically detect fake news, especially for news of influential domains like politics and health because they may lead to serious social impact, e.g., panic in the COVID-19 pandemic. Some studies indicate the correlation between domains and perform multi-domain fake news detection. However, these multi-domain methods suffer from a seesaw problem that the performance of some domains is often improved by hurting the performance of other domains, which could lead to an unsatisfying performance in the specific target domains. To address this issue, we propose a Domain- and Instance-level Transfer Framework for Fake News Detection (DITFEND), which could improve the performance of specific target domains. To transfer coarse-grained domain-level knowledge, we train a general model with data of all domains from the meta-learning perspective. To transfer fine-grained instance-level knowledge and adapt the general model to a target domain, a language model is trained on the target domain to evaluate the transferability of each data instance in source domains and re-weight the instance’s contribution. Experiments on two real-world datasets demonstrate the effectiveness of DITFEND. According to both offline and online experiments, the DITFEND shows superior effectiveness for fake news detection.

Human motion transfer refers to synthesizing photo-realistic and temporally coherent videos that enable one person to imitate the motion of others. However, current synthetic videos suffer from the temporal inconsistency in sequential frames that significantly degrades the video quality, yet is far from solved by existing methods in the pixel domain. Recently, some works on DeepFake detection try to distinguish the natural and synthetic images in the frequency domain because of the frequency insufficiency of image synthesizing methods. Nonetheless, there is no work to study the temporal inconsistency of synthetic videos from the aspects of the frequency-domain gap between natural and synthetic videos. Therefore, in this paper, we propose to delve into the frequency space for temporally consistent human motion transfer. First of all, we make the first comprehensive analysis of natural and synthetic videos in the frequency domain to reveal the frequency gap in both the spatial dimension of individual frames and the temporal dimension of the video. To close the frequency gap between the natural and synthetic videos, we propose a novel Frequency-based human MOtion TRansfer framework, named FreMOTR, which can effectively mitigate the spatial artifacts and the temporal inconsistency of the synthesized videos. FreMOTR explores two novel frequency-based regularization modules: 1) the Frequency-domain Appearance Regularization (FAR) to improve the appearance of the person in individual frames and 2) Temporal Frequency Regularization (TFR) to guarantee the temporal consistency between adjacent frames. Finally, comprehensive experiments demonstrate that the FreMOTR not only yields superior performance in temporal consistency metrics but also improves the frame-level visual quality of synthetic videos. In particular, the temporal consistency metrics are improved by nearly 30% than the state-of-the-art model.

Neural implicit function based on signed distance field (SDF) has achieved impressive progress in reconstructing 3D models with high fidelity. However, such approaches can only represent closed shapes. Recent works based on unsigned distance function (UDF) are proposed to handle both watertight and open surfaces. Nonetheless, as UDF is signless, its direct output is limited to point cloud, which imposes an additional challenge on extracting high-quality meshes from discrete points.To address this issue, we present a new learnable implicit representation, coded HSDF, that connects the good ends of SDF and UDF. In particular, HSDF is able to represent arbitrary topologies containing both closed and open surfaces while being compatible with existing iso-surface extraction techniques for easy field-to-mesh conversion. In addition to predicting a UDF, we propose to learn an additional sign field via a simple classifier. Unlike traditional SDF, HSDF is able to locate the surface of interest before level surface extraction by generating surface points following NDF~\cite{chibane2020ndf}. We are then able to obtain open surfaces via an adaptive meshing approach that only instantiates regions containing surface into a polygon mesh. We also propose HSDF-Net, a dedicated learning framework that factorizes the learning of HSDF into two easier problems. Experiments on multiple datasets show that HSDF outperforms state-of-the-art techniques both qualitatively and quantitatively.

MOTIVATION: Cryo-electron microscopy (cryo-EM) is a widely used technology for ultrastructure determination, which constructs the 3D structures of protein and macromolecular complex from a set of 2D micrographs. However, limited by the electron beam dose, the micrographs in cryo-EM generally suffer from the extremely low signal-to-noise ratio (SNR), which hampers the efficiency and effectiveness of downstream analysis. Especially, the noise in cryo-EM is not simple additive or multiplicative noise whose statistical characteristics are quite different from the ones in natural image, extremely shackling the performance of conventional denoising methods. RESULTS: Here, we introduce the Noise-Transfer2Clean (NT2C), a denoising deep neural network (DNN) for cryo-EM to enhance image contrast and restore specimen signal, whose main idea is to improve the denoising performance by correctly learning the noise distribution of cryo-EM images and transferring the statistical nature of noise into the denoiser. Especially, to cope with the complex noise model in cryo-EM, we design a contrast-guided noise and signal re-weighted algorithm to achieve clean-noisy data synthesis and data augmentation, making our method authentically achieve signal restoration based on noise's true properties. Our work verifies the feasibility of denoising based on mining the complex cryo-EM noise patterns directly from the noise patches. Comprehensive experimental results on simulated datasets and real datasets show that NT2C achieved a notable improvement in image denoising, especially in background noise removal, compared with the commonly used methods. Moreover, a case study on the real dataset demonstrates that NT2C can greatly alleviate the obstacles caused by the SNR to particle picking and simplify the identifying of particles. AVAILABILITYAND IMPLEMENTATION: The code is available at https://github.com/Lihongjia-ict/NoiseTransfer2Clean/. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Fake news spread widely on social media in various domains, which lead to real-world threats in many aspects like politics, disasters, and finance. Most existing approaches focus on single-domain fake news detection (SFND), which leads to unsatisfying performance when these methods are applied to multi-domain fake news detection. As an emerging field, multi-domain fake news detection (MFND) is increasingly attracting attention. However, data distributions, such as word frequency and propagation patterns, vary from domain to domain, namely domain shift. Facing the challenge of serious domain shift, existing fake news detection techniques perform poorly for multi-domain scenarios. Therefore, it is demanding to design a specialized model for MFND. In this paper, we first design a benchmark of fake news dataset for MFDN with domain label annotated, namely Weibo21, which consists of 4,488 fake news and 4,640 real news from 9 different domains. We further propose an effective Multi-domain Fake News Detection Model (MDFEND) by utilizing domain gate to aggregate multiple representations extracted by a mixture of experts. The experiments show that MDFEND can significantly improve the performance of multi-domain fake news detection. Our dataset and code are available at https://github.com/kennqiang/MDFEND-Weibo21.

Automatic breast cancer grading methods based on HE stained pathological images can be summarized into two categories. The first category is to use learning-based methods to directly extract the features of the pathological image for breast cancer grading. However, unlike the coarse-grained problem of breast cancer classification, grading of breast Invasive Ductal Carcinoma (IDC) is a fine-grained classification problem. Only using general methods cannot classify IDC well. The second category is to conduct the three evaluation criteria of Nottingham Grading System (NGS) separately, and then integrate the results of the three criteria to obtain the final IDC grading result. However, NGS is only a semi-quantitative evaluation method. The inherent medical motivation of NGS is to grade IDC with the help of nuclei-related features. In this paper, we proposed a nuclei-aware network for IDC grading in pathological images. The entire network achieves an effect similar to the attention mechanism in end-to-end learning, so as to learn fine-grained and nuclei-related feature representations for IDC grading. It should to be pointed out that our method can emphasize custom areas, thus providing a way to model medical knowledge into the network structure. This is different from the general attention mechanism that cannot artificially control the area of attention. Experimental results show that the performance of proposed method is better than the state-of-the-art.

Solving long-tail large vocabulary object detection with deep learning based models is a challenging and demanding task, which is however under-explored. In this work, we provide the first systematic analysis on the underperformance of state-of-the-art models in front of long-tail distribution. We find existing detection methods are unable to model few-shot classes when the dataset is extremely skewed, which can result in classifier imbalance in terms of parameter magnitude. Directly adapting long-tail classification models to detection frameworks can not solve this problem due to the intrinsic difference between detection and classification. In this work, we propose a novel balanced group softmax (BAGS) module for balancing the classifiers within the detection frameworks through group-wise training. It implicitly modulates the training process for the head and tail classes and ensures they are both sufficiently trained, without requiring any extra sampling for the instances from the tail classes. Extensive experiments on the very recent long-tail large vocabulary object recognition benchmark LVIS show that our proposed BAGS significantly improves the performance of detectors with various backbones and frameworks on both object detection and instance segmentation. It beats all state-of-the-art methods transferred from long-tail image classification and establishes new state-of-the-art. Code is available at https://github.com/FishYuLi/BalancedGroupSoftmax.

Many existing scene parsing methods adopt Convolutional Neural Networks with receptive fields of fixed sizes and shapes, which frequently results in inconsistent predictions of large objects and invisibility of small objects. To tackle this issue, we propose perspective-adaptive convolutions to acquire receptive fields of flexible sizes and shapes during scene parsing. Through adding a new perspective regression layer, we can dynamically infer the position-adaptive perspective coefficient vectors utilized to reshape the convolutional patches. Consequently, the receptive fields can be adjusted automatically according to the various sizes and perspective deformations of the objects in scene images. Our proposed convolutions are differentiable to learn the convolutional parameters and perspective coefficients in an end-to-end way without any extra training supervision of object sizes. Furthermore, considering that the standard convolutions lack contextual information and spatial dependencies, we propose a context adaptive bias to capture both local and global contextual information through average pooling on the local feature patches and global feature maps, followed by flexible attentive summing to the convolutional results. The attentive weights are position-adaptive and context-aware, and can be learned through adding an additional context regression layer. Experiments on Cityscapes and ADE20K datasets well demonstrate the effectiveness of the proposed methods.

Retweeting is a powerful driving force in information propagation on microblogging sites. However, identifying the most effective retweeters of a message (called the ”key retweeter prediction” problem) has become a significant research topic. Conventional approaches have addressed this topic from two main aspects: by analyzing either the personal attributes of microblogging users or the structures of user graph networks. However, according to sociological findings, author-retweeter dependencies also play a crucial role in influencing message propagation. In this paper, we propose a novel model to solve the key retweeter prediction problem by incorporating the auxiliary relations between a tweet author and potential retweeters. Without loss of generality, we formulate the relations from four relational factors: status relation, temporal relation, locational relation, and interactive relation. In addition, we propose a novel method, called “Relation-based Learning to Rank (RL2R),” to determine the key retweeters for a given tweet by ranking the potential retweeters in terms of their spreadability. The experimental results show that our method outperforms the state-of-the-art algorithms at top-k retweeter prediction, achieving a significant relative average improvement of 19.7-29.4 percent. These findings provide new insights for understanding user behaviors on social media for key retweeter prediction purposes.

Mobile edge computing (MEC)-enabled vehicular networks can improve the quality of service (QoS) of vehicular networks, such as the round-trip time (RTT) and transmission control protocol (TCP) throughput. However, the high mobility of vehicles requires frequent service migrations among MEC servers to maintain the QoS. Frequent service migrations incur prohibitive migration cost. To achieve the tradeoff between the QoS and migration cost, this paper proposes a novel dynamic service migration scheme, which considers the effect of velocities of vehicles. The main idea is that the QoS and migration cost are modeled as the function of velocity, and then they are jointly considered economically. The system captures incomes from vehicles according to their QoS. The cost (expenditure) consists of the migration cost and service cost for the use of computing, communication and memory resources to provide service. The system utility is defined as the difference between incomes and costs. A novel deep reinforcement learning algorithm, i.e., deep Q-learning is employed to maximize the system utility, by designing dynamic service migration scheme. Simulation results show that compared with existing migration schemes, the proposed dynamic scheme can increase the system utility with various velocities. The system utility gain increases as the velocity increases, which reaches about 2 times when the velocity is larger than 30m/s. Moreover, it improves the QoS of vehicles with the higher mobility, where the RTT can be decreased by 2 times, and the TCP throughput can be increased by 1 time.

The increasing popularity of social media promotes the proliferation of fake news. With the development of multimedia technology, fake news attempts to utilize multimedia content with images or videos to attract and mislead readers for rapid dissemination, which makes visual content an important part of fake news. Fake-news images, images attached to fake news posts, include not only fake images that are maliciously tampered but also real images that are wrongly used to represent irrelevant events. Hence, how to fully exploit the inherent characteristics of fake-news images is an important but challenging problem for fake news detection. In the real world, fake-news images may have significantly different characteristics from real-news images at both physical and semantic levels, which can be clearly reflected in the frequency and pixel domain, respectively. Therefore, we propose a novel framework Multi-domain Visual Neural Network (MVNN) to fuse the visual information of frequency and pixel domains for detecting fake news. Specifically, we design a CNN-based network to automatically capture the complex patterns of fake-news images in the frequency domain; and utilize a multi-branch CNN-RNN model to extract visual features from different semantic levels in the pixel domain. An attention mechanism is utilized to fuse the feature representations of frequency and pixel domains dynamically. Extensive experiments conducted on a real world dataset demonstrate that MVNN outperforms existing methods with at least 9.2% in accuracy, and can help improve the performance of multi-modal fake news detection by over 5.2%.

Most existing semantic segmentation methods employ atrous convolution to enlarge the receptive field of filters, but neglect partial information. To tackle this issue, we firstly propose a novel Kronecker convolution which adopts Kronecker product to expand the standard convolutional kernel for taking into account the partial feature neglected by atrous convolutions. Therefore, it can capture partial information and enlarge the receptive field of filters simultaneously without introducing extra parameters. Secondly, we propose a Tree-structured Feature Aggregation (TFA) module which follows a recursive rule to expand and forms a hierarchical structure. Thus, it can naturally learn representations of multi-scale objects and encode hierarchical contextual information in complex scenes. Finally, we design a Tree-structured Kronecker Convolutional Network (TKCN) which employs Kronecker convolution and TFA module. Extensive experiments on three datasets, PAS-CAL VOC 2012, PASCAL-Context and Cityscapes, verify the effectiveness of our proposed approach.

In ultra-dense cellular networks (UDNs) with user/control plane (U/C) splitting, frequent handovers in user planes are unavoidable. This seriously degrades MS's transmission control protocol (TCP) throughput. This paper proposes a prediction-based user plane handover scheme to improve the TCP throughput in UDNs. Firstly, based on algorithms used in recommender systems, a mobility prediction algorithm called content-based collaborative hybrid filters (CCHF) is proposed to predict the target small base station (SBS). When the mobile station (MS) moves into the cell-edge of the source SBS, it can set up connections to the predicted target SBS and the source SBS simultaneously. An accurate prediction and a simultaneous connection can enhance the signal to interference and noise ratio (SINR) at cell-edge and reduce the handover interruption ratio (HIR). Thus packet loss can be reduced and the MS's TCP throughput will be improved. Simulations are carried out to verify the effectiveness of the proposed CCHF-handover. It is shown that using CCHF, the prediction accuracy of random trajectory can be improved by more than 100% compared with existing prediction algorithm. Moreover, the CCHF-handover improves the average TCP throughput significantly by more than 3 times compared with that of existing handover schemes.

Unpaired image-to-image translation problem aims to model the mapping from one domain to another with unpaired training data. Current works like the well-acknowledged Cycle GAN provide a general solution for any two domains through modeling injective mappings with a symmetric structure. While in situations where two domains are asymmetric in complexity, i.e., the amount of information between two domains is different, these approaches pose problems of poor generation quality, mapping ambiguity, and model sensitivity. To address these issues, we propose Asymmetric GAN (AsymGAN) to adapt the asymmetric domains by introducing an auxiliary variable (aux) to learn the extra information for transferring from the information-poor domain to the information-rich domain, which improves the performance of state-of-the-art approaches in the following ways. First, aux better balances the information between two domains which benefits the quality of generation. Second, the imbalance of information commonly leads to mapping ambiguity, where we are able to model one-to-many mappings by tuning aux, and furthermore, our aux is controllable. Third, the training of Cycle GAN can easily make the generator pair sensitive to small disturbances and variations while our model decouples the ill-conditioned relevance of generators by injecting aux during training. We verify the effectiveness of our proposed method both qualitatively and quantitatively on asymmetric situation, label-photo task, on Cityscapes and Helen datasets, and show many applications of asymmetric image translations. In conclusion, our AsymGAN provides a better solution for unpaired image-to-image translation in asymmetric domains.

Microblogs have become one of the most popular platforms for news sharing. However, due to its openness and lack of supervision, rumors could also be easily posted and propagated on social networks, which could cause huge panic and threat during its propagation. In this paper, we detect rumors by leveraging hierarchical representations at different levels and the social contexts. Specifically, we propose a novel hierarchical neural network combined with social information (HSA-BLSTM). We first build a hierarchical bidirectional long short-term memory model for representation learning. Then, the social contexts are incorporated into the network via attention mechanism, such that important semantic information is introduced to the framework for more robust rumor detection. Experimental results on two real world datasets demonstrate that the proposed method outperforms several state-of-the-arts in both rumor detection and early detection scenarios.

Both accuracy and speed are equally important in urban scene parsing. Most of the existing methods mainly focus on improving parsing accuracy, ignoring the problem of low inference speed due to large-sized input and high resolution feature maps. To tackle this issue, we propose a High Resolution Feature Recovering (HRFR) framework to accelerate a given parsing network. A Super-Resolution Recovering module is employed to recover features of large original-sized images from features of down-sampled input. Therefore, our framework can combine the advantages of (1) fast speed of networks with down-sampled input and (2) high accuracy of networks with large original-sized input. Additionally, we employ auxiliary intermediate supervision and boundary region re-weighting to facilitate the optimization of the network. Extensive experiments on the two challenging Cityscapes and CamVid datasets well demonstrate the effectiveness of the proposed HRFR framework, which can accelerate the scene parsing inference process by about 3.0x speedup from 1/2 down-sampled input with negligible accuracy reduction.

Style synthesis attracts great interests recently, while few works focus on its dual problem "style separation". In this paper, we propose the Style Separation and Synthesis Generative Adversarial Network (S3-GAN) to simultaneously implement style separation and style synthesis on object photographs of specific categories. Based on the assumption that the object photographs lie on a manifold, and the contents and styles are independent, we employ S3-GAN to build mappings between the manifold and a latent vector space for separating and synthesizing the contents and styles. The S3-GAN consists of an encoder network, a generator network, and an adversarial network. The encoder network performs style separation by mapping an object photograph to a latent vector. Two halves of the latent vector represent the content and style, respectively. The generator network performs style synthesis by taking a concatenated vector as input. The concatenated vector contains the style half vector of the style target image and the content half vector of the content target image. Once obtaining the images from the generator network, an adversarial network is imposed to generate more photo-realistic images. Experiments on CelebA and UT Zappos 50K datasets demonstrate that the S3-GAN has the capacity of style separation and synthesis simultaneously, and could capture various styles in a single model.

In this paper, we focus on improving the proposal classification stage in the object detection task and present implicit negative sub-categorization and sink diversion to lift the performance by strengthening loss function in this stage. First, based on the observation that the "background" class is generally very diverse and thus challenging to be handled as a single indiscriminative class in existing state-of-the-art methods, we propose to divide the background category into multiple implicit sub-categories to explicitly differentiate diverse patterns within it. Second, since the ground truth class inevitably has low-value probability scores for certain images, we propose to add a "sink" class and divert the probabilities of wrong classes to this class when necessary, such that the ground truth label will still have a higher probability than other wrong classes even though it has low probability output. Additionally, we propose to use dilated convolution, which is widely used in the semantic segmentation task, for efficient and valuable context information extraction. Extensive experiments on PASCAL VOC 2007 and 2012 data sets show that our proposed methods based on faster R-CNN implementation can achieve state-of-the-art mAPs, i.e., 84.1%, 82.6%, respectively, and obtain 2.5% improvement on ILSVRC DET compared with that of ResNet.

Compared with traditional image classification, fine-grained visual categorization is a more challenging task, because it targets to classify objects belonging to the same species, e.g., classify hundreds of birds or cars. In the past several years, researchers have made many achievements on this topic. However, most of them are heavily dependent on the artificial annotations, e.g., bounding boxes, part annotations, and so on. The requirement of artificial annotations largely hinders the scalability and application. Motivated to release such dependence, this paper proposes a robust and discriminative visual description named Automated Bi-level Description (AutoBD). “Bi-level” denotes two complementary part-level and object-level visual descriptions, respectively. AutoBD is “automated,” because it only requires the image-level labels of training images and does not need any annotations for testing images. Compared with the part annotations labeled by the human, the image-level labels can be easily acquired, which thus makes AutoBD suitable for large-scale visual categorization. Specifically, the part-level description is extracted by identifying the local region saliently representing the visual distinctiveness. The object-level description is extracted from object bounding boxes generated with a co-localization algorithm. Although only using the image-level labels, AutoBD outperforms the recent studies on two public benchmark, i.e., classification accuracy achieves 81.6% on CUB-200-2011 and 88.9% on Car-196, respectively. On the large-scale Birdsnap data set, AutoBD achieves the accuracy of 68%, which is currently the best performance to the best of our knowledge.

Apache Hive has been widely used for big data processing over large scale clusters by many companies. It provides a declarative query language called HiveQL. The efficiency of filtering out query-irrelevant data from HDFS closely affects the performance of query processing. This is especially true for multi-dimensional, high-selective, and few columns involving queries, which provides sufficient information to reduce the amount of bytes read. Indexing (Compact Index, Aggregate Index, Bitmap Index, DGFIndex, and the index in ORC file) and columnar storage (RCFile, ORC file, and Parquet) are powerful techniques to achieve this. However, it is not trivial to choosing a suitable index and columnar storage based on data and query features. In this paper, we compare the data filtering performance of the above indexes with different columnar storage formats by conducting comprehensive experiments using uniform and skew TPC-H data sets and various multi-dimensional queries, and suggest the best practices of improving multi-dimensional queries in Hive under different conditions.

Integrating complementary features from multiple channels is expected to solve the description ambiguity problem in video captioning, whereas inappropriate fusion strategies often harm rather than help the performance. Existing static fusion methods in video captioning such as concatenation and summation cannot attend to appropriate feature channels, thus fail to adaptively support the recognition of various kinds of visual entities such as actions and objects. This paper contributes to: 1)The first in-depth study of the weakness inherent in data-driven static fusion methods for video captioning. 2) The establishment of a task-driven dynamic fusion (TDDF) method. It can adaptively choose different fusion patterns according to model status. 3) The improvement of video captioning. Extensive experiments conducted on two well-known benchmarks demonstrate that our dynamic fusion method outperforms the state-of-the-art results on MSVD with METEOR scores 0.333, and achieves superior METEOR scores 0.278 on MSR-VTT-10K. Compared to single features, the relative improvement derived from our fusion method are 10.0% and 5.7% respectively on two datasets.

Many existing scene parsing methods adopt Convolutional Neural Networks with fixed-size receptive fields, which frequently result in inconsistent predictions of large objects and invisibility of small objects. To tackle this issue, we propose a scale-adaptive convolution to acquire flexiblesize receptive fields during scene parsing. Through adding a new scale regression layer, we can dynamically infer the position-adaptive scale coefficients which are adopted to resize the convolutional patches. Consequently, the receptive fields can be adjusted automatically according to the various sizes of the objects in scene images. Thus, the problems of invisible small objects and inconsistent large-object predictions can be alleviated. Furthermore, our proposed scale-adaptive convolutions are not only differentiable to learn the convolutional parameters and scale coefficients in an end-to-end way, but also of high parallelizability for the convenience of GPU implementation. Additionally, since the new scale regression layers are learned implicitly, any extra training supervision of object sizes is unnecessary. Extensive experiments on Cityscapes and ADE20K datasets well demonstrate the effectiveness of the proposed scaleadaptive convolutions.

This paper targets to bring together the research efforts on two fields that are growing actively in the past few years: multicamera person Re-Identification (ReID) and large-scale image retrieval. We demonstrate that the essentials of image retrieval and person ReID are the same, i.e., measuring the similarity between images. However, person ReID requires more discriminative and robust features to identify the subtle differences of different persons and overcome the large variance among images of the same person. Specifically, we propose a coarse-to-fine (C2F) framework and a Convolutional Neural Network structure named as Conv-Net to tackle the large-scale person ReID as an image retrieval task. Given a query person image, the C2F firstly employ Conv-Net to extract a compact descriptor and perform the coarse-level search. A robust descriptor conveying more spatial cues is hence extracted to perform the fine-level search. Extensive experimental results show that the proposed method outperforms existing methods on two public datasets. Further, the evaluation on a large-scale Person-520K dataset demonstrates that our work is significantly more efficient than existing works, e.g., only needs 180ms to identify a query person from 520K images.

Prediction of popularity has profound impact for social media, since it offers opportunities to reveal individual preference and public attention from evolutionary social systems. Previous research, although achieves promising results, neglects one distinctive characteristic of social data, i.e., sequentiality. For example, the popularity of online content is generated over time with sequential post streams of social media. To investigate the sequential prediction of popularity, we propose a novel prediction framework called Deep Temporal Context Networks (DTCN) by incorporating both temporal context and temporal attention into account. Our DTCN contains three main components, from embedding, learning to predicting. With a joint embedding network, we obtain a unified deep representation of multi-modal user-post data in a common embedding space. Then, based on the embedded data sequence over time, temporal context learning attempts to recurrently learn two adaptive temporal contexts for sequential popularity. Finally, a novel temporal attention is designed to predict new popularity (the popularity of a new user-post pair) with temporal coherence across multiple time-scales. Experiments on our released image dataset with about 600K Flickr photos demonstrate that DTCN outperforms state-of-the-art deep prediction algorithms, with an average of 21.51% relative performance improvement in the popularity prediction (Spearman Ranking Correlation).

Most of existing scene parsing methods suffer from the serious problems of both inconsistent parsing results and object boundary shift. To tackle these problems, we first propose an iterative Global-residual Refinement Network (GRN) through exploiting global contextual information to predict the parsing residuals and iteratively smoothen the inconsistent parsing labels. Furthermore, we propose a Local-boundary Refinement Network (LRN) to learn the position-adaptive propagation coefficients so that local contextual information from neighbors can be optimally captured for refining object boundaries. Finally, we cascade the proposed two refinement networks after a fully residual convolutional neural network within a uniform framework. Extensive experiments on ADE20K and Cityscapes datasets well demonstrate the effectiveness of the two refinement methods for refining scene parsing predictions.

Fine-Grained Visual Categorization (FGVC) has achieved significant progress recently. However, the number of fine-grained species could be huge and dynamically increasing in real scenarios, making it difficult to recognize unseen objects under the current FGVC framework. This raises an open issue to perform large-scale fine-grained identification without a complete training set. Aiming to conquer this issue, we propose a retrieval task named One-Shot Fine-Grained Instance Retrieval (OSFGIR). "One-Shot" denotes the ability of identifying unseen objects through a fine-grained retrieval task assisted with an incomplete auxiliary training set. This paper first presents the detailed description to OSFGIR task and our collected OSFGIR-378K dataset. Next, we propose the Convolutional and Normalization Networks (CN-Nets) learned on the auxiliary dataset to generate a concise and discriminative representation. Finally, we present a coarse-to-fine retrieval framework consisting of three components, i.e., coarse retrieval, fine-grained retrieval, and query expansion, respectively. The framework progressively retrieves images with similar semantics, and performs fine-grained identification. Experiments show our OSFGIR framework achieves significantly better accuracy and efficiency than existing FGVC and image retrieval methods, thus could be a better solution for large-scale fine-grained object identification.

Learning image similarity plays a critical role in real-world multimedia information retrieval applications, especially in Content-Based Image Retrieval (CBIR) tasks, in which an accurate retrieval of visually similar objects largely relies on an effective image similarity function. Crafting a good similarity function is very challenging because visual contents of images are often represented as feature vectors in high-dimensional spaces, for example, via bag-of-words (BoW) representations, and traditional rigid similarity functions, for example, cosine similarity, are often suboptimal for CBIR tasks. In this article, we address this fundamental problem, that is, learning to optimize image similarity with sparse and high-dimensional representations from large-scale training data, and propose a novel scheme of Sparse Online Learning of Image Similarity (SOLIS). In contrast to many existing image-similarity learning algorithms that are designed to work with low-dimensional data, SOLIS is able to learn image similarity from large-scale image data in sparse and high-dimensional spaces. Our encouraging results showed that the proposed new technique achieves highly competitive accuracy as compared to the state-of-the-art approaches but enjoys significant advantages in computational efficiency, model sparsity, and retrieval scalability, making it more practical for real-world multimedia retrieval applications.

When packing for a journey, have you ever asked “what clothes should I take with me?” Wearing appropriate and aesthetically pleasing clothing when traveling is a concern for many of us. Our data observation of photos from several popular travel websites reveals that people's choice of clothing items and their color combinations have strong correlations with the weather, the season, and the main type of attraction at the destination. This leads to an interesting and novel problem: can the correlation between clothing and locations be automatically learned from social photos and leveraged for location-oriented clothing recommendations? In this paper, we systematically study this problem and propose a hybrid multilabel convolutional neural network combined with the support vector machine (mCNN-SVM) approach to capture the intrinsic and complex correlations between clothing attributes and location attributes. Specifically, we adapt the CNN architecture to multilabel learning and fine-tune it using each fine-grained clothing item. Then, the recognized items are fed to the SVM to learn the correlations. Experiments on three fashion datasets and a benchmark journey outfit dataset show that our proposed approach outperforms several baselines by over 10.52-16.38% in terms of the mAP for clothing item recognition and outperforms several alternative methods by over 9.59-29.41% in terms of the mAP when ranking clothing by appropriateness for travel destinations. Finally, an interesting case study demonstrates the effectiveness of our method by answering what items to wear, how to match them, and how to dress in an aesthetically pleasing manner for a journey.

Recent years have witnessed the significant advance in fine-grained visual categorization, which targets to classify the objects belonging to the same species. To capture enough subtle visual differences and build discriminative visual description, most of the existing methods heavily rely on the artificial part annotations, which are expensive to collect in real applications. Motivated to conquer this issue, this paper proposes a multi-level coarse-to-fine object description. This novel description only requires the original image as input, but could automatically generate visual descriptions discriminative enough for fine-grained visual categorization. This description is extracted from five sources representing coarse-to-fine visual clues: 1) original image is used as the source of global visual clue; 2) object bounding boxes are generated using convolutional neural network (CNN); 3) with the generated bounding box, foreground is segmented using the proposed k nearest neighbour-based co-segmentation algorithm; and 4) two types of part segmentations are generated by dividing the foreground with an unsupervised part learning strategy. The final description is generated by feeding these sources into CNN models and concatenating their outputs. Experiments on two public benchmark data sets show the impressive performance of this coarse-to-fine description, i.e., classification accuracy achieves 82.5% on CUB-200-2011, and 86.9% on fine-grained visual categorization-Aircraft, respectively, which outperform many recent works.

We present a deep learning strategy to fuse multiple semantic cues for complex event recognition. In particular, we tackle the recognition task by answering how to jointly analyze human actions (who is doing what), objects (what), and scenes (where). First, each type of semantic features (e.g., human action trajectories) is fed into a corresponding multi-layer feature abstraction pathway, followed by a fusion layer connecting all the different pathways. Second, the correlations of how the semantic cues interacting with each other are learned in an unsupervised cross-modality autoencoder fashion. Finally, by fine-tuning a large-margin objective deployed on this deep architecture, we are able to answer the question on how the semantic cues of who, what, and where compose a complex event. As compared with the traditional feature fusion methods (e.g., various early or late strategies), our method jointly learns the essential higher level features that are most effective for fusion and recognition. We perform extensive experiments on two real-world complex event video benchmarks, MED'11 and CCV, and demonstrate that our method outperforms the best published results by 21% and 11%, respectively, on an event recognition task.

This demo presents an extremely efficient concept detection system based on a novel bag of words extraction method and sparse ensemble learning. We will show that the presented system can efficiently build the concept detectors upon millions of images, and achieve real-time concept detection on unseen images with the state-of-the-arts accuracy. To do so, we first develop an efficient bag of visual words (BoW) construction method based on sparse non-negative matrix factorization (NMF) and GPU enabled SIFT feature extraction. We then develop a sparse ensemble learning method to build the detection model, which drastically reduces learning time in order of magnitude over traditional methods like Support Vector Machine. The demo video of the system is available at YouTube: http://youtu.be/57obnlCxqAs

Conventional local detectors often extract numerous small repeated regions in textured areas, which easily results in false matching. In order to find representative and distinctive local invariant regions, this paper proposes a Maximally Visual-Homogeneous Region (MVHR) detector. The main contributions can be summarized as 2 parts: (1) Being different from original MSER which employs single pixel intensity as ranking unit, we propose a novel sorting method based on visual homogeneity analysis on a local patch. (2) Identifying the observation scale has a close relationship with visual homogeneity analysis, a heuristic scale selection algorithm is developed to choose a proper scale according to the changes of visual homogeneity evaluation over a range of scales. Experiments demonstrate our detector can find less but representative regions with high repeatability, while still perserving competitive precision compared to the state-of-art detectors for large scale image retrieval.

The task of recognizing events from video has attracted a lot of attention in recent years. However, due to the complex nature of user-defined events, the use of purely audio- visual content analysis without domain knowledge has been found to be grossly inadequate. In this paper, we propose to construct a semantic-visual knowledge base to encode the rich event-centric concepts and their relationships from the well- established lexical databases, including FrameNet, as well as the concept-specific visual knowledge from ImageNet. Based on this semantic-visual knowledge bases, we design an effective system for video event recognition. Specifically, in order to narrow the semantic gap between the high-level complex events and low-level visual representations, we utilize the event-centric semantic concepts encoded in the knowledge base as the intermediate-level event representation, which offers both human-perceivable and machine-interpretable semantic clues for event recognition. In addition, in order to leverage the abundant ImageNet images, we propose a robust transfer learning model to learn the noise- resistant concept classifiers for videos. Extensive experiments on various real-world video datasets demonstrate the superiority of our proposed system as compared to the state-of-the-art approaches.

Image similarity search plays a key role in many multimediaapplications, where multimedia data (such as images and videos) areusually represented in high-dimensional feature space. In thispaper, we propose a novel Sparse Online Metric Learning (SOML)scheme for learning sparse distance functions from large-scalehigh-dimensional data and explore its application to imageretrieval. In contrast to many existing distance metric learningalgorithms that are often designed for low-dimensional data, theproposed algorithms are able to learn sparse distance metrics fromhigh-dimensional data in an efficient and scalable manner. Ourexperimental results show that the proposed method achieves betteror at least comparable accuracy performance than thestate-of-the-art non-sparse distance metric learning approaches, butenjoys a significant advantage in computational efficiency andsparsity, making it more practical for real-world applications.

Recently, several binary descriptors are proposed, which represent interest points in image using binary codes. In these binary feature schemes, two descriptors are considered as a match, if the Hamming distance between them is below a threshold. Applying Hamming distance to measure the similarity between binary descriptors can extremely promote the computational efficiency. However, our experimental results presents that there exists a large number of bits in the binary feature vector cannot maintain the robustness while image conditions change. Rather than ignore the impacts of those unstable bits, we take into account the difference of robustness among the feature bits and propose a novel similarity measurement, which called the Fragile Bit Ratio (FBR). FBR is used in binary feature matching to measure how two features differ. High FBRs are associated with genuine matches between two binary features and low FBRs are associated with impostor ones. Based on this metric, we propose a new binary feature matching scheme to fuse the Hamming distance and Fragile Bit Ratio. In our approach, we match the descriptors using the Hamming distance threshold roughly, and then filtered by the Fragile Bits Ratio to refine the candidate set. In experiments, using Fragile Bits Radio can effectively remove the false matches and highly improve the accuracy of image search. Furthermore, our method can easily be integrated into the other well-established binary features schemes.

Local features have been widely used in many computer vision related researches, such as near-duplicate image and video retrieval. However, the storage and query cost of local features become prohibitive on large-scale database. In this paper, we propose a representative local features mining method to generate a compact but more effective feature subset. First, we do an unsupervised annotation for all similar images(or frames in video) in the database. Second, we compute a comprehensive score for every local feature. The score function combines the robustness and discrimination. Finally, we sort all the local features in an image by their scores and the low-score local features can be removed. The selected local features are robust and discriminative, which can guarantee the better retrieval quality than using full of the original feature set. By our method, the number of local features can be significantly reduced and a large amount of storage and computational cost can be saved. The experimental results show that we can use 30% of the features to get a better query performance than that of full feature set.

Compressed Sensing (CS) has drawn quite an amount of attention as novel digital signal sampling theory in recent years when the signal is sparse in some domain. However, signal reconstruction from undersampled data has always been challenging due to its implicit ill-posed nature. This paper proposes an image compressed sensing reconstruction algorithm for image CS application, which consists of iteratively collaborative filtering of non local similar image patches in 3D transform domain and solving the least squares problems. In addition, the linearization technique is exploited to reduce the computation complexity. The results of various experiments on natural images and MRI images consistently demonstrate that the proposed algorithm can efficiently reconstruct images and gain more 2dB as compared to the current leading CS image reconstruction algorithm.

Learning effective feature representations and similarity measures are crucial to the retrieval performance of a content-based image retrieval (CBIR) system. Despite extensive research efforts for decades, it remains one of the most challenging open problems that considerably hinders the successes of real-world CBIR systems. The key challenge has been attributed to the well-known ``semantic gap'' issue that exists between low-level image pixels captured by machines and high-level semantic concepts perceived by human. Among various techniques, machine learning has been actively investigated as a possible direction to bridge the semantic gap in the long term. Inspired by recent successes of deep learning techniques for computer vision and other applications, in this paper, we attempt to address an open problem: if deep learning is a hope for bridging the semantic gap in CBIR and how much improvements in CBIR tasks can be achieved by exploring the state-of-the-art deep learning techniques for learning feature representations and similarity measures. Specifically, we investigate a framework of deep learning with application to CBIR tasks with an extensive set of empirical studies by examining a state-of-the-art deep learning method (Convolutional Neural Networks) for CBIR tasks under varied settings. From our empirical studies, we find some encouraging results and summarize some important insights for future research.

In Smart Grid applications, as the number of deployed electric smart meters increases, massive amounts of valuable meter data is generated and collected every day. To enable reliable data collection and make business decisions fast, high throughput storage and high-performance analysis of massive meter data become crucial for grid companies. Considering the advantage of high efficiency, fault tolerance, and price-performance of Hadoop and Hive systems, they are frequently deployed as underlying platform for big data processing. However, in real business use cases, these data analysis applications typically involve multidimensional range queries (MDRQ) as well as batch reading and statistics on the meter data. While Hive is high-performance at complex data batch reading and analysis, it lacks efficient indexing techniques for MDRQ. In this paper, we propose DGFIndex, an index structure for Hive that efficiently supports MDRQ for massive meter data. DGFIndex divides the data space into cubes using the grid file technique. Unlike the existing indexes in Hive, which stores all combinations of multiple dimensions, DGFIndex only stores the information of cubes. This leads to smaller index size and faster query processing. Furthermore, with pre-computing user-defined aggregations of each cube, DGFIndex only needs to access the boundary region for aggregation query. Our comprehensive experiments show that DGFIndex can save significant disk space in comparison with the existing indexes in Hive and the query performance with DGFIndex is 2-50 times faster than existing indexes in Hive and HadoopDB for aggregation query, 2-5 times faster than both for non-aggregation query, 2-75 times faster than scanning the whole table in different query selectivity.

In this article, we propose a unified geolocation framework to automatically determine where on the earth a web video was shot. We analyze different social, visual, and textual relationships from a real-world dataset and find four relationships with apparent geography clues that can be used for web video geolocation. Then, the geolocation process is formulated as an optimization problem that simultaneously takes the social, visual, and textual relationships into consideration. The optimization problem is solved by an iterative procedure, which can be interpreted as a propagation of the geography information among the web video social network. Extensive experiments on a real-world dataset clearly demonstrate the effectiveness of our proposed framework, with the geolocation accuracy higher than state-of-the-art approaches.

In this paper, we study the problem of image retrieval by introducing contextual query expansion to address the shortcomings of bag-of-words based frameworks: semantic gap of visual word quantization, and the efficiency and storage loss due to query expansion. Our method is built on common visual patterns (CVPs), which are the distinctive visual structures between two images and have rich contextual information. With CVPs, two contextual query expansions on visual word-level and image-level are explored, respectively. For visual word-level expansion, we find contextual synonymous visual words (CSVWs) and expand a word in the query image with its CSVWs to boost retrieval accuracy. CSVWs are the words that appear in the same CVPs and have same contextual meaning, i.e. similar spatial layout and geometric transformations. For image-level expansion, the database images that have the same CVPs are organized by linked list and the images that have the same CVPs as the query image, but not included in the results are automatically expanded. The main computation of these two expansions is carried out offline, and they can be integrated into the inverted file and efficiently applied to all images in the dataset. Experiments conducted on three reference datasets and a dataset of one million images demonstrate the effectiveness and efficiency of our method.

This paper presents a novel algorithm for fast and robust video copy detection. The idea is to use local features to estimate the copy transformation parameters first and then use the estimated parameters to guide the global-feature-based matching at a later stage. It is based on the fact that the copy transformations generally remain unchanged in a continuous video clip even in the whole video. Local-feature-based matching can find the candidates which are difficult to be detected only using global features. Furthermore, the matched local feature points can provide enough information to estimate the copy transformations. After the copy transformations are estimated, the subsequent detection can be accelerated by doing global-feature-based matching. The experimental results show that the proposed algorithm can get the same good robustness as the local-feature-based method but the faster detection speed.

This paper describes a motion estimation algorithm based on sparse representation, which can be applied in video coding to reduce the temporal redundancy. The sparse coefficients are firstly calculated in support region by orthogonal matching pursuit (OMP) algorithm using the reference blocks as dictionary elements, and then these optimal sparse coefficients are utilized to predict the current block. To get the same prediction in decoder, the number of iterations in OMP is transmitted to decoder as side information. Simulation results show that gain up to 2.87dB in terms of the PSNR when compared with traditional translational motion estimation model.

Binary representation for large scale nearest neighbor search received more and more concern recently. Although binary codes can be directly used as indices of the hash tables, correlations between the bits may lead to non-uniform codes distribution and reduce the performance of the hash table. In this paper, we propose a data driven multi-index hashing method for exact nearest neighbor search in Hamming space. By exploring the statistics properties of the dataset, we can separate the correlated bits into different segments during the process of building multiple hash tables, and thus make binary codes distributed as uniformly as possible in each hash table. Experiments conducted on a huge amount of binary codes extracted from the UK Bench dataset show that our method can achieve significant acceleration in searching speed for large scale dataset.

This demonstration presents a novel geo-social event detection and tracking system based on geographical pattern mining and content analysis, called "GeSoDeck". A user can capture what events happened by our system. Unlike most existing social event detection applications, GeSoDeck aims to detect events with high accuracy and efficiency, and track them as well. Given a geographical area, the system can not only detect diverse social events in this area using the geographical pattern mining and density-based K-means clustering, but also track the representative tweets of the detected event in real time, mining geographical diffusion trajectory on the map and temporal pattern of retweeting process. On a realistic dataset collected from Sina Weibo, the system can outperform the state-of-the-art methods.

Mobile video is quickly becoming a mass consumer phenomenon. More and more people are using their smartphones to search and browse video content while on the move. In this paper, we have developed an innovative instant mobile video search system through which users can discover videos by simply pointing their phones at a screen to capture a very few seconds of what they are watching. The system is able to index large-scale video data using a new layered audio-video indexing approach in the cloud, as well as extract light-weight joint audio-video signatures in real time and perform progressive search on mobile devices. Unlike most existing mobile video search applications that simply send the original video query to the cloud, the proposed mobile system is one of the first attempts at instant and progressive video search leveraging the light-weight computing capacity of mobile devices. The system is characterized by four unique properties: 1) a joint audio-video signature to deal with the large aural and visual variances associated with the query video captured by the mobile phone, 2) layered audio-video indexing to holistically exploit the complementary nature of audio and video signals, 3) light-weight fingerprinting to comply with mobile processing capacity, and 4) a progressive query process to significantly reduce computational costs and improve the user experience---the search process can stop anytime once a confident result is achieved. We have collected 1,400 query videos captured by 25 mobile users from a dataset of 600 hours of video. The experiments show that our system outperforms state-of-the-art methods by achieving 90.79% precision when the query video is less than 10 seconds and 70.07% even when the query video is less than 5 seconds.

Spectral hashing (SpH) is an efficient and simple binary hashing method, which assumes that data are sampled from a multidimensional uniform distribution. However, this assumption is too restrictive in practice. In this paper we propose an improved method, Fitted Spectral Hashing, to relax this distribution assumption. Our work is based on the fact that one-dimensional data of any distribution could be mapped to a uniform distribution without changing the local neighbor relations among data items. We have found that this mapping on each PCA direction has certain regular pattern, and could fit data well by S-Curve function, Sigmoid function. With more parameters Fourier function also fit data well. Thus with Sigmoid function and Fourier function, we propose two binary hashing methods. Experiments show that our methods are efficient and outperform state-of-the-art methods.

Binary hashing has been widely used for efficient similarity search. Learning efficient codes has become a research focus and it is still a challenge. In many cases, the real-world data often lies on a low-dimensional manifold, which should be taken into account to capture meaningful neighbors with hashing. The importance of a manifold is its topology, which represents the neighborhood relationships between its subregions and the relative proximities between the neighbors of each subregion, e.g. the relative ranking of neighbors of each subregion. Most existing hashing methods try to preserve the neighborhood relationships by mapping similar points to close codes, while ignoring the neighborhood rankings. Moreover, most hashing methods lack in providing a good ranking for query results since they use Hamming distance as the similarity metric, and in practice, there are often a lot of results sharing the same distance to a query. In this paper, we propose a novel hashing method to solve these two issues jointly. The proposed method is referred to as Topology Preserving Hashing (TPH). TPH is distinct from prior works by preserving the neighborhood rankings of data points in Hamming space. The learning stage of TPH is formulated as a generalized eigendecomposition problem with closed form solutions. Experimental comparisons with other state-of-the-art methods on three noted image benchmarks demonstrate the efficacy of the proposed method.

Accurate estimation of human body orientation can significantly enhance the analysis of human behavior, which is a fundamental task in the field of computer vision. However, existing orientation estimation methods cannot handle the various body poses and appearances. In this paper, we propose an innovative RGB-D-based orientation estimation method to address these challenges. By utilizing the RGB-D information, which can be real time acquired by RGB-D sensors, our method is robust to cluttered environment, illumination change and partial occlusions. Specifically, efficient static and motion cue extraction methods are proposed based on the RGB-D superpixels to reduce the noise of depth data. Since it is hard to discriminate all the 360 (°) orientation using static cues or motion cues independently, we propose to utilize a dynamic Bayesian network system (DBNS) to effectively employ the complementary nature of both static and motion cues. In order to verify our proposed method, we build a RGB-D-based human body orientation dataset that covers a wide diversity of poses and appearances. Our intensive experimental evaluations on this dataset demonstrate the effectiveness and efficiency of the proposed method.

Optimized placement of clients in a distributed publish/subscribe system is an important technique to improve overall system efficiency. Current methods, like interest clustering or publisher placement, treat a client as, either a pure publisher, or subscriber, but not as both. Also, the cost of client movement is usually ignored. However, many applications based on publish/subscribe systems model clients as publisher and subscriber at the same time, which breaks the assumptions made by current approaches. Considering the complex dependency among clients, we propose a new community-oriented clustering approach, based on the forming of client clusters that exhibit intense communication relationships, while keeping client movement cost low. The evaluation based on a public data set shows that our method is efficient, adapts to different settings of experimental conditions, and wins over the popular interest clustering approach with respect to number of messages sent, propagation hop count and end-to-end latency.

Locality Sensitive Hashing (LSH) is proposed to construct indexes for high-dimensional approximate similarity search. Multi-Probe LSH (MPLSH) is a variation of LSH which can reduce the number of hash tables. Based on the idea of MPLSH, this paper proposes a novel probability model and a query-adaptive algorithm to generate the optimal multi-probe sequence for range queries. Our probability model takes the query range into account to generate the probe sequence which is optimal for range queries. Furthermore, our algorithm does not use a fixed number of probe steps but a query-adaptive threshold to control the search quality. We do the experiments on an open dataset to evaluate our method. The experimental results show that our method can probe fewer points than MPLSH for getting the same recall. As a result, our method can get an average acceleration of 10% compared to MPLSH.

This paper considers the problem of web video geolocation: we hope to determine where on the Earth a web video was taken. By analyzing a 6.5-million geotagged web video dataset, we observe that there exist inherent geography intimacies between a video with its relevant videos (related videos and same-author videos). This social relationship supplies a direct and effective cue to locate the video to a particular region on the earth. Based on this observation, we propose an effective web video geolocation algorithm by propagating geotags among the web video social relationship graph. For the video that have no geotagged relevant videos, we aim to collect those geotagged relevant images that are content similar with the video (share some visual or textual information with the video) as the cue to infer the location of the video. The experiments have demonstrated the effectiveness of both methods, with the geolocation accuracy much better than state-of-the-art approaches. Finally, an online web video geolocation system: Video2Locatoin (V2L) is developed to provide public access to our algorithm.

In state-of-the-art image retrieval systems, an image is represented by bag-of-features (BOF). As BOF representation discards geometric relationships among local features, exploiting geometric constraints as post-processing procedure has been shown to greatly improve retrieval precision. However, full geometric constraints are computationally expensive and weak geometric constraints have limited range of applications. To efficiently handle common transformations and deformations, we present a novel local geometric consistency constraint (LGC) method. It utilizes the local similarity characteristic of deformations, and measures the pairwise geometric similarity of matches between two sets of local features. Besides, we propose a new method to accurately calculate the transformation matrix between two matched features, with the information provided by their local neighbors. Experiments performed on famous datasets show the excellent performance of our method.

Seabird populations are considered an important and accessible indicator of the health of marine environments: variations have been linked with climate change and pollution [1]. However, manual monitoring of large populations is labour-intensive, and requires significant investment of time and effort. In this paper, we propose a novel detection system for monitoring a specific population of Common Guillemots on Skomer Island, West Wales (UK). We incorporate two types of features, Histograms of Oriented Gradients (HOG) and Local Binary Pattern (LBP), to capture the edge/local shape information and the texture information of nesting seabirds. Optimal features are selected from a large HOG-LBP feature pool by boosting techniques, to calculate a compact representation suitable for the SVM classifier. A comparative study of two kinds of detectors, i.e., whole-body detector, head-beak detector, and their fusion is presented. When the proposed method is applied to the seabird detection, consistent and promising results are achieved.

State-of-the-art image search systems mostly build on bag-of-features (BOF) representation. As BOF ignores geometric relationships among local features, geometric consistency constraints have been proposed to improve search precision. However, exploiting full geometric constraints are too computational expensive. Weak geometric constraints have strong assumptions and can only deal with uniform transformations. To handle view point changes and nonrigid deformations, in this paper we present a novel pairwise weak geometric consistency constraint (P-WGC) method. It utilizes the local similarity characteristic of deformations, and measures the pairwise geometric similarity of matches between two sets of local features. Experiments performed on four famous datasets and a dataset of one million of images show a significant improvement due to P-WGC as well as its efficiency. Further improvement of search accuracy is obtained when it is combined with full geometric verification.

Discovering common visual patterns (CVPs) between two images is a challenging problem, due to the significant photometric and geometric transformations, and the high computational cost. In this paper, we formulate CVPs discovery as a graph matching problem, depending on pairwise geometric compatibility between feature correspondences. To efficiently find all CVPs, we propose two algorithms--Preliminary Initialization Optimization (PIO) and Post Agglomerative Combining (PAC). PIO reduces the search space of CVPs discovery based on the internal homogeneity of CVPs, while PAC refines the discovery result in an agglomerative way. Experiments on object recognition and near-duplicate image re-trieval validate the effectiveness and efficiency of our method.

Nearest neighbor search in Euclidean space is a fundamental problem in multimedia retrieval. The difficulty of exact nearest neighbor search has led to approximate solutions that sacrifice precision for efficiency. Among such solutions, approaches that embed data into binary codes in Hamming space have gained significant success for their efficiency and practical memory requirements. However, binary code searching only finds a big and coarse set of similar neighbors in Hamming space, and hence expensive Euclidean distance based ranking of the coarse set is needed to find nearest neighbors. Therefore, to improve nearest neighbor search efficiency, we proposed a novel binary code method called Integrated Binary Code (IBC) to get a compact set of similar neighbors. Experiments on public datasets show that our method is more efficient and effective than state-of-the-art in approximate nearest neighbor search.

Surveillance video privacy protection has drawn significant attention recently. In this paper, we describe a privacy protected video surveillance system which utilizes the emerging compressive sensing (CS) theory. Privacy regions are scrambled through block based CS sampling on quantized coefficients during compression. Security is ensured by key controlled chaotic sequence which is used to construct CS measurement matrix. To prevent drift error caused by scrambling, a coding restricted scheme is exploited. Experimental results show that the proposed system effectively protects privacy with the scene intelligible. Compared with the existing ones, this system has high security and dramatic coding efficiency improvement.

In recent years, Locality Sensitive Hashing (LSH) (and its variant Euclidean LSH) has become a popular index structure for large-scale and high-dimensional similarity search problem. In this paper, we analyze a phenomenon we called "Non-Uniform" that degrades the query performance of LSH and propose a pivot-based algorithm to improve the query performance. We also provide a method to get optimal pivot for even larger improvement. Experiments show that our algorithm significantly improves the query performance of LSH.

Despite the massive growth of web-shared videos in Internet, efficient organization and monitoring of videos remains a practical challenge. While nowadays broadcasting channels are keen to monitor online events, identifying topics of interest from huge volume of user uploaded videos and giving recommendation to emerging topics are by no means easy. Specifically, such process involves discovering of new topic, visualization of the topic content, and incremental monitoring of topic evolution. This paper studies the problem from three aspects. First, given a large set of videos collected over months, an efficient algorithm based on salient trajectory extraction on a topic evolution link graph is proposed for topic discovery. Second, topic trajectory is visualized as a temporal graph in 2-D space, with one dimension as time and another as degree of hotness, for depicting the birth, growth, and decay of a topic. Finally, giving the previously discovered topics, an incremental monitoring algorithm is proposed to track newly uploaded videos, while discovering new topics and giving recommendation to potentially hot topics. We demonstrate the application on three months' videos crawled from YouTube during December 2008 to February 2009. Both objective and user studies are conducted to verify the performance.

Spatial matching for object retrieval is often time-consuming and susceptible to viewpoint changes. To address this problem, we propose a novel spatial matching method that is robust to viewpoint changes and implement it on modern graphics processing unit (GPU) in parallel for real-time applications. Unlike previous spatial matching methods used in object retrieval, in which the affine transformation estimation is based on the gravity vector assumption, our method abandons this strong assumption by matching the affine covariant neighbors (ACNs) of corresponding local regions and estimating affine transformation from each single pair of corresponding local regions. Taking into account real-time applications, we implement the method on modern GPU in parallel to speed up the process. Computations are distributed evenly to threads with load balancing, and device memory accesses are optimized with bitmap-based parallel scan. Experimental results demonstrate that our method is more robust and more efficient than previous methods especially when the viewpoints are changed, and the parallel implementation on GPU obtains ten times speedup.

State-of-the-art near-duplicate image search systems mostly build on the bag-of-local features (BOF) representation. While favorable for simplicity and scalability, these systems have three shortcomings: 1) high time complexity of the local feature detection; 2) discriminability reduction of local descriptors due to BOF quantization; and 3) neglect of the geometric relationships among local features after BOF representation. To overcome these shortcomings, we propose a novel framework by using graphics processing units (GPU). The main contributions of our method are: 1) a new fast local feature detector coined Harris-Hessian (H-H) is designed according to the characteristics of GPU to accelerate the local feature detection; 2) the spatial information around each local feature is incorporated to improve its discriminability, supplying semi-local spatial coherent verification (LSC); and 3) a new pairwise weak geometric consistency constraint (P-WGC) algorithm is proposed to refine the search result. Additionally, part of the system is implemented on GPU to improve efficiency. Experiments conducted on reference datasets and a dataset of one million images demonstrate the effectiveness and efficiency of H-H, LSC, and P-WGC.

Most existing query expansion approaches for ad-hoc retrieval adopt overly simplistic textual representations that treat documents as bags of words and ignore inherent document structure. These simple representations often lead to incorrect independence assumptions in the proposed approaches and result in limited retrieval effectiveness. In this paper, we propose a novel query expansion technique that models the various types of dependencies that exist between original query terms and expansion terms within a robust, unified framework. The proposed model is called Hierarchical Markov random fields (HMRFs), based on Latent Concept Expansion (LCE). By exploiting implicit (or explicit) hierarchical structure within documents, HMRFs can incorporate hierarchical interactions which are important for modeling term dependencies in an efficient manner. Our rigorous experimental evaluation carried out using several TREC data sets shows that our proposed query expansion technique consistently and significantly outperforms the current state-of-the-art query expansion approaches, including relevance-based language models and LCE.

Spatial matching for visual words based object retrieval often involves generating affine transformation hypotheses and then choosing the best hypothesis to measure the spatial consistency. In existing methods, generating an affine transformation hypothesis either requires three correspondences or assumes images are taken in restricted range of viewpoints in using a single correspondence. In this paper, we propose a novel spatial matching method, in which the transformation hypothesis can be estimated from only a single correspondence without the assumption of the viewpoints from which the images are taken. Firstly, affine covariant neighborhoods(ACNs) of features are used to eliminate possible false matches. Secondly, we decompose the affine transformation into three sub-transforms and conquer each sub-transform by exploiting the shape information and the ACNs of a single pair of corresponding features. Experiment results demonstrate that this method improves the average retrieval precision evidently with less computation in comparison with the previous methods.

To take full advantage of the powerful computing capability of graphics processing units (GPU) to speed up local feature detection, we present a novel GPU-based scale invariant interest point detector, coined Harris-Hessian(H-H). H-H detects Harris points in low scale and refines their location and scale in higher scale-space with the determinant of Hessian matrix. Compared to the existing methods, H-H significantly reduces the pixel-level computation complexity and has better parallelism. The experiment results show that with the assistance of GPU, H-H achieves up to a 10-20x speedup than CPU-based method. It only takes 6.3ms to detect a 640 × 480 image with high detection accuracy, meeting the need of real-time detection.

Scrambling is widely used to protect privacy in surveillance video. However, as a critical issue in privacy protected video scrambling, drift error has not been adequately studied. In this paper, we focus on drift error prevention for different elements scrambling in privacy protected H.264/AVC video, which is the prevailing coding standard. A restricted coding scheme is proposed to prevent drift error in Transform Coefficient (TC), Intra Prediction Mode (IPM) and Motion Vector (MV) scrambling, respectively. Experiments show that the proposed scheme effectively prevents drift error with coding efficiency dramatically improved.

Spatial matching for object retrieval is often time-consuming and susceptible to viewpoint changes. To address this problem, we propose a novel spatial matching method and implement it on modern GPU in parallel. Unlike previous spatial matching methods, in which the affine transformation estimation is based on the gravity vector assumption, our method abandons this strong assumption by matching the ACNs (affine covariant neighbors) of corresponding local regions and estimating affine transformation from a single pair of corresponding local regions. To speed up the process, we implement the method on modern GPU in parallel. Computations are distributed evenly to threads with load balancing, and the memory accesses are optimized and bitmap based parallel scan is exploited. Experimental results demonstrate that our method is more robust and more efficient than previous methods especially when the viewpoints are changed, and the parallel implementation on GPU obtains ten times speedup.

To address the problem of service composition in pervasive computing, and to provide the IGRS ubiquitous devices the abilities of inter-operation and cooperation, we design the UDIO, a middleware of dynamic service composition based on IGRS protocol. In this paper, the system architecture of UIDO, as well as a matrix-based path-finding algorithm for service composition, are introduced which prove to be available and efficient in our IGRS smart home.

The major scientific problem for content-based video retrieval is the semantic gap. Generally speaking, there are two appropriate ways to bridge the semantic gap: the first one is from human perspective (top-down) and the other one is from computer perspective (bottom-up). The top-down method defines a concept lexicon from human perspective, trains the detector for each concept based on supervised learning, and then indexes the corpus with concept detectors. Since each concept has an explicit semantic meaning, we call this concept as an explicit concept. The bottom-up approach directly discovers the underlying latent topics from video corpus by machine perspective using an unsupervised learning. The video corpus is indexed subsequently by these latent topics. As opposite to explicit concepts, we name latent topics as implicit concepts. Given the explicit concept set is pre-defined and independent of the corpus, it is impossible to completely describe corpus and users' queries. On the other hand, the implicit concepts are dynamic and dependent on the corpus, which is able to fully describe corpus and users' queries. Therefore, combining explicit and implicit concepts could be a promising way to bridge the semantic gap effectively. In this paper, a Bipartite Graph Propagation Model (BGPM) is applied to automatically balance influences from explicit and implicit concepts. Concept nodes with strong connections to queries are reinforced no matter explicit or implicit. Demonstrated by the experiments on TREVID 2008 video dataset, BGPM successfully fuses explicit and implicit concepts to achieve a significant improvement on 48 search tasks.

This paper presents a novel approach for web video categorization by leveraging Wikipedia categories (WikiCs) and open resources describing the same content as the video, i.e., content-duplicated open resources (CDORs). Note that current approaches only collect CDORs within one or a few media forms and ignore CDORs of other forms. We explore all these resources by utilizing WikiCs and commercial search engines. Given a web video, its discriminative Wikipedia concepts are first identified and classified. Then a textual query is constructed and from which CDORs are collected. Based on these CDORs, we propose to categorize web videos in the space spanned by WikiCs rather than that spanned by raw tags. Experimental results demonstrate the effectiveness of both the proposed CDOR collection method and the WikiC voting categorization algorithm. In addition, the categorization model built based on both WikiCs and CDORs achieves better performance compared with the models built based on only one of them as well as state-of-the-art approach.

Human annotations (titles and tags) of web videos facilitate most web video applications. However, the raw tags are noisy, sparse and structureless, which limit the effectiveness of tags. In this paper, we propose a tag transformer schema to solve these problems. We first eliminate those imprecise and meaningless tags with Wikipedia, and then transform the remaining tags to the Wikipedia category set to gather a precise, complete and structural description of the tags. Our experimental results on web video categorization demonstrate the superiority of the transformed space. We also apply tag transformer into the first study of using Wikipedia category system to structurally recommend the related videos. The online user study of the demo system suggests that our method could bring fantastic experience to the web users.

Video encryption plays an important role in data security guarantee, which is increasingly important with the development of multimedia technology. A great deal of effort has been made in recent years to develop video encryption methods. However, few studies focus on visual security evaluation, which has significant impact in measuring the effectiveness of these methods. In this paper, a new metric for video encryption is proposed, which evaluates visual security based on color and edge features of original and cipher-videos. The metric is easy to be incorporated into video encryption system for visual security based encryption decision. In addition, subjective tests for visual security assessment have been fully carried out. Experiments show that the proposed metric had better correlation with subjective results than others.

Locality Sensitive Hashing (LSH) has been proposed as a scalable and high-dimensional index for approximate similarity search. Euclidean LSH is a variation of LSH and has been successfully used in many multimedia applications. However, hash functions of the basic Euclidean LSH project data points over randomly selected directions, which reduces accuracy when data are non-uniformly distributed. So more hash tables are needed to guarantee the accuracy, and thus more memory is consumed. Since heavy memory cost is a significant drawback of Euclidean LSH, we propose Data-Oriented LSH to reduce memory consumption when data are non-uniformly distributed. Most of existing methods are query-directed, such as multi-probe and query expansion methods. We focused on the hash table construction, and thus the query-directed methods can be applied to our index to improve further. The experiment shows that to achieve the same accuracy, our method uses less time and less memory compared with original Euclidean LSH.

Tag recommendation is a common way to enrich the textual annotation of multimedia contents. However, state-of-the-art recommendation methods are built upon the pair-wised tag relevance, which hardly capture the context of the web video, i.e., when who are doing what at where. In this paper we propose the context-oriented tag recommendation (CtextR) approach, which expands tags for web videos under the context-consistent constraint. Given a web video, CtextR first collects the multi-form WWW resources describing the same event with the video, which produce an informative and consistent context; and then, the tag recommendation is conducted based on the obtained context. Experiments on an 80,031 web video collection show CtextR recommends various relevant tags to web videos. Moreover, the enriched tags improve the performance of web video categorization.

The problem of recognizing actions in realistic videos is challenging yet absorbing owing to its great potentials in many practical applications. Most previous research is limited due to the use of simplified action databases under controlled environments or focus on excessively localized features without sufficiently encapsulating the spatio-temporal context. In this paper, we propose to model the spatio-temporal context information in a hierarchical way, where three levels of context are exploited in ascending order of abstraction: 1) point-level context (SIFT average descriptor), 2) intra-trajectory context (trajectory transition descriptor), and 3) inter-trajectory context (trajectory proximity descriptor). To obtain efficient and compact representations for the latter two levels, we encode the spatiotemporal context information into the transition matrix of a Markov process, and then extract its stationary distribution as the final context descriptor. Building on the multichannel nonlinear SVMs, we validate this proposed hierarchical framework on the realistic action (HOHA) and event (LSCOM) recognition databases, and achieve 27% and 66% relative performance improvements over the state-of-the-art results, respectively. We further propose to employ the Multiple Kernel Learning (MKL) technique to prune the kernels towards speedup in algorithm evaluation.

The paper originally presents a confusion network based framework for video OCR post-processing. The framework consists of four parts: selection of reference and hypotheses, construction of confusion network, decoding for final output, and a novel metric of quantitatively evaluating Video OCR post-processing approaches. By integrating both visual and textual information, we construct the character transition network to reduce the error rate for OCR outputs. The large-scale experimental results demonstrate that this approach can significantly improve the accuracy of Video OCR results with only little incremental time. Moreover, with comparison and the detailed analysis, we conclude that ldquoVoting+2-gramrdquo is the most applicable method for real application.

Pseudo Relevance Feedback (PRF) has shown effective performance in information retrieval, but it has seldom been applied in the area of high level feature detection (HLF). In this paper, we explicitly propose to introduce PRF into HLF. Our contributions mainly lie in two-fold: (1) proposing three novel PRF approaches to extract pseudo positive samples, i.e., Nearest-Neighbor (NN) based PRF, Score-Evaluation (SE) based PRF and Multi-Classifier Decision (MCD) based PRF; (2) utilizing incremental learning to reduce the re-training time. We evaluate our approaches on the benchmark of TRECVID2008. Reported results have shown that MCD based approach outperforms the other two and obtain an excellent gain in average precision with respect to the baseline without PRF.

A successful interactive image retrieval system is expected to quickly return as many relevant results as possible while costing less users' effort. Considering these system demands, firstly we propose a novel semi-supervised learning algorithm called Locally Non-negative Linear Structure Learning (LNLS), which is based on the assumption that the labels of each data should be sufficiently smooth with respect to the locally non-negative linear structure of dataset. It has two main merits: first, it is robust to the small sample learning problem since it learns structure from both labeled and unlabeled data; second, by emphasizing the non-negativity of locally linear structure, this algorithm preserves the non-negative inherent characteristic of image data and can truly reveal the intrinsic structure of the images corpus, especially the asymmetric relationship between images. Meanwhile, we explore an online updating algorithm for LNLS to tackle the large computation cost. Thus the model can be generalized to the new queries or the newly-labeled samples without retraining. Furthermore, an active learning method for LNLS is proposed to make the most of users' effort to improve the learner. The encouraging experimental results demonstrate the effectiveness and efficiency of our proposed methods.

Facing the vast amount of novel production in the movie industry, people are in favor of choosing their favorite candidates quickly and previewing movie contents conveniently so as to decide whether they appeal to their personal taste. To meet this growing need, researchers are paying more attention on Personalization and Recommendation, the new trends of multimedia information retrieval, by integrating content and contextual information. In this paper, we propose a hierarchical framework for personalized movie recommendation. First, movie weekly ranking information is utilized for movie association and recommendation. Then, an integrated graph with both movie content and user preference is constructed to generate dynamic movie synopsis for personalized navigation. The superiorities of the proposed method have two aspects: 1) The prior knowledge independent recommendation scheme is implemented to replace the traditional ranking method for novel information access; 2) Personalized movie synopsis is interactively produced to replace the current movie trailer for preview. The promising results of subjective evaluation indicate that the proposed framework can discover the latent relationship between movies as well as movie highlights and therefore provide personalized movie recommendation to effectively lead movie access in an individualized manner.

With the advent of video sharing websites, the amount of videos on the internet grows rapidly. Web video categorization is an efficient methodology to organize the huge amount of data. In this paper, we propose an effective web video categorization algorithm for the large scale dataset. It includes two factors: 1) For the great diversity of web videos, we develop an effective semantic feature space called Concept Collection for Web Video Categorization (CCWV-CD) to represent web videos, which consists of concepts with small semantic gap and high distinguishing ability. Meanwhile, the online Wikipedia API is employed to diffuse the concept correlations in this space. 2) We propose an incremental support vector machine with fixed number of support vectors (n-ISVM) to fit the large scale incremental learning problem in web video categorization. Extensive experiments are conducted on the dataset of 80024 most representative videos on YouTube demonstrate that the semantic space with Wikipedia prorogation is more representative for web videos, and n-ISVM outperforms other algorithms in efficiency when performs the incremental learning.

We present PornProbe, a pornography detection system that detects pornographic contents in videos. To build such a detection system, we leverage a large scale training data set with 65,827 positive training image samples out of a total of 420,615 training samples, and a novel detection scheme based on hierarchical LDA-SVM. The system combines the unsupervised clustering in Latent Dirichlet Allocation (LDA) and supervised learning in Support Vector Machine, so as to achieve both high precision and recall while ensuring efficiency in both training and testing. This demonstration shows how the system detects the pornographic scenes in restricted artistic (RA) movies.

This paper presents a novel motion localization approach for recognizing actions and events in real videos. Examples include StandUp and Kiss in Hollywood movies. The challenge can be attributed to the large visual and motion variations imposed by realistic action poses. Previous works mainly focus on learning from descriptors of cuboids around space time interest points (STIP) to characterize actions. The size, shape and space-time position of cuboids are fixed without considering the underlying motion dynamics. This often results in large set of fragmentized cuboids which fail to capture long-term dynamic properties of realistic actions. This paper proposes the detection of spatio-temporal motion volumes (namely Volume of Interest, VOI) of scale and position adaptive to localize actions. First, motions are described as bags of point trajectories by tracking keypoints along the time dimension. VOIs are then adaptively extracted by clustering trajectory on the motion mainfold. The resulting VOIs, of varying scales and centering at arbitrary positions depending on motion dynamics, are eventually described by SIFT and 3D gradient features for action recognition. Comparing with fixed-size cuboids, VOI allows comprehensive modeling of long-term motion and shows better capability in capturing contextual information associated with motion dynamics. Experiments on a realistic Hollywood movie dataset show that the proposed approach can achieve 20\% relative improvement compared to the state-of-the-art STIP based algorithm.

In a ubiquitous computing environment, contexts are initially got and stored on those nodes scattered over the environment. However, the traditional reasoning about contexts applied a centralized approach which aggravated the load of reasoning server and cost for communication of context. As an important approach for context reasoning, the rule-based reasoning can be easily decomposed of independent propositions. Therefore, the rule-based context reasoning can be decomposed and distributed over those nodes of the ubiquitous environment to decrease the computing load of reasoning server. In this paper, we propose a distributed fuzzy reasoning Petri net model (dFRPN) towards the decomposition of distributed fuzzy reasoning. dFRPN model is able to formalize the distribution of fuzzy reasoning over every node of the whole environment. dFRPN model is characterized as a hierarchical structure to make a description of reasoning on nodes more clear. Considering the limited capabilities of some mobile nodes such as PDA and smart phone, we add the special migrating transition to define the load detection and corresponding actions. At the end of the paper, the feasibility of dFRPN model is validated through a case of context-awareness based personalized recommendation system.

Generating similar queries for a query, named query suggestion, is an important technology for helping search engine users. Since query data is very diverse and sparse, it is still challenging to measure the similarity of each query pair. We propose a novel algorithm called QueryTrans, which can efficiently compute pairwise similarity scores between all queries with respect to the global structure of a query trace graph mined from search engine logs. Compared with previous query suggestion approaches, QueryTrans is robust for different queries and stable for different parameter settings. We also present the performance of QueryTrans on large scale query logs. Experiments on about 100,000 queries show: QueryTrans can efficiently computes almost 10 billion pairwise similarity scores within 15 minutes on a single computer; and its results are significantly better than all 4 recent approaches on query suggestion.

In this paper, we specifically propose the Weber-Fechner Law-based human attention model for semantic scene analysis in movies. Different from traditional video processing techniques, we pay more attention on bringing in the related subjects, such as psychology, physiology and cognitive informatics, for content-based video analysis. The innovation of our work has two aspects. Firstly, we originally construct the human attention model with temporal information instructed by the Weber-Fechner Law. Secondly, motivated by cognitive informatics, we formulate the computational methodology of features in visual, audio and textual modalities in the uniform metric of information quantity. With human attention analysis and semantic scene detection, we build a system for hierarchical browse and edit with semantics annotation. Large-scale experiments demonstrate the effectiveness and generality of the proposed human attention model for movie analysis.

Given countless web videos available online, one problem is how to help users find videos to their taste in an efficient way. In this paper, to facilitate user’s browsing we propose relevant and exploratory recommendation algorithms utilizing multimodal similarity and contextual network to organize web videos of various topics. Comparison experiments demonstrate proposed approach generates more accurate video relevancy. And our method is more flexible in discovering user latent interests in long tail videos.

Large scale video copy detection task requires compact feature insensitive to various copy changes. Based on local feature trajectory behavior we discover invariant visual patterns for generating robust feature. Bag of Trajectory (BoT) technical is adopted for fast pattern matching. Our algorithm with lower cost is more robust compared to the state-of-art schemes.

In this paper, we present an innovative model of tempo and its application in action scene detection for movie analysis. For the first time, we clearly propose that tempo indicates the rhythm of both movie scenarios and human perception. By thoroughly analyzing both aspects, we classify the factors of tempo into two sorts. The first is based on the film grammar and we use the low level features of shot length and camera motion to describe filmmaking by directors. The second is based on the human perception and we originally propose the information measure for perception depending on the cognitive informatics, a newly emerging and significative subject. With the information in both visual and auditory modalities, the low level features of motion intensity, motion complexity, audio energy and audio pace are integrated for the formulation of information to describe the viewers' emotional changes to continuously developing storyline. With both aspects, tempo is defined and tempo flow plot is derived as the clue of storyline. On the basis of video structuralization and movie tempo analysis, we build a system for hierarchical browse and edit with action scene annotation. The large-scale experiments demonstrate the effectiveness and generality of tempo for action movie analysis.In this paper, we present an innovative model of tempo and its application in action scene detection for movie analysis. For the first time, we clearly propose that tempo indicates the rhythm of both movie scenarios and human perception. By thoroughly analyzing both aspects, we classify the factors of tempo into two sorts. The first is based on the film grammar and we use the low level features of Shot Length and Camera Motion to describe filmmaking by directors. The second is based on the human perception and we originally propose the information measure for perception depending on the cognitive informatics, a newly emerging and significative subject. With the information in both visual and auditory modalities, the low level features of Motion Intensity, Motion Complexity, Audio Energy and Audio Pace are integrated for the formulation of information to describe the viewers' emotional changes to continuously developing storyline. With both aspects, tempo is defined and tempo flow plot is derived as the clue of storyline. On the basis of video structuralization and movie tempo analysis, we build a system for hierarchical browse and edit with action scene annotation. The large-scale experiments demonstrate the effectiveness and generality of tempo for action movie analysis.

Today with the rapid increasing popularity of web video sharing, digital copyright protection encounters many troubles. Video copy detection schemes are emerging to cope with the digital video piracy and illegal distribution problems. But the large amount of video data and diversity of copy attacks pose difficulties on copy detection. This paper presents a hierarchical scheme to detect video copies, especially the temporal attacked and re-encoded ones. Our algorithm which is based on the ordinal signature of intra frames and effective R*-tree indexing structure archives real time performance. Comparison experiments are conducted on the benchmarked database of CIVR 2007 copy detection showcase and demonstrate the promising results of the proposed approach.

In this paper, we highlight the use of multimedia technology in generating intrinsic summaries of tourism related information. The system utilizes an automated process to gather, filter and classify information on various tourist spots on the Web. The end result present to the user is a personalized multimedia summary generated with respect to users queries filled with text, image, video and real-time news made retrievable for mobile devices. Preliminary experiments demonstrate the superiority of our presentation scheme to traditional methods.

Earlier research in news video has been focusing mainly on improving retrieval accuracies given the limited amount of extractable video semantics. In this paper, we propose an enhancement to news video searching by leveraging extractable video semantics coupled with relevant external information resources to support event-based analysis; leading to discovery of topic hierarchy for browsing key events and supporting question answering (QA). We introduce topic browsing based on news structures obtained through hierarchical clustering and threading, with emphasis on interesting events determined by measuring the amount of "web activities" on these events on Blog sites. For QA, we employ extensive query analysis to obtain various query features in addition to the topic hierarchical structures to answer both context-oriented and visual-oriented questions. Our main contributions includes: (a) combining multimodal event information extracted from news video, web news articles and news blogs to support event analysis, (b) introducing topic evolution browsing based on users' interest and (c) extending QA on top of topic hierarchy to handle various types of specialized video queries. Experiments performed on 70 hours of multilingual news from TRECVID 2005 dataset shows that the proposed approach is effective and appealing to users.

Ubiquitous computing environment has become more popular as a highly open, heterogeneous and service-rich domain, which makes it essential and important for end-users to build a reasonable service provision model. The critical issue for service provision is how to sense context information and provide personalized services according to current context and user's requirement. This paper presents a service provision model based on context-awareness for dynamic ubiquitous computing environment called C-SPM. C-SPM provides context-aware service that is aware of context information, and uses intelligent provision mechanism to provide personalized services for users according to their context and requirements.

This paper points out some defects in the techniques used in H.264 rate control and presents several new algorithms to improve them. The improved algorithm has the following main features: 1) the bits allocated to each P-frame is proportional to the local motion in it, i.e, more bits are allocated to a frame if the local motion in it is stronger; 2) the quantization parameter for I-frame is choosed based on a new bits allocation scheme for I-frame; 3)the quantization parameter calculation is based on a simple encoding complexity prediction scheme, which is more robust and of less complexity than the quadratic model used by H.264 in low bit rate video coding. Experimental results and analysis show that the improved rate-control scheme has significantly increased the average peak signal-to-noise ratio up to 1.53 dB, reduced the variation of buffer level, improved the perpetual quality of the reconstructed video and reduced the computation complexity.

This paper points out some defects in the techniques used in H.264 rate control and presents two new techniques to improve them. The improved scheme has the following main features: 1) the bits allocated to each P-frame is proportional to the local motion in it, i.e, more bits are allocated to a frame if the local motion in it is stronger; 2) the quantization paramter (QP) calculation is based on a simple encoding complexity prediction scheme, which is more robust and simple than the quadratic model used by H.264 in low bit rate video coding. Simulation results show that compared to rate control scheme in H.264, the improved scheme has significantly improved R-D performance (up to 1.29dB).

This paper presents an efficient anticollision algorithm for the RFID tags communication conflict. The novelty of our algorithm is that we map a set of n tags into a corresponding IDbinary tree, and see the process of collision arbitration as a process of building the ID-binary tree. In order to efficiently construct an ID-binary tree, the reader uses a stack to store the threads of the construction information, and while the tag uses a counter to keep track of the stack position where the tag is on. Theoretic results in both the number of queries sent by the reader and the tags communication complexity are derived to demonstrate the efficiency of our algorithm.

H.264/AVC is the latest standard for video coding drafted jointly by the ISO/IEC Moving Picture Experts Group and the ITU-T Video Coding Experts Group. H.264/AVC provides up to 50% gain in compression efficiency over a wide range of bit rates and video resolutions compared to previous standards. On the other hand, the decoder complexity is about four times that of MPEG-2 and two times that of MPEG-4 visual simple profile. In VLSI implementations of the H.264/AVC decoder, off-chip memory access is the main time and power consuming operation, and the motion compensation module is the main memory access bottleneck. This paper proposes four optimization strategies to reduce memory access data cycles and improve memory data bus utilization. Experimental results show that about 60% data cycle reduction can be achieved and more than 20% memory data bus utilization can be improved by these strategies over typical test sequences.

Quantization step (QS) selection for I-frame has an obvious impact on rate-distortion performance in rate-constrained video coding. An efficient QS selection scheme for I-frame is presented which is based on a balanced bits allocation scheme between I and P frames and a new experimental rate-quantization step (R-Q) model. In the balanced bits allocation scheme, firstly a model on the ratio of bits allocation between I and P frame is derived under the constraint of certain target bit rate; then based on the model, a feasible bits allocation algorithm is proposed by experimental analysis among different target bit rates. The analysis and final simulation results show that compared to the existing methods, the proposed algorithm improves PSNR performance significantly (up to 1.2 dB).

We propose a novel method for image authentication which can distinguish incidental manipulations from malicious ones. The authentication fingerprint is based on the Hotelling's T-square statistic (HTS) via Principal Component Analysis (PCA) of block DCT coefficients. HTS values of all blocks construct an unique and stable "block-edge image", i.e., Structural and Statistical Fingerprint (SSF). The characteristic of the SSF is that it is short, and can tolerate content-preserving modifications while keeping sensitive to content-changing modifications, and can locate tampered blocks easily. Furthermore, we use Fisher criterion to obtain optimal threshold for distinguishing manipulations. The security of the SSF is also achieved by encryption of the DCT coefficients with chaotic sequences. Experiments show that the proposed method is effective for authentication.

Automatic detection of semantic events in sport videos is a challenging task. In this paper, we propose a multimodal multilayer statistical inference framework for semantic sports video analysis using Dynamic Bayesian Networks (DBNs). Based on this framework, three instances including factorial hierarchical hidden Markov model (FHHMM), coupled hierarchical hidden Markov model (CHHMM), and product hierarchical hidden Markov model (PHHMM), are constructed and compared. Play-break detection in soccer videos is used as a testbed with hierarchical hidden Markov model (HHMM) as a baseline. Experimental results indicate the superior capability of the PHHMM, because it not only effectively models dynamic interactions between different modalities, but also sufficiently utilizes context constraints in multilayer structures.

This paper presents a ubiquitous computing technology called Virtual Personal Computing Environment (VPCE), which enables users to access their personalized computing environments anywhere using any available terminals. The key feature of the VPCE is that it can be constructed dynamically by reassembling application services and resources distributed over networks, which is different from the traditional desktop personal computing environment that is tied to an individual computer to execute. In VPCE, users work with a heterogeneous collection of application services, and data access by application as well as the application accessed by users are distributed over networks, while the applications and data distribution is transparent to the users.

In this paper, we propose a novel approach to event detection in soccer videos using dynamic Bayesian networks (DBNs). Based on such high level semantics, say, events, more meaningful soccer highlights are extracted. As a powerful statistical tool for time series signal processing, DBNs provide us with a feasible method to model sports events by combining contextual information and prior knowledge. In particular, we first develop a DBN model to interpret high-level events composed of low-level primitives in a soccer video. Then, we select a set of robust statistical features as observation input. Finally, the DBN model is gleaned to figure out the most likely series of events. The effectiveness of the proposed method has been demonstrated by our experiments.

An adaptive weighted prediction algorithm is proposed to improve the coding efficiency for video scenes that contain global brightness variations caused by fade in/out or local brightness variations caused by camera flashes. A two brightness-variation parameters model is used, which represent multiplier and offset components of the brightness variation. Brightness variations at frame and macroblock level, respectively, are detected; if there are brightness variations, we use weighted prediction to compensate these brightness variations; otherwise the conventional prediction method is used. Simulation results show that the proposed algorithm can improve the coding efficiency sufficiently when the coded scenes contain brightness variations. This technology is adopted by AVS.

In order to reduce the bit rate of video signals, motion compensation prediction is applied in modern video coding technology. This is a form of temporal redundancy reduction in which the current coding frame is predicted by a motion compensated prediction from some other already decoded frames according to a motion vector. As real motion has arbitrary precision, many video coding standards allow motion vectors to have "sub-pixel" resolution. The 1/4-pel displacement vector resolution is a part of AVS (audio video system). In order to estimate and compensate 1/4-pel displacements, the image signal on sub-pel position has to be generated by interpolation. This paper introduces a new 1/4-pel interpolation method, named as two steps four taps interpolation (TSFT). Compared with the 1/4-pel interpolation method of H.264, 11% spatial complexity is reduced, computation complexity is about the same, and a little more average SNR is gained.