The proliferation of collaborative training and multi-person sports has underscored the necessity for concurrent whole-field action sensing. However, Electromyography (EMG) recognition, which plays a pivotal role in Wearable Human Activity Recognition (WHAR) for analyzing muscle activity and decoding action intent, still faces challenges in achieving a balance between performance, cost, and efficiency in multi-person scenarios. Unlike current channel-expansion solutions, we propose a wireless wearable Single-Dimensional Sparse EMG (2SEMG) Sensor for efficient personal sampling. These action-unaffected sensors leverage the proposed lightweight One-Dimensional Motion Network (OMONet) to facilitate concurrent action sensing. Experiments demonstrate that OMONet achieves leading performance and efficiency in action signal recognition, and two real-world badminton matches further confirm the performance, robustness, and real-time efficiency of the whole-field action sensing network constructed via 2SEMG Sensors and OMONet.

Most modern Convolutional Neural Networks (CNNs) employ a multi-branch structure with various-sized convolutions to capture long- and short-range dependencies. However, these CNNs use large kernel convolutions (e.g., astonishingly 101 kernels) and specialized techniques (e.g., reparameterization and sparsity), increasing complexity in both training and inference stages. This paper focuses on designing an efficient CNN based on pure 3×3 convolutions without introducing complex operations and techniques. Specifically, we propose a Spatial Pyramid (SP) block, which consists of the Multi-branch Residual (MbR) module and the Gated-branch Residual (GbR) module. The MbR introduces multiscale pooling as the key component, thus capturing long-range visual cues through large down-sampling rates and shorter-range dependencies through low down-sampling rates while maintaining low computational complexity. Besides, the GbR uses one 3×3 convolution to refine dependencies along spatial and channel dimensions. Based on the SP block, we construct the Spatial Pyramid CNN (SPCNN), a model composed exclusively of Point-Wise Convolution and 3×3 Depth-Wise Convolution. Under comparable computational complexity, SPCNN significantly outperforms the state-of-the-art CNN PeLK (83.6% vs 82.6%) with only 3 × 3 kernels (compared to 101 × 101 kernels in PeLK). Besides, our SPCNN demonstrates comparability with state-of-the-art backbones in lightweight models, object detection, instance segmentation, and semantic segmentation. Moreover, evaluations of four image retrieval benchmarks also demonstrate the effectiveness. All codes are released at https://github.com/xiaolai-sqlai/SPCNN.

With the development of deep learning in recent years, the performance of object detection under conventional cameras has been significantly improved. Nevertheless, due to the distortion caused by the fisheye cameras, detecting objects in this scenario remains a significant challenge. The dominant approaches focus on modifying the shape of the bounding box to better align the boundaries of the distorted object. However, these methods neglect the learning of spatial distortion information, which prevents them from satisfactory results. In this paper, we propose a novel fisheye camera detection network to learn distortion features better, dubbed SDANet. SDANet is composed of a series of SDABlocks, which are designed to learn spatial distortion features. Each SDABlock consists of multiple convolution kernels of different sizes, and it can generate the most suitable kernel based on the current input's distortion characteristics. Moreover, to address the limitations of the scarcity and uneven spatial distribution of fisheye image datasets on performance improvement, we propose a dedicated data augmentation strategy called Prominent Fisheye Distortion Augmentation (PFDAug). PFDAug can further introduce distortions to fisheye images, effectively alleviating these problems. Experimental results on the CEPDOF, MW-R, HABBOF, LOAF, and FishEye8k fisheye image datasets demonstrate that our method achieves state-ofthe-art performance.

Human pose estimation (HPE) is a fundamental problem in computer vision, aiming to locate anatomical keypoints of the human body in a given picture. Benefiting from recent progress in deep learning, dominant HPE methods can achieve more advanced performance. Unfortunately, these methods rely heavily on large-scale, high-quality datasets captured expensively, resulting in limited learning capabilities in data-constrained low-light situations. Existing methods enhance the model's ability for low-light scenarios by performing intermediate feature alignment between low-light image and its well-lit counterpart. However, these methods fall short in fully exploiting explicit semantic feature exploitation that is independent of lighting conditions, resulting in sub-optimal performance. In this paper, we propose a Progressive Decomposition-Modulation network (PDMNet) for human pose estimation in extremely low-light condition. In particular, PDMNet mainly consists of 1) a semantic-specific decomposition module (SDM) for decomposing reflectance component with rich semantic information, and 2) a semantic-specific modulation mechanism (SMM) that enables the reflectance component to modulate the representation learning of human body parts in a tailored manner. Two closely-related components cooperate with each other to achieve more effective content-specific feature learning in low-light conditions. We further equip them progressively into different scales to enhance the feature learning. Experimental results demonstrate the superiority of PDMNet over state-of-the-art models on publicly available datasets. Our code will be released soon.

Vision Transformer (ViT) on object re-identification (ReID) has attracted significant attention recently. However, ViT-based ReID substantially increases computational complexity, imposing significant burdens during training and inference. This paper presents an efficient and effective ViT-based backbone for ReID tasks, called the Locally Enhanced Vision Transformer (LEViT). ViT models typically emphasize global relationship modeling, yet ReID tasks are more sensitive to local information. To address this gap, we propose a Locally Enhanced (LE) block to enhance local information by performing self-attention within local split windows. Since part-based models dominate ReID, calculating self-attention across all patches is computationally inefficient. We also replace the traditional Query-Key-Value projector with the Group Convolution (G-Conv) projector, enabling the model to capture local details more efficiently. Furthermore, G-Conv is integrated into the channel MLP to strengthen local feature sensitivity. Using these components, we develop two LEViT variants: LEViT-S and LEViT-L. To our knowledge, LEViT is the first highly adaptable ViT backbone for ReID tasks. Experimental evaluations demonstrate the effectiveness in five ReID datasets: Market1501, DukeMTMC, MSMT17, VeRi-776, and VehicleID. Notably, LEViT-S outperforms TransReID while requiring less than 10% computational complexity. Furthermore, LEViT obtains the state-of-the-art on three deep metric learning datasets: CUB-200-2011, Cars196, and University-1652. Our code will be available athttps://github.com/YuhuiWang99/LEViT.

Existing multimedia recommender systems provide users with suggestions of media by evaluating similarities, such as games and movies. To enhance the semantics and explainability of embeddings, it is a consensus to apply additional information (e.g., interactions, contexts, popularity). However, without systematically considering representativeness and value, the utility and explainability of embedding drop drastically. Hence, we introduceRVRec, a plug-and-play model-agnostic embedding enhancement approach that can improve both personality and explainability of existing systems. Specifically, we propose a probability-based embedding optimization method that uses a contrastive loss based on negative 2-Wasserstein distance to learn to enhance the representativeness of the embeddings. In addition, we introduce a reweighing method based on a multivariate Shapley values strategy to evaluate and explore the value of interactions and embeddings. Extensive experiments on multiple backbone recommenders and real-world datasets show that RVRec can improve the personalization and explainability of existing recommenders, outperforming state-of-the-art baselines.

Audio classification plays a crucial role within fields such as human-machine interaction and intelligent robotics. However, high-performance audio classification systems typically demand significant computational and storage resources, posing substantial challenges when deploying to the resource-constrained edge devices with an urgent need for such capabilities. To achieve a new level of balance between model complexity and performance, we introduce a novel multi-view method for the separated time-frequency features extraction and utilization, which exists within the proposed Mini Mirror Multi-View Network (M3Net) in the form of the Mirror Attention mechanism. M3Net enables reversible spatial transformation of spectral features is capable of efficiently leverages robust local and global features in the time and frequency domains with low requirements for parameters. Experiments based on Mel-Spectrogram without data augmentation and pre-training indicate that M3Net can achieve classification accuracy over 97% on the UrbanSound8K and SpeechCommandsV2 datasets with only 0.03 million parameters. The contribution of each functional segment in M3Net is fully verified and explained in the ablation experiments.

Unpaired image dehazing has attracted increasing attention due to its flexible data requirements during model training. Dominant methods based on contrastive learning not only introduce haze-unrelated content information, but also ignore haze-specific properties in the frequency domain (\ie,~haze-related degradation is mainly manifested in the amplitude spectrum). To address these issues, we propose a novel frequency domain-based diffusion model, named \ours, for fully exploiting the beneficial knowledge in unpaired clear data. In particular, inspired by the strong generative ability shown by Diffusion Models (DMs), we tackle the dehazing task from the perspective of frequency domain reconstruction and perform the DMs to yield the amplitude spectrum consistent with the distribution of clear images. To implement it, we propose an Amplitude Residual Encoder (ARE) to extract the amplitude residuals, which effectively compensates for the amplitude gap from the hazy to clear domains, as well as provide supervision for the DMs training. In addition, we propose a Phase Correction Module (PCM) to eliminate artifacts by further refining the phase spectrum during dehazing with a simple attention mechanism. Experimental results demonstrate that our \ours outperforms other state-of-the-art methods on both synthetic and real-world datasets.

Conversational Recommendation Systems (CRS) play a pivotal role in personalized cultural discovery by guiding user attention and mitigating information overload through interactive dialogue. However, existing Transformer-based CRS models predominantly focus on token-level generation, limiting their ability to capture sentence-level semantic interaction patterns. Furthermore, while user preferences are often inferred through interaction entities among these entities, current approaches typically overlook the sequential dependencies, either semantic or ID-based, which are crucial for accurate and context-aware recommendations. To overcome these limitations, we propose SDCRS, a sequence-augmented CRS based on diffusion models, which integrates sentence-level and entity-level sequential modeling to enhance the response generation and recommendation modules. By integrating diffusion mechanisms, SDCRS not only improves the diversity of generated responses but also enhances user preference modeling, particularly under cold-start conditions. Comprehensive experiments clearly demonstrate that SDCRS achieves superior performance over all baselines.

Oriented object detection seeks to determine both the position and orientation of objects, yet angle periodicity often limits performance. To solve this issue, we rethink label assignment and sampling strategies and propose a pair of orientation-aware assigner and sampler (OAS) for a two-stage detector. The orientation-aware assigner (OA) incorporates angle and location to improve positive and negative sample assignment, while the orientation-aware sampler (OS) ranks positive samples by their angular difference from ground truth, adjusting learning weights by soft sampling. Such a design significantly mitigates the angle periodicity problem and enables detector focusing on high-quality samples with a more consistent orientation for training. Experimental results on two challenging oriented object detection benchmarks demonstrate that OAS can consistently boost the detection accuracy based on many existing two-stage detectors (e.g., Oriented R-CNN and RPGAOD) without additional cost. Both code and pretrained models are available athttps://github.com/skyandkibo/OAS.

Super-resolution (SR) aims to restore a high-resolution (HR) image from its low-resolution (LR) counterpart. Existing works try to achieve an overall average recovery over all regions to provide better visual quality for human viewing. If we desire to explore the potential that performs super-resolution for machine recognition instead of human viewing, the solution should change accordingly. From this insight, we propose a new SR pipeline, called InstanceSR, which treats each region in the LR image differentially and consumes more resources to focus on the recovery of the foreground region where the instances exist. In particular, InstanceSR consists of an encoder that formulates the LR image into a set of various difficulty tokens according to the instances distribution in each sub-region, and a decoder based on a multi-exit network structure to recover the sub-regions corresponding to various difficulty tokens by consuming different computational resources. Experimental results demonstrate the superiority of the proposed InstanceSR over state-of-the-art models, especially the recovery of regions where instances exist, by extensive quantitative and qualitative evaluations on three widely used benchmarks containing small instances. Besides, the comparisons using SR results on three challenging small object detection benchmarks verify that our InstanceSR can consistently boost the detection accuracy and has great potential for subsequent machine recognition.

Deep neural network advancements have led to significant progress in industrial applications, particularly in product recognition for smart automated vending machines (AVMs). This area has seen increased market demand as a fundamental part of automatedretail. However, the existing works possess critical gaps: 1) densely placed and obscured objects in AVMs lead to inaccurate product recognition results, necessitating auxiliary information for achieving precise detection; 2) the lack of datasets with auxiliary information hinders further development in this field. To address these gaps, we propose a depth-guided product recognition network, which consists of two novel components: a depth-aware feature pyramid network (DFPN) and a depth-aware regression head (DRH). Our DFPN can adaptively select features that are beneficial for regression from both red, green, blue (RGB) and depth data, whereas the DRH refines the regression branch via depth information without affecting the classification process. In addition, to overcome dataset limitations, we develop an extended and fully annotated depth information dataset namedSmartUVM-D, which includes depth information for each image based on the existing SmartUVM dataset. The experimental results obtained on our SmartUVM-D benchmark show that our method effectively solves the inaccurate product recognition problem and achieves substantial gains over the baseline approaches. Specifically, our method (based on the ATSS framework) achieves a mean average precision of 84.4, representing a 2.3-point improvement over the previously developed ATSS method and establishing a new state-of-the-art approach.

Single-Domain Generalization Object Detection (Single-DGOD) refers to training a model with only one source domain, enabling the model to generalize to any unseen domain. For instance, a detector trained on a sunny daytime dataset should also perform well in scenarios such as rainy nighttime. The main challenge is to improve the detector's ability to learn the domain-invariant representation (DIR) while removing domain-specific information. Recent progress in Single-DGOD has demonstrated the efficacy of removing domain-specific information by adjusting feature distributions. Nonetheless, simply adjusting the global feature distribution in Single-DGOD task is insufficient to learn the potential relationship from sunny to adverse weather, as these ignore the significant domain gaps between instances across different weathers. In this paper, we propose a novel object detection method for more robust single-domain generalization. In particular, it mainly consists of a frequency-aware selective whitening module (FSW) for removing redundant domain-specific information and a contrastive feature alignment module (CFA) for enhancing domain-invariant information among instances. Specially, FSW extracts the magnitude spectrum of the feature and uses a group whitening loss to selectively eliminate redundant domain-specific information in the magnitude. To further eliminate domain differences among instances, we apply the style transfer method for data augmentation and use the augmented data in the CFA module. CFA formulates both the original and the augmentd RoI features into a series of groups with different categories, and utilizes contrastive learning across them to facilitate the learning of DIR in various categories. Experiments show that our method achieves favorable performance on existing standard benchmarks.

Out-of-distribution detection aims to protect models against overconfidently categorizing samples from unknown categories,i.e., out-of-distribution data (OOD), into known categories,i.e., in-distribution data (ID). From the perspective of feature distribution, the difference between OOD samples and ID samples can be decomposed into semantic shifts and covariate shifts. Most DL-based methods only extract deeper features, which represent semantic shifts, to discern feature variances in the data, ignoring the exploration of covariance shifts. In this paper, we propose a Shallow Layer-driven Enhanced OOD detection method (SLE), which enhances the difference of OOD samples by exploiting covariate shifts in shallow features. Specifically, it contains three main components: Hierarchical Feature Extractor (HFE), Adaptive Dimensionality Reduction Strategy (ADR), Cross-layer Score Aggregator (CSA). HFE is responsible for extracting both deeper and shallow features from the deep network. ADR adaptively reduces all hierarchical feature dimensionality according to sample characteristics, avoiding feature redundancy. CSA defines a novel confidence score for OOD samples, that effectively prevents confusion in the feature representation space at each layer. In SLE, these three closely related components cooperate with each other to effectively enhance the representation ability of OOD samples and divide OOD data better. We conduct extensive experiments to examine the performance of SLE in four benchmarks and discuss its individual components. This method performs well on the OOD datasets.

Compressed sensing (CS) provides a robust and simple framework for compressing images in resource-constrained environments. However, CS-based image coding schemes often have poor rate-distortion (R-D) performance, particularly due to the quantization process. Our research indicates that leveraging the image prior enables the estimation of most significant bits (MSBs) from least significant bits (LSBs), which provides a quantization strategy to improve R-D performance without increasing coding complexity. That is discarding MSBs of measurements, and only transmitting LSBs to the decoder side. At the decoder side, we reconstruct images by solving an inverse-quantization set-constrained CS optimization problem. Our approach further employs a tailored designed deep denoiser as the proximal operator to enhance the reconstructed image quality. Extensive experimental results demonstrate that the proposed scheme achieves satisfactory performance, with promising R-D results (PSNR gains over 1.71 dB than JPEG at 0.50 bpp compression ratio), and robust bit error and loss resilience (reconstructed 29.98 dB even with 50% bit loss at 0.50 bpp compression ratio), meanwhile having lower encoding complexity (less than half encoding time of CCSDS-IDC).

Under-display camera (UDC) systems are the foundation of full-screen display devices in which the lens mounts under the display. The pixel array of light-emitting diodes used for display diffracts and attenuates incident light, causing various degradations as the light intensity changes. Unlike general video restoration which recovers video by treating different degradation factors equally, video restoration for UDC systems is more challenging that concerns removing diverse degradation over time while preserving temporal consistency. In this paper, we introduce a novel video restoration network, called D2RNet, specifically designed for UDC systems. It employs a set of Decoupling Attention Modules (DAM) that effectively separate the various video degradation factors. More specifically, a soft mask generation function is proposed to formulate each frame into flare and haze based on the diffraction arising from incident light of different intensities, followed by the proposed flare and haze removal components that leverage long- and short-term feature learning to handle the respective degradations. Such a design offers an targeted and effective solution to eliminating various types of degradation in UDC systems. We further extend our design into multi-scale to overcome the scale-changing of degradation that often occur in long-range videos. To demonstrate the superiority of D2RNet, we propose a large-scale UDC video benchmark by gathering HDR videos and generating realistically degraded videos using the point spread function measured by a commercial UDC system. Extensive quantitative and qualitative evaluations demonstrate the superiority of D2RNet compared to other state-of-the-art video restoration and UDC image restoration methods.

Eliminating image blur produced by various kinds ofmotion has been a challenging problem. Dominant approaches rely heavily on model capacity to remove blurring by reconstructing residual from blurry observation in feature space. These practices not only prevent the capture of spatially variable motion in the real world but also ignore the tai-lored handling of various motions in image space. In this paper, we propose a novel real-world deblurring filtering model called the Motion-adaptive Separable Collaborative (MISC) Filter. In particular, we use a motion estimation net-work to capture motion information from neighborhoods, thereby adaptively estimating spatially-variant motion flow, mask, kernels, weights, and offsets to obtain the MISC Fil-ter. The MISC Filter first aligns the motion-induced blur-ring patterns to the motion middle along the predicted flow direction, and then collaboratively filters the aligned image through the predicted kernels, weights, and offsets to generate the output. This design can handle more general-ized and complex motion in a spatially differentiated man-ner. Furthermore, we analyze the relationships between the motion estimation network and the residual reconstruction network. Extensive experiments on four widely used bench-marks demonstrate that our method provides an effective solution for real-world motion blur removal and achieves state-of-the-art performance. Code is available at https://github.com/ChengxuLiu/MISCFilter.

Improving model prediction capability based on continuously collected data is one of the biggest challenges in a mature, intelligent weather forecasting system. To tackle this problem, we propose an online incremental learning strategy for meteorological models with asynchronous updates. In short, we divide two different meteorological incremental learning settings according to the characteristics of meteorological data, distinguishing between increments within short-term time windows and increments for long-term data. For short-term data, a structure-based incremental learning method with a fast iteration rate is used, and a Residual-Net (R-Net) is proposed to improve the performance of the model; for long-term data, a replay-based incremental learning method called Gradient-based Core-set Selection and Weighting method (GCSW) is proposed, by performing cosine distance and normalization calculations to obtain sample weights and weighting the coreset samples to avoid catastrophic forgetting. The comparative experiments on temperature prediction incremental datasets in Beijing and Xi’an show that our proposed method achieves the best results in both short-term and long-term incremental experimental settings than all other baseline methods, demonstrating the advantages of the algorithm we proposed on meteorological datasets. The code is available as an open source repository on GitHub https://github.com/liujunjiao1/Enhancing-Weather-Model.

Composed image retrieval (CIR) is the task of searching target images using an image-text pair as a query. Given the straightforward relation of query pair-target image, the dominant methods follow the learning paradigm of common image-text retrieval and simply model this problem as the query-target matching problem. Particularly, the common practice first encodes the multi-modal query into one feature and then aligns it with the target image. However, such a learning paradigm only explores the naive relation in the triplets. We argue that CIR triplets encompass additional associations besides the primary query-target relation, which is overlooked in existing works. In this paper, we disclose two new relations residing in the triplets by viewing the triplet as a graph node. In analogy with the graph node, we mine two associations of text-bridged image alignment and complementary text reasoning. The text-bridged image alignment considers composed image retrieval as a specialized form of image retrieval, where the query text acts as a bridge between the query image and the target one, and a hinge-based cross attention is proposed to incorporate this relation into the network learning. On the other hand, the association of complementary text reasoning regards composed image retrieval as a specific type of cross-modal retrieval, where the composite two images are used to reason the complementary text. To integrate these views effectively, a twin attention-based compositor is designed. By combining these two types of complementary associations with the explicit query pair-target image relation, we establish a comprehensive set of constraints for composed image retrieval. With the above designs, we finally developed our CaLa, a Complementary Association Learning framework for Augmenting Composed Image Retrieval. Experimental evaluations are conducted on the widely-used CIRR and FashIionIQ benchmarks with multiple backbones to validate the effectiveness of our CaLa. The results demonstrate the superiority of our method in the composed image retrieval task. Our code and models are available at https://github.com/Chiangsonw/CaLa

With the increasing popularity of intelligent surveillance systems, abnormal behavior detection of human beings based on computer vision is attracting more attention. It aims to classify and locate the abnormal behaviors and coordinates of human beings, respectively, and is a fundamental technology for intelligent security. Existing approaches mainly focus on exploring abnormal behavior features through object detectors. However, in office scenarios, almost all abnormal behaviors are closely associated with the fine-grained feature around the nose, wrist, elbow, and other human joint points regions. Detectors for generic objects cannot adequately capture such differences between abnormal behaviors, resulting in sub-optimal performance. In this paper, we focus on human joints and take one step further to enable effective behavior characteristics learning in office scenarios. In particular, we propose a novel Adaptive Joints Enhancement Network (AJENet), which includes two closely-related components, Joints Predict block (JP) and Adaptive Key Joints Enhancement block (AKJE). JP block is used to predict the human joints and facilitates the feature learning around them implicitly. By inputting the features around joints, the AKJE block enhances the feature representations of key joints according to the abnormal behavior characteristics adaptively. Experimental results demonstrate that our method outperforms other state-of-the-art methods on the collected real office scenario Office Behavior Dataset. Besides, to verify the generalization capabilities and potential of AJENet, we construct comparisons on another generic dataset PASCAL VOC 2012 Action.

Few-shot class-incremental learning (FSCIL) aims to continually learn new classes using a few samples while not forgetting the old classes. The scarcity of new training data will seriously destroy the model’s stability and plasticity. Continually Evolved Classifiers (CEC) (Zhang et al., 2021), a kind of framework, maintains the stability by freezing the encoder and achieves the plasticity by evolving the classifier along with a pseudo incremental learning scheme. However, the performance of CEC is limited due to 1) inequitable information gains between classifier weights and test features, and 2) inefficient learning task construction strategy. To address the first issue, we propose a Knowledge-guided Relation Refinement Module (KRRM) to update both the classifier weights and test features. The main function of KRRM is achieved through cross-attention to propagate the knowledge represented by old encoded data. To address the second issue, we design a Pseudo Incremental relation Refinement Learning (PIRL) that utilizes a novel hard concepts mining strategy to mine hard concept tasks globally and locally. By successfully addressing the two issues, our proposed method, named Improved Continually Evolved Classifiers (CEC+), extends the potential of CEC without introducing any additional parameters. More precisely, extensive experiments on CIFAR100, miniImageNet, and Caltech-UCSD Birds-200-2011, demonstrate that our proposed method surpasses prior state-of-the-art methods.

High-resolution networks have made significant progress in dense prediction tasks such as human pose estimation and semantic segmentation. To better explore this high-resolution mechanism on mobile devices, Lite-HRNet incorporates shuffle operations to reduce computational complexity in the channel dimension, while Dite-HRNet employs dynamic convolution and pooling to capture long-range interactions with low computational complexity in the spatial dimension. The core idea behind both approaches is to efficiently capture information in either the channel or spatial dimension. However, shuffle operations and dynamic operations are not hardware-friendly. As a result, both Lite-HRNet and Dite-HRNet cannot achieve the desired inference speed on specialized devices, including Neural Processing Units (NPUs) and Graphics Processing Units (GPUs). To overcome these limitations, we present a simple Hardware-Friendly Lightweight High-resolution Network (HF-HRNet) based on our proposed Hardware-Friendly Uniform-sized Mug (HUM) block. HUM block mainly consists of the Cascaded Depthwise (CAD) block and Multi-Scale Context Embedding (MCE) block. The CAD block cascades depthwise convolutions to obtain a larger receptive field in the spatial dimension, while the MCE block aggregates multi-scale spatial feature information from different scales and adjusts channel features. Extensive experiments are conducted on human pose estimation (COCO, MPII) and semantic segmentation (Cityscapes), resulting in a better trade-off between inference speed and accuracy on both NPUs and GPUs. It is noteworthy that on the COCO test-dev set, HF-HRNet-30 outperforms Dite-HRNet-30 and Lite-HRNet-30 by 1.9 AP and 2.8 AP, respectively, while running about 13 times faster and 9 times faster on NPUs, respectively. Our code are publicly available for use: https://github.com/zhanghao5201/HF-HRNet.

AI-based methods are shining across a variety of industries, especially unmanned retail. Product recognition is the problem of recognizing the category and quantity of products (e.g., beverages and mineral water) in intelligent unmanned vending machines (UVMs) to automatic checkout during purchase. However, for similar products in hundreds of categories, the existing method is not accurate enough. Besides, they cannot be extended for new products without retraining. In this article, we propose a product recognition approach based on intelligent UVMs, calledSplit-Check, which first splits the region of interest of products by detection and then check product by instance-level retrieval. Split-Check is the combination of two important components. The preliminary detection distinguishes items that contain the different coarse-grained features, then locates items, and classifies them into coarse-grained categories as a candidate. The retrieval further distinguishes the candidate items that contain the different fine-grained features. Besides, we reconstruct a large-scale categories product dataset GOODS-85 based on actual UVMs scenarios, in which the number of categories of items is larger than the existing dataset. Experimental results demonstrate the effectiveness of the proposed approach. Our method significantly improves the recognition performance of hundreds of products and increases the scalability of products.

We present ReGO (Reference-Guided Outpainting), a new method for the task of sketch-guided image outpainting. Despite the significant progress made in producing semantically coherent content, existing outpainting methods often fail to deliver visually appealing results due to blurry textures and generative artifacts. To address these issues, ReGO leverages neighboring reference images to synthesize texture-rich results by transferring pixels from them. Specifically, an Adaptive Content Selection (ACS) module is incorporated into ReGO to facilitate pixel transfer for texture compensating of the target image. Additionally, a style ranking loss is introduced to maintain consistency in terms of style while preventing the generated part from being influenced by the reference images. ReGO is a model-agnostic learning paradigm for outpainting tasks. In our experiments, we integrate ReGO with three state-of-the-art outpainting models to evaluate its effectiveness. The results obtained on three scenery benchmarks, i.e. NS6K, NS8K and SUN Attribute, demonstrate the superior performance of ReGO compared to prior art in terms of texture richness and authenticity. Our code is available at https://github.com/wangyxxjtu/ReGO-Pytorch.

Conversational recommendation systems (CRS) can actively discover users’ preferences and perform recommendations during conversations. The majority of works on CRS tend to focus on a single conversation and dig it using knowledge graphs, language models, etc. However, they often overlook the abundant and rich preference information that exists in the user's historical conversations. Meanwhile, end-to-end generation of recommendation results may lead to a decrease in recommendation quality. In this work, we propose a personalized conversational recommendation system infused with historical interaction information. This framework leverages users’ preferences extracted from their historical conversations and integrates them with the users’ preferences in current conversations. We find that this contributes to higher accuracy in recommendations and fewer recommendation turns. Moreover, we improve the interactive pattern between the recommendation module and the dialogue generation module by utilizing the slot filling method. This enables the results inferred by the recommendation module to be integrated into the conversation naturally and accurately. Our experiments on the benchmark dataset demonstrate that our model significantly outperforms the state-of-the-art methods in the evaluation of recommendations and dialogue generation.

Sequential recommendation mines the user's interaction sequence or time information to get better recommendations and thus is gaining more and more attention. Existing sequential recommendations tend to build new models, and the study of the loss function is seriously neglected. Despite the increasing attention paid to contrastive learning recently, we believe that the key to contrastive learning is contrastive loss(CL), which also provides a new option for sequential recommendation. However, we find it works against the personalized representation of features. First, it is a relative constraint that keeps positive and negative samples away from each other but without an absolute constraint. Second, recent studies have shown that all embeddings should be uniformly distributed. However, CL only widens the distance of positive and negative samples within the training batch, rather than making a uniform distribution of all items. These two shortcomings make the embedding space too compact, which is harmful to personalized representation and recommendation. Therefore, this article proposes Personalized Contrastive Loss (PCL) to combine CL with absolute constraints of BCE/CE and employs regularization methods to make the representations uniformly distributed. State-of-the-art results are obtained in experiments on several commonly used datasets. The code and data will be available on GitHub.

Existing point-of-interest (POI) recommendation methods only show the direct recommendation results and lack the proper reasons for recommendation. In recent years, explainable recommendation has become an increasingly important subfield in recommendation systems. The aim of explainable recommendation is to provide a reason why an item is recommended to a user. In this way, it helps to improve the transparency, persuasiveness and user satisfaction of recommendation systems. The explainable recommendation should indicate users' preferences for POIs, such as the category and the price. In addition, to increase the diversity of the results, we take emotional intensity into account in our model to generate more vivid reasons. To this end, we propose a hierarchical attention-based transformer model to generate reasons with specific topics and different emotions. With a hierarchical attention mechanism, we can capture the word-level and attribute-level preferences of users. In addition, we also learn the latent representation of the emotion score to generate diverse recommendation reasons. We evaluate the proposed model on a new real-world dataset collected from three travel service websites. The experimental results demonstrate that our method outperforms the related approaches for reason generation.

Practical applications with visual question answering (VQA) systems are challenging, and recent research has aimed at investigating this important field. Many issues related to real-world VQA applications must be considered. Although existing methods have focused on adding external knowledge and other descriptive information to assist in reasoning, they are limited by the impact of information retrieval errors on downstream tasks and the misalignment of the aggregated information. Thus, the overall performance of these models must be improved. To address these challenges, we propose a novel VQA model that utilizes a differentiated pretrained model to represent the input information and connects the input data with three external knowledge components through a common feature space. To combine the information in the three feature spaces, we propose an information aggregation strategy that employs a weighted score to aggregate the information in the relation and entity spaces in the answer prediction process. The experimental results show that our method achieves good performance in fact-based and zero-shot VQA tasks and achieves state-of-the-art performance with the ZS-F-VQA dataset.

Detecting objects in large-scale drone-view images is notoriously challenging due to their uneven distribution and scale variation caused by photoing angles. Common approaches promote drone-view object detection by two-step detection (i.e., detecting sub-regions first) and multi-scale input. However, all these methods suffer from onerous computational costs since the high model complexity and input resolution. In this paper, we propose a novel one-step detector, calledSDPDet, to enable effective object learning in drone-view images. In particular, a Scale-separated Activation Pyramid (SAP) serves to focus on the regions with objects aggregated at each scale, and a Scale-separated Learnable Proposals (SLP) mechanism learns proposal boxes and corresponding features on these regions. By such design, the quantity of learnable proposals allows dynamic adjustment at each scale separately, which facilitates the objects learning of various distributions and scales with less computational costs. Experiments demonstrate SDPDet can significantly outperform the state-of-the-art one-step detectors on three widely-used benchmarks. On the most challenging VisDrone dataset, SDPDet with ResNet50 gains5.4%AP and6.9%AP$_{s}$improvements while running1.9×faster than previous models.

Cross-Domain Recommendation (CDR) aims to alleviate the cold-start problem by transferring knowledge from a data-rich domain (source domain) to a data-sparse domain (target domain), where knowledge needs to be transferred through a bridge connecting the two domains. Therefore, constructing a bridge connecting the two domains is fundamental for enabling cross-domain recommendation. However, existing CDR methods often overlook the valuable of natural relationships between items in connecting the two domains. To address this issue, we propose DKTCDR: a Domain-oriented Knowledge Transfer method for Cross-Domain Recommendation. In DKTCDR, We leverages the rich relationships between items in a cross-domain knowledge graph as bridges to facilitate both intra- and inter-domain knowledge transfer. Additionally, we design a cross-domain knowledge transfer strategy to enhance inter-domain knowledge transfer. Furthermore, we integrate the semantic modality information of items with the knowledge graph modality information to enhance item modeling. To support our investigation, we construct two high-quality cross-domain recommendation datasets, each containing a cross-domain knowledge graph. Our experimental results on these datasets validate the effectiveness of our proposed method. Source code is available athttps://github.com/zxxxl123/DKTCDR.

Recently, the emerging concept of "unmanned retail" has drawn more and more attention, and the unmanned retail based on the intelligent unmanned vending machines (UVMs) scene has great market demand. However, existing product recognition methods for intelligent UVMs cannot adapt to large-scale categories and have insufficient accuracy. In this article, we propose a method for large-scale categories product recognition based on intelligent UVMs. It can be divided into two parts: 1) first, we explore the similarities and differences between products through manifold learning, and then we build a hierarchical multigranularity label to constrain the learning of representation; and 2) second, we propose a hierarchical label object detection network, which mainly includes coarse-to-fine refine module (C2FRM) and multiple granularity hierarchical loss (MGHL), which are used to assist in capturing multigranularity features. The highlights of our method are mine potential similarity between large-scale category products and optimization through hierarchical multigranularity labels. Besides, we collected a large-scale product recognition dataset GOODS-85 based on the actual UVMs scenario. Experimental results and analysis demonstrate the effectiveness of the proposed product recognition methods.

Visual Geometry Group (VGG)-style ConvNet is an neural-network process units (NPU)-friendly network; however, the accuracy of this architecture cannot keep up with other well-designed network structures. Although some reparameterization methods are proposed to remedy this weakness, their performance suffers from the homogenization issue of parallel branches, and the preset shape of convolution kernels also influences spatial perception. To address this problem, we propose a diversity-learning (DL) block to build the DLNet, which could adaptively learn various features to enrich the feature space. To balance floating point of operations (FLOPs) and accuracy, groupwise operation is introduced and finally, a lightweight DL ConvNet DLGNet is obtained. Extensive evaluations have been conducted on different computer vision tasks, e.g., image classification [Canadian Institute For Advanced Research (CIFAR) and ImageNet], object detection [PASCAL visual object classes (VOC) and Microsoft Common Objects in Context (MS COCO)], and semantic segmentation (Cityscapes). The experimental results show that our proposed DLGNet can achieve comparable performance with the state-of-the-art networks while the speed is 183% faster than GhostNet and even over 600% faster than MobileNetV3 with similar accuracy when running on NPU.

Domain Adaptive Object Detection (DAOD) aims to improve the detection performance of target domains by minimizing the feature distribution between the source and target domain. Recent approaches usually align such distributions in terms of categories through adversarial learning and some progress has been made. However, when objects are non-uniformly distributed at different scales, such category-level alignment causes imbalanced object feature learning, refer as the inconsistency of category alignment at different scales. For better category-level feature alignment, we propose a novel DAOD framework of joint category and scale information, dubbed CSDA, such a design enables effective object learning for different scales. Specifically, our framework is implemented by two closely-related modules: 1) SGFF (Scale-Guided Feature Fusion) fuses the category representations of different domains to learn category-specific features, where the features are aligned by discriminators at three scales. 2) SAFE (Scale-Auxiliary Feature Enhancement) encodes scale coordinates into a group of tokens and enhances the representation of category-specific features at different scales by self-attention. Based on the anchor-based Faster-RCNN and anchor-free FCOS detectors, experiments show that our method achieves state-of-the-art results on three DAOD benchmarks.

Under-display camera (UDC) systems remove the screen notch for bezel-free displays and provide a better interactive experience. The main challenge is that the pixel array of light-emitting diodes used for display diffracts and attenuates the incident light, leading to complex degradation. Existing models eliminate spatial diffraction by maximizing model capacity through complex design and ignore the periodic distribution of diffraction in the frequency domain, which prevents these approaches from satisfactory results. In this paper, we introduce a new perspective to handle various diffraction in UDC images by jointly exploring the feature restoration in the frequency and spatial domains, and present a Frequency and Spatial Interactive Learning Network (FSI). It consists of a series of well-designed Frequency-Spatial Joint (FSJ) modules for feature learning and a color transform module for color enhancement. In particular, in the FSJ module, a frequency learning block uses the Fourier transform to eliminate spectral bias, a spatial learning block uses a multi-distillation structure to supplement the absence of local details, and a dual transfer unit to facilitate the interactive learning between features of different domains. Experimental results demonstrate the superiority of the proposed FSI over state-of-the-art models, through extensive quantitative and qualitative evaluations in three widely-used UDC benchmarks.

Recently, prompt-based learning has made impressive progress on image class-incremental learning, but it still lacks sufficient exploration in the video domain. In this paper, we will fill this gap by learning multiple prompts based on a powerful image-language pre-trained model, i.e., CLIP, making it fit for video class-incremental learning (VCIL). For this purpose, we present a space-time prompting approach (ST-Prompt) which contains two kinds of prompts, i.e., task-specific prompts and task-agnostic prompts. The task-specific prompts are to address the catastrophic forgetting problem by learning multi-grained prompts, i.e., spatial prompts, temporal prompts and comprehensive prompts, for accurate task identification. The task-agnostic prompts maintain a globally-shared prompt pool, which can empower the pre-trained image models with temporal perception abilities by exchanging contexts between frames. By this means, ST-Prompt can transfer the plentiful knowledge in the image-language pre-trained models to the VCIL task with only a tiny set of prompts to be optimized. To evaluate ST-Prompt, we conduct extensive experiments on three standard benchmarks. The results show that ST-Prompt can significantly surpass the state-of-the-art VCIL methods, especially it gains 9.06% on HMDB51 dataset under the 1 × 25 stage setting.

Existing real-world video super-resolution (VSR) methods focus on designing a general degradation pipeline for open-domain videos while ignoring data intrinsic characteristics which strongly limit their performance when applying to some specific domains (e.g., animation videos). In this paper, we thoroughly explore the characteristics of animation videos and leverage the rich priors in real-world animation data for a more practical animation VSR model. In particular, we propose a multi-scale Vector-Quantized Degradation model for animation video Super-Resolution (VQD-SR) to decompose the local details from global structures and transfer the degradation priors in real-world animation videos to a learned vector-quantized codebook for degradation modeling. A rich-content Real Animation Low-quality (RAL) video dataset is collected for extracting the priors. We further propose a data enhancement strategy for high-resolution (HR) training videos based on our observation that existing HR videos are mostly collected from the Web which contains conspicuous compression artifacts. The proposed strategy is valid to lift the upper bound of animation VSR performance, regardless of the specific VSR model. Experimental results demonstrate the superiority of the proposed VQD-SR over state-of-the-art methods, through extensive quantitative and qualitative evaluations of the latest animation video super-resolution benchmark. The code and pre-trained models can be downloaded at https://github.com/researchmm/VQD-SR.

The high rate of false arrhythmia alarms in intensive care units (ICUs) can negatively impact patient care and lead to slow staff response time due to alarm fatigue. To reduce false alarms in ICUs, previous works proposed conventional supervised learning methods which have inherent limitations in dealing with high-dimensional, sparse, unbalanced, and limited data. We propose a deep generative approach based on the conditional denoising diffusion model to detect false arrhythmia alarms in the ICUs. Conditioning on past waveform data of a patient, our approach generates waveform predictions of the patient during an actual arrhythmia event, and uses the distance between the generated and the observed samples to classify the alarm. We design a network with residual links and self-attention mechanism to capture long-term dependencies in signal sequences, and leverage the contrastive learning mechanism to maximize distances between true and false arrhythmia alarms. We demonstrate the effectiveness of our approach on the MIMIC II arrhythmia dataset for detecting false alarms in both retrospective and real-time settings.

Abstract This paper proposed an image compression‐encryption scheme based on compressive sensing theory, which achieves high security, strong robustness, and high rate‐distortion performance. First, the denoising preprocessing strategy is applied at the encoder side, which can enhance the rate‐distortion performance without sacrificing security and robustness. Second, the preprocessed image is randomly down‐sampled using scrambled block Bernoulli sampling with diffusion noise (SBBS‐DN), which is generated by combining a hyper‐chaotic system and SHA256 hash of the plain image. Third, a deep‐learned plug‐and‐play is embedded prior for plain image reconstruction at the decoder side. Simulation results show that the proposed scheme has desirable security performance (being resistant to different attacks), high R‐D performance (PSNR gains over 1.3 dB than JPEG at 0.50 bpp compression ratio), and high error resilience (reconstructed 29.92 dB at 0.50 bpp compression ratio even with 50% bit loss).

Many lightweight networks have been proposed for resource-limited applications, however, they cannot be efficiently applied to neural-network processing units (NPUs) due to the limited operations supported by the NPUs, and few works focus on efficient network design on the NPUs. The basic blocks of networks such as MobileNetV2 and RegNet use smaller convolution kernels with relatively small receptive fields, which are not conducive to capturing large-scale spatial information. To address this weakness, we propose Shifting and Cascaded Group (SCG) block, where we cascade group convolutions with larger kernels to exploit multi-scale information and propose shifting group convolution to communicate channel information between different groups. Besides, we carefully devise our architecture guided by some principles and finally build a very efficient network called Shifting and Cascaded Group Network (SCGNet) on NPUs. To verify the superiority of our method, we conduct extensive experiments on various tasks including image classification, object detection, human pose estimation, person re-identification, and semantic segmentation to comprehensively evaluate the performance. Results on widely used datasets such as ImageNet, PASCAL VOC, COCO, MPII, Market-1501, DukeMTMC-ReID, CUHK03, and Cityscapes demonstrate that the proposed network is a more effective network on the corresponding vision tasks.

Short-to-medium term temperature prediction in high resolution is a very challenging task, involving meteorology, physics, mathematics, geography, and many other subjects. Its purpose is to fit a complex function from historical meteorological data to predict the future 1–5 days temperature, which is a typical spatio-temporal prediction problem. Meteorological data show complex correlations in local space. Most of the existing machine learning methods are based on image pixel-level tasks or spatio-temporal prediction tasks, which model meteorological data without considering the characteristics of meteorological data and use rough global patterns to model local space which would lose many details. To address the above issues, our work fine-grained conditional convolution network (FCCN) proposes a novel grid-level conditional convolution module, including a local geographic adaptive weight (GAW) and a local data adaptive weight (DAW). These two components are integrated into a multiscale meteorological fusion gated recurrent unit (GRU) architecture for the end-to-end temperature prediction. Experiments in real-world datasets from ERA-5 show our FCCN model has a better performance than all other baseline methods.

Video frame interpolation (VFI) aims to synthesize an intermediate frame between two consecutive frames. State-of-the-art approaches usually adopt a two-step solution, which includes 1) generating locally-warped pixels by calculating the optical flow based on pre-defined motion patterns (e.g., uniform motion, symmetric motion), 2) blending the warped pixels to form a full frame through deep neural synthesis networks. However, for various complicated motions (e.g., non-uniform motion, turn around), such improper assumptions about pre-defined motion patterns introduce the inconsistent warping from the two consecutive frames. This leads to the warped features for new frames are usually not aligned, yielding distortion and blur, especially when large and complex motions occur. To solve this issue, in this paper we propose a novel Trajectory-aware Transformer for Video Frame Interpolation (TTVFI). In particular, we formulate the warped features with inconsistent motions as query tokens, and formulate relevant regions in a motion trajectory from two original consecutive frames into keys and values. Self-attention is learned on relevant tokens along the trajectory to blend the pristine features into intermediate frames through end-to-end training. Experimental results demonstrate that our method outperforms other state-of-the-art methods in four widely-used VFI benchmarks. Both code and pre-trained models will be released at https://github.com/ChengxuLiu/TTVFI.

Image enhancement aims at improving the aesthetic visual quality of photos by retouching the color and tone, and is an essential technology for professional digital photography. Recent years deep learning-based image enhancement algorithms have achieved promising performance and attracted increasing popularity. However, typical efforts attempt to construct a uniform enhancer for all pixels' color transformation. It ignores the pixel differences between different content (e.g., sky, ocean, etc.) that are significant for photographs, causing unsatisfactory results. In this paper, we propose a novel learnable context-aware 4-dimensional lookup table (4D LUT), which achieves content-dependent enhancement of different contents in each image via adaptively learning of photo context. In particular, we first introduce a lightweight context encoder and a parameter encoder to learn a context map for the pixel-level category and a group of image-adaptive coefficients, respectively. Then, the context-aware 4D LUT is generated by integrating multiple basis 4D LUTs via the coefficients. Finally, the enhanced image can be obtained by feeding the source image and context map into fused context-aware 4D LUT via quadrilinear interpolation. Compared with traditional 3D LUT, i.e., RGB mapping to RGB, which is usually used in camera imaging pipeline systems or tools, 4D LUT, i.e., RGBC(RGB+Context) mapping to RGB, enables finer control of color transformations for pixels with different content in each image, even though they have the same RGB values. Experimental results demonstrate that our method outperforms other state-of-the-art methods in widely-used benchmarks.

Dating recommendation becomes a critical task since the rapid development of online dating sites and it is beneficial for users to find their ideal relationships from a large number of registered members. Different users usually have different tastes when choosing their dating partners. Therefore, it is necessary to distinguish the users personal features and preferences in dating recommendation methods. However, present approaches dont capture enough user preferences from social graph and attribute data. They also ignore user attributes, which is the complementary and consistent side information of user social graphs. In this paper, we propose a Matching Individual Preferences with Features (MIPF) model to recommend dating partners jointly using user attributes and social graphs. We aim to model user features and preferences to identify what the user has and what the user likes. We also distinguish user preferences into explicit preferences and implicit preferences. The implicit preferences are mined from social graphs, while the explicit preferences are captured from the social links. Additionally, convolutional neural networks are used to extract the latent non-linear information in user attributes. Experiments on real-world online dating datasets demonstrate our MIPF model is superior to existing methods.

Most recommendation systems focus on predicting rating or finding aspect information in reviews to understand user preferences and item properties. However, these methods ignore the effectiveness and persuasiveness of recommendation results. Consequently, explainable recommendation, namely providing recommendation results with recommendation reasons at the same time, has attracted increasing attention of researchers due to its ability in fostering transparency and trust. It is lucky that some E-commerce websites provide a kind of new interaction box called Tips and users can express their comments on items with a simple sentence. This brings us an opportunity to realize explainable recommendation. Under the supervision of two explicit feedback, namely rating and textual tips, we can implement a multi-task learning model which can provide recommendation results and generate recommendation reasons at the same time. In this paper, we propose an Encoder-Decoder and Multi-Layer Perception (MLP) based Explainable Recommendation model named EMER to simultaneously implement reason generation and rating prediction. Items title contains significant product-related information and plays an important role in grabbing users attention, so we fuse it in our model to generate recommendation reasons. Numerous experiments on benchmark datasets demonstrate that our model is superior to the state-of-the-art models.

Object re-identification (re-ID) is one of the core technologies in Multi-Object Tracking (MOT) that requires real-time decision-making. A Neural Processing Unit (NPU) is a low-power device that is dedicated to deploying neural network-based algorithms and has become one of the most important devices in today's mobile onboard systems. However, the current mainstream re-ID methods rarely consider the NPU characteristics, which makes it difficult for these methods to achieve both high onboard frame rates and high accuracies on an NPU. To address this problem, this paper focuses on designing a re-ID algorithm suitable for NPU deployment. The model of the object re-ID can be divided into two parts: theencoder(backbone) and thedecoder. In this article, a Mobile-efficient Pure Part Model (MPPM) is presented for re-ID task. First, for thebackboneof re-ID, we propose an efficient structure GogglesNet, which is composed of traditional convolutions. GogglesNet performs well on the re-ID task and can be comparable to lightweight networks on ImageNet with regard to accuracy and is faster on NPU. We then revisit the architectures of Pure Part Model (PPM) in person re-ID, including PCB and MGN, and propose a mobile-efficientdecoderDual Pattern Network (DPN) for re-ID. The proposed MPPM achieves comparable performance with MGN on five re-ID datasets Market-1501, DukeMTMC-reID, MSMT17, VeRi-776, and VehicleID, while the proposed parameter amount is only 10.2% of it, and the speed on NPU is more than eight times higher.

Top-k recommendation is a fundamental task in recommendation systems that is generally learned by comparing positive and negative pairs. The contrastive loss (CL) is the key in contrastive learning that has recently received more attention, and we find that it is well suited for top-k recommendations. However, CL is problematic because it treats the importance of the positive and negative samples the same. On the one hand, CL faces the imbalance problem of one positive sample and many negative samples. On the other hand, there are so few positive items in sparser datasets that their importance should be emphasized. Moreover, the other important issue is that the sparse positive items are still not sufficiently utilized in recommendations. Consequently, we propose a new data augmentation method by using multiple positive items (or samples) simultaneously with the CL loss function. Therefore, we propose a multisample-based contrastive loss (MSCL) function that solves the two problems by balancing the importance of positive and negative samples and data augmentation. Based on the graph convolution network (GCN) method, experimental results demonstrate the state-of-the-art performance of MSCL. The proposed MSCL is simple and can be applied in many methods. Our code is available athttps://github.com/haotangxjtu/MSCL.

Conversational recommendation system (CRS) attracts increasing attention in various application domains such as retail and travel. It offers an effective way to capture users' dynamic preferences with multi-turn conversations. However, most current studies center on the recommendation aspect while over-simplifying the conversation process. The negligence of complexity in data structure and conversation flow hinders their practicality and utility. In reality, there exist various relationships among slots and values, while users' requirements may dynamically adjust or change. Moreover, the conversation often involves visual modality to facilitate the conversation. These actually call for a more advanced internal state representation of the dialogue and a proper reasoning scheme to guide the decision making process. In this paper, we explore multiple facets of multimodal conversational recommendation and try to address the above mentioned challenges. In particular, we represent the structured back-end database as a multimodal knowledge graph which captures the various relations and evidence in different modalities. The user preferences expressed via conversation utterances will then be gradually updated to the state graph with clear polarity. Based on these, we train an end-to-end State Graph-based Reasoning model SGR to perform reasoning over the whole state graph. The prediction of our proposed model benefits from the structure of the graph. It not only allows for zero-shot reasoning for items unseen in training conversations, but also provides a natural way to explain the policies. Extensive experiments show that our model achieves better performance compared with existing methods.

Depth estimation aims to predict depth map from RGB images without high cost equipments. Deep learning based depth estimation methods have shown their effectiveness. However in existing methods, depth information is represented by a per-pixel depth map. Such depth map representation is fragile facing different kinds of depth changes. This paper proposes a Compressive Sensing based Depth Representation (CSDR) scheme, which formulates the problem of depth estimation in pixel space into the task of fixed-length vector regression in representation space. In this way, deep model training errors will not directly interfere depth estimation, and distortions in estimated depth maps can be restrained in the greatest extent. In addition, we improve depth estimation from two other aspects: model structure and loss function. To capture the features in different scales, we propose a Multiscale Encoder \& Multiscale Decoder (MEMD) structure as the vector regression model. To further deal with depth change, we also modify the loss function, where the curvature difference between ground truth and estimation is directly incorporated. With the support of CSDR, MEMD and the curvature loss, the proposed approach achieves superior performance on a challenging depth estimation dataset: NYU-Depth-v2. A range of experiments support our claim that regression in CSDR space performs better than traditionally direct depth map estimation in pixel space.

Since recent facial landmark localization methods achieve satisfying accuracy, few of them enable fast inference speed, which, however, is critical in many real-world facial applications. Existing methods typically employ complicated network structure and predict all the key points through uniform computation, which is inefficient since individual facial part might take different computation to obtain the best performance. Taking both accuracy and efficiency into consideration, we propose the PicassoNet, a lightweight cascaded facial landmark detector with adaptive computation for individual facial part. Different from the conventional cascaded methods, PicassoNet integrates refinement submodules into a single network with group convolution, where each convolution group predicts landmarks from an individual facial part. Note that the groups’ structures are flexible in the training process. Then, a novel grouping search algorithm is proposed to optimize the group division. With formulating the optimization as a network architecture search (NAS) problem, the grouping search adaptively allocates computation to each group and obtains an efficient structure. In addition, we propose a boundary-aware loss to optimize along tangent and normal of facial boundaries, instead of optimizing along horizontal and vertical as the conventional loss (L2, SmoothL1, WingLoss, and so on) do. The novel loss improves the joint locations of predicted keypoints. Experiments on three benchmark datasets AFLW, 300W, and WFLW show that the proposed method runs over$6\times $times faster than the state of the arts and meanwhile achieves comparable accuracy.

Dialogue state is a key information in traditional task-oriented dialogue systems, which represents the user's dialogue intention at each moment through a set of (slot, value). The recent methods model the slot and the dialogue context to keep track of the state, but there is a lack of refinement of context information. They do not consider the influence of dialogue context in different scenarios. Our proposed approach utilizes a fine-grained representation of each slot at multiple levels and incorporates an interaction mechanism to obtain a weight of past memory, present utterance and relevance of the slots. Besides, to address the problem that the dialogue utterance is semantically distant from the corresponding slot value, we introduce the contrastive learning to make the utterance embedding mapped under each slot name more suitable with the ground truth value and away from other slot values. This improves the accuracy of mapping between feature space and semantic space. In the predefined ontology-based approaches, our model achieves leading results with both MultiWOZ2.0 and MultiWOZ2.1 datasets.

Video super-resolution (VSR) aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Although some progress has been made, there are grand challenges to effectively utilize temporal dependency in entire video sequences. Existing approaches usually align and aggregate video frames from limited adjacent frames (e.g., 5 or 7 frames), which prevents these approaches from satisfactory results. In this paper, we take one step further to enable effective spatio-temporal learning in videos. We propose a novel Trajectory-aware Transformer for Video Super-Resolution (TTVSR). In particular, we formulate video frames into several pre-aligned trajectories which consist of continuous visual tokens. For a query token, self-attention is only learned on relevant visual tokens along spatio-temporal trajectories. Compared with vanilla vision Transformers, such a design significantly reduces the computational cost and enables Transformers to model long-range features. We further propose a cross-scale feature tokenization module to over-come scale-changing problems that often occur in long-range videos. Experimental results demonstrate the superiority of the proposed TTVSR over state-of-the-art models, by extensive quantitative and qualitative evaluations in four widely-used video super-resolution benchmarks. Both code and pre-trained models can be downloaded at https://github.com/researchmm/TTVSR.

Recent incremental learning for action recognition usually stores representative videos to mitigate catastrophic forgetting. However, only a few bulky videos can be stored due to the limited memory. To address this problem, we propose FrameMaker, a memory-efficient video class-incremental learning approach that learns to produce a condensed frame for each selected video. Specifically, FrameMaker is mainly composed of two crucial components: Frame Condensing and Instance-Specific Prompt. The former is to reduce the memory cost by preserving only one condensed frame instead of the whole video, while the latter aims to compensate the lost spatio-temporal details in the Frame Condensing stage. By this means, FrameMaker enables a remarkable reduction in memory but keep enough information that can be applied to following incremental tasks. Experimental results on multiple challenging benchmarks, i.e., HMDB51, UCF101 and Something-Something V2, demonstrate that FrameMaker can achieve better performance to recent advanced methods while consuming only 20% memory. Additionally, under the same memory consumption conditions, FrameMaker significantly outperforms existing state-of-the-arts by a convincing margin.

Emoji recommendation is an important task to help users find appropriate emojis from thousands of candidates based on a short tweet text. Traditional emoji recommendation methods lack personalized recommendation and ignore user historical information in selecting emojis. In this paper, we propose a personalized emoji recommendation with dynamic user preference (PERD) which contains a text encoder and a personalized attention mechanism. In text encoder, a BERT model is contained to learn dense and low-dimensional representations of tweets. In personalized attention, user dynamic preferences are learned according to semantic and sentimental similarity between historical tweets and the tweet which is waiting for emoji recommendation. Informative historical tweets are selected and highlighted. Experiments are carried out on two real-world datasets from Sina Weibo and Twitter. Experimental results validate the superiority of our approach on personalized emoji recommendation.

Occluded person re-identification is one of the most challenging tasks in security surveillance. Most existing methods focus on extracting human body features from occluded pedestrian images. This paper prioritizes a difference between occluded and non-occluded person re-ID: When computing the similarity between a holistic pedestrian image and an occluded pedestrian image, a certain part of the human body in this holistic image can be distractive for pedestrian retrieval. To solve this problem, we propose an occluded person re-ID framework named attribute-based shift attention network (ASAN). First, unlike other methods that use off-the-shelf tools to locate pedestrian body parts in the occluded images, we design an attribute-guided occlusion-sensitive pedestrian segmentation (AOPS) module. AOPS is a weakly supervised method that leverages the semantic-level attribute annotations in person re-ID datasets. Second, guided by the pedestrian masks provided by AOPS, a shift feature adaption (SFA) module extracts the visible part of the human body feature in a part-based manner. After that, a visible region matching (VRM) algorithm is proposed to filter out the interfer-ence information in the holistic person images during the retrieval phase and further purify the representation of pedestrian features. Extensive experiments with ablation analysis demonstrate our method’s effectiveness. And the state-of-the-art results are achieved on four occluded datasets Partial-REID, Partial-iLIDS, Occluded-DukeMTMC, and Occluded REID. Moreover, the experiments on two holistic person re-ID datasets Market-1501 and DukeMTMC-reID, and a vehicle re-ID dataset VeRi-776 show that ASAN also has a good generality.

Face detection generally requires prior boxes and an extra non-maximum suppression(NMS) post-processing in modern deep learning methods. However, anchor design and anchor matching strategy significantly affect the performance of face detectors, so we have to spend a lot of time on anchor designing for different business scenarios. The other issue is that NMS cannot be easily parallelized and it may become a bottleneck of detection speed. In this paper, we propose a simple yet efficient pure convolutional neural network face detection method, named dual-branch center face detector(DBCFace for short), which solve face detection via a dual branch fully convolutional framework without extra anchor design and NMS. Extensive experiments are conducted on four popular face detection benchmarks, including AFW, PASCAL face, FDDB, and WIDER FACE, demonstrating that our method is comparable with state-of-the-art methods while the speed is faster.

Next POI recommendation has been studied extensively in recent years. The goal is to recommend next POI for users at specific time given users’ historical check-in data. Therefore, it is crucial to model both users’ general taste and recent sequential behaviors. Moreover, different users show different dependencies on the two parts. However, most existing methods learn the same dependencies for different users. Besides, the locations and categories of POIs contain different information about users’ preference. However, current researchers always treat them as the same factors or believe that categories determine where to go. To this end, we propose a novel method named Personalized Long- and Short-term Preference Learning (PLSPL) to learn the specific preference for each user. Specially, we combine the long- and short-term preference via user-based linear combination unit to learn the personalized weights on different parts for different users. Besides, the context information such as the category and check-in time is also essential to capture users’ preference. Therefore, in long-term module, we consider the contextual features of POIs in users’ history records and leverage attention mechanism to capture users’ preference. In the short-term module, to better learn the different influences of locations and categories of POIs, we train two LSTM models for location- and category-based sequence, respectively. Then we evaluate the proposed model on two real-world datasets. The experiment results demonstrate that our method outperforms the state-of-art approaches for next POI recommendation.

The number of mitotic cells present in histopathological slides is an important predictor of tumor proliferation in the diagnosis of breast cancer. However, the current approaches can hardly perform precise pixel-level prediction for mitosis datasets with only weak labels (i.e., only provide the centroid location of mitotic cells), and take no account of the large domain gap across histopathological slides from different pathology laboratories. In this work, we propose a Domain adaptive Box-supervised Instance segmentation Network (DBIN) to address the above issues. In DBIN, we propose a high-performance Box-supervised Instance-Aware (BIA) head with the core idea of redesigning three box-supervised mask loss terms. Furthermore, we add a Pseudo-Mask-supervised Semantic (PMS) head for enriching characteristics extracted from underlying feature maps. Besides, we align the pixel-level feature distributions between source and target domains by a Cross-Domain Adaptive Module (CDAM), so as to adapt the detector learned from one lab can work well on unlabeled data from another lab. The proposed method achieves state-of-the-art performance across four mainstream datasets. A series of analysis and experiments show that our proposed BIA and PMS head can accomplish mitosis pixel-wise localization under weak supervision, and we can boost the generalization ability of our model by CDAM.

Sequential recommendations aim to predict the user’s next behaviors items based on their successive historical behaviors sequence. It has been widely applied in lots of online services. However, current sequential recommendations use the adjacent behaviors to capture the features of the sequence, ignoring the features among nonadjacent sequential items and the summarized features of the sequence. To address the above problems, in this paper, we propose an annular-graph attention based sequential recommendation (AGSR) model by exploring user’s long-term and short-term preferences for the personalized sequential recommendation. For user’s short-term preferences, AGSR builds an annular-graph on the sequence of user behavior. Then, AGSR proposes an annular-graph attention applying on the sub annular-graph to explore local features and applying annular-graph attention on entire annular-graph to explore the global features and the skip features. For user’s long-term preferences, the latent factor model are introduced in AGSR. The experimental results on two public datasets show that our model outperforms the state-of-the-art methods.

Neural-network Processing Units (NPU), which specializes in the acceleration of deep neural networks (DNN), is of great significance to latency-sensitive areas like robotics or edge computing. However, there are few works focusing on the network design for NPU in recent studies. Most of the popular lightweight structures (e.g. MobileNet) are designed with depthwise convolution, which has less computation in theory but is not friendly to existing hardwares, and the speed tested on NPU is not always satisfactory. Even under similar FLOPs (the number of multiply-accumulates), vanilla convolution operation is always faster than depthwise one. In this paper, we will propose a novel architecture named Serial and Parallel Group Network (SPGNet), which can capture discriminative multi-scale information and at the same time keep the structure compact. Extensive evaluations have been conducted on different computer vision tasks, e.g. image classification (CIFAR and ImageNet), object detection (PASCAL VOC and MS COCO) and person re-identification (Market-1501 and DukeMTMC-ReID). The experimental results show that our proposed SPGNet can achieve comparable performance with the state-of-the-art networks while the speed is 120% faster than MobileNetV2 under similar FLOPS and over 300% faster than GhostNet with similar accuracy on NPU.

Recently, some approaches are proposed to harness deep convolutional networks to facilitate superpixel segmentation. The common practice is to first evenly divide the image into a pre-defined number of grids and then learn to associate each pixel with its surrounding grids. However, simply applying a series of convolution operations with limited receptive fields can only implicitly perceive the relations between the pixel and its surrounding grids. Consequently, existing methods often fail to provide an effective context when inferring the association map. To remedy this issue, we propose a novel Association Implantation (AI) module to enable the network to explicitly capture the relations between the pixel and its surrounding grids. The proposed AI module directly implants the grid features to the surrounding of its corresponding central pixel, and conducts convolution on the padded window to adaptively transfer knowledge between them. With such an implantation operation, the network could explicitly harvest the pixel-grid level context, which is more in line with the target of superpixel segmentation comparing to the pixelwise relation. Furthermore, to pursue better boundary precision, we design a boundary-perceiving loss to help the network discriminate the pixels around boundaries in hidden feature level, which could benefit the subsequent inferring modules to accurately identify more boundary pixels. Extensive experiments on BSDS500 and NYUv2 datasets show that our method could achieve state-of-the-art performance. Code and pre-trained model are available at https://github.com/wangyxxjtu/AINet-ICCV2021.

Abstract Image search re-ranking is one of the most important approaches to enhance the text-based image search results. Extensive efforts have been dedicated to improve the accuracy and diversity of tag-based image retrieval. However, how to make the top-ranked results relevant and diverse is still a challenging problem. In this paper, we propose a novel method to diversify the retrieval results by latent topic analysis. We first employNMF(Non-negative Matrix Factorization) Lee and Seung (Nature 401(6755):788–791, 1999) to estimate the initial relevance score to the queryq. Then, the initial relevance score is fed into an adaptive multi-feature fusion model to learn the final relevance score. Next, the diversification process is conducted. We group all the images by semantic clustering and estimate the topic distribution of each cluster by topic analysis. The clusters are ranked based on the topic distribution vector and the final retrieval image list is obtained by a greedy selection mechanism based on the estimated relevances. Experimental results on the NUS-Wide dataset show the effectiveness of the proposed approach.

Counting passengers in railway compartments is an essential problem for improving service quality, user experience, public security, and disaster relief in the railway system. Considering many limitations in the compartment, the infrared sensor, 3D camera, etc. are not practical in this scene. Due to the flexibility and lower cost, solutions with standard cameras attract much attention in real applications. However, since the problem with scale variation in the narrow space is different from universal detection or counting problems, the specific benchmark of dataset and methods should be provided and proposed for this task. In this paper, we provide a passenger counting dataset. Relying on this dataset, we propose a passenger counting method. The solution contains a motion supervised multi-scale representation method which provides proposals against scale variation, a spatially-temporally enhanced counting which provides precise counting numbers, and a partial proposal method which conducts methods to be utilized in reality. With the proposed solution, the passengers counting task is solved in higher accuracy and practicable in the compartment environment. In experiments, the results show that all the modules in our solution are useful and efficient, and our method outperforms in comparison with others in the compartment scene.

Fine-grained food recognition is the detailed classification that provides more specialized and professional attribute information of food. It is the basic work to realize healthy diet recommendations and cooking instructions, nutrition intake management, and cafeteria self-checkout system. Chinese food lacks structured information, and ingredients composition is an important consideration. The current approaches mostly focus on global dish appearance without any analysis of ingredient composition and fully considering the attention of regional features. In this paper, we propose an Attention Fusion Network (AFN) and Food-Ingredient Joint Learning module for fine-grained food and ingredients recognition. The AFN first focuses on the food discrimination region against unstructured defeat and generates the feature embeddings jointly aware of the ingredients and food. The Food-Ingredient Joint Learning module aims at alleviating the issue of ingredients imbalance. Therefore, we propose a balance focal loss to optimize the feature expression ability of the network for ingredients. In experiments, the results of ingredients recognition show the state-of-the-art performances on fine-grained Chinese food dataset VIREO Food-172.

Deep supervised hashing has emerged as an influential solution to large-scale semantic image retrieval problems in computer vision. In the light of recent progress, image label is the common way to define whether two images belong to the same category, but it contains little supervised information. The one-hot label can't accurately define the similarity of two images, which is important for image retrieval. In this paper, we propose an effective method, Deep Transfer Hashing(DTH) which uses the knowledge from teacher model as the supervised information. Inspired by knowledge distillation for model compression and deep hashing for fast image retrieval, we transfer the knowledge from a complex convolutional neural network(teacher) to a small neural network(student) which is used for fast image retrieval. The distance of the knowledge from teacher model can indicate the similarity of images. By minimizing the hashing codes distribution between the hashing layers of teacher model and student model, we can improve the retrieval performance. And we also evaluate the performance of the compressed model at inference stage. We test our method on widely used datasets CIFAR-10 and NUS-WIDE and we compare our method with other state-of-the-art methods in image retrieval domain. The experimental results show that our method can improve the image retrieval baseline by a large margin and better than other methods.

The outpainting results produced by existing approaches are often too random to meet users' requirements. In this work, we take the image outpainting one step forward by allowing users to harvest personal custom outpainting results using sketches as the guidance. To this end, we propose an encoder-decoder based network to conduct sketch-guided outpainting, where two alignment modules are adopted to impose the generated content to be realistic and consistent with the provided sketches. First, we apply a holistic alignment module to make the synthesized part be similar to the real one from the global view. Second, we reversely produce the sketches from the synthesized part and encourage them be consistent with the ground-truth ones using a sketch alignment module. In this way, the learned generator will be imposed to pay more attention to fine details and be sensitive to the guiding sketches. To our knowledge, this work is the first attempt to explore the challenging yet meaningful conditional scenery image outpainting. We conduct extensive experiments on two collected benchmarks to qualitatively and quantitatively validate the effectiveness of our approach compared with the other state-of-the-art generative models.

Crowd scene analysis receives growing attention due to its wide applications. Grasping the accurate crowd location is important for identifying high-risk regions. In this article, we propose a Compressed Sensing based Output Encoding (CSOE) scheme, which casts detecting pixel coordinates of small objects into a task of signal regression in encoding signal space. To prevent gradient vanishing, we derive our own sparse reconstruction backpropagation rule that is adaptive to distinct implementations of sparse reconstruction and makes the whole model end-to-end trainable. With the support of CSOE and the backpropagation rule, the proposed method shows more robustness to deep model training error, which is especially harmful to crowd counting and localization. The proposed method achieves state-of-the-art performance across four mainstream datasets, especially achieves excellent results in highly crowded scenes. A series of analysis and experiments support our claim that regression in CSOE space is better than traditionally detecting coordinates of small objects in pixel space for highly crowded scenes.

With the popularity of social platforms, emoji appears and becomes extremely popular with a large number of users. It expresses more beyond plaintexts and makes the content more vivid. Using appropriate emojis in messages and microblog posts makes you lovely and friendly. Recently, emoji recommendation becomes a significant task since it is hard to choose the appropriate one from thousands of emoji candidates. In this paper, we propose a Context-Aware Personalized Emoji Recommendation (CAPER) model fusing the contextual information and the personal information. It is to learn latent factors of contextual and personal information through a score-ranking matrix factorization framework. The personal factors such as user preference, user gender, and the current time can make the recommended emojis meet users' individual needs. Moreover, we consider the co-occurrence factors of the emojis which could improve the recommendation accuracy. We conduct a series of experiments on the real-world datasets, and experiment results show better performance of our model than existing methods, demonstrating the effectiveness of the considering contextual and personal factors.

Viewpoint recommendation can recommend several viewpoints for taking aesthetic photographs of a place-of-interest (POI) and is of great importance for photography assistance. In this paper, we propose a system that can assist a user in choosing good viewpoints for taking high-quality photographs. Our system is based on social media and 3D reconstruction. To reduce the time cost and improve the quality of 3D reconstruction, we propose a weakly supervised object detection method that is used before 3D reconstruction. The camera pose of images is recovered by the subsequent 3D reconstruction pipeline. We use a convolutional neural network (CNN) to extract 2D image features, and we fuse them with 3D camera pose features to learn their relationships to image aesthetics. The trained model is utilized to evaluate the aesthetics of images. Finally, the 3D space of all possible camera poses is divided into 3D grids, and the aesthetics score of each grid is evaluated. We combine the aesthetics and diversity of all viewpoints and recommend several high-quality viewpoints. Experimental results indicate that our approach can help users choose viewpoints that will result in high-quality photographs while maintaining diversity.

When planning a trip, users tend to browse Place-of-Interest (POI) information on the Internet and then depart. Many works aimed at summarizing POIs by visual and textual analysis, while many of them ignored the inter-relationship between different views offered by the community-contributed information. In this paper, we propose a City-POI-LOI (CPL) summarization method to automatically mine POIs from the city-level landmark images. And a Location-Appearance-Semantic-Temporal (LAST) clustering method is proposed to mine the popular viewpoints termed Location-Of-Interest (LOI) in each POI by taking location, appearance, semantic, and temporal feature into consideration. We perform text and image summarization for each LOI, and we further summarize the POIs based on season. We conduct a series of experiments based on DIV400 and ATCF Dataset. Experimental results show the effectiveness of the proposed POI summarization approach.

Bi-directional image-text retrieval and matching attract much attention recently. This cross-domain task demands a fine understanding of both modalities for learning a measure of different modality data. In this paper, we propose a novel position focused attention network to investigate the relation between the visual and the textual views. This work integrates the prior object position to enhance the visual-text joint-embedding learning. The image is first split into blocks, which are treated as the basic position cells, and the position of an image region is inferred. Then, we propose a position attention to model the relations between the image region and position cells. Finally, we generate a valuable position feature to further enhance the region expression and model a more reliable relationship between the visual image and the textual sentence. Experiments on the popular datasets Flickr30K and MS-COCO show the effectiveness of the proposed method. Besides the public datasets, we also conduct experiments on our collected practical large-scale news dataset (Tencent-News) to validate the practical application value of the proposed method. As far as we know, this is the first attempt to test the performance on the practical application. Our method achieves the competitive performance on all of these three datasets.

Learning an efficient news representation is a fundamental yet important problem for many tasks. Most existing news-relevant methods only take the textual information while abandoning the visual clues from the illustrations. We argue that the textual title and tags together with the visual illustrations form the main force of a piece of news and are more efficient to express the news content. In this paper, we develop a novel framework, namely Semantic Gated Network (SGN), to integrate the news title, tags and visual illustrations to obtain an efficient joint textual-visual feature for the news, by which we can directly measure the relevance between two pieces of news. Particularly, we first harvest the tag embeddings by the proposed self-supervised classification model. Besides, news title is fed into a sentence encoder pretrained by two semantically relevant news to learn efficient contextualized word vectors. Then the feature of the news title is extracted based on the learned vectors and we combine it with features of tags to obtain textual feature. Finally, we design a novel mechanism named semantic gate to adaptively fuse the textual feature and the image feature. Extensive experiments on benchmark dataset demonstrate the effectiveness of our approach.

Compressive sensing (CS)-based image coding scheme has been enthusiastically studied, but it still has a poor rate-distortion performance compared with the traditional image coding techniques. In this paper, we propose a CS multi-layer residual coding scheme to rectify this problem to a certain extent. By dividing CS measurements into multi-layers and predicting a particular layer's measurements with all its preceding layers' measurements, we can transform CS measurements into multi-layer residual coefficients, which are easier to compress. By calculating the residual between the quantized ground-truth CS measurements and their corresponding quantized inference measurements and using Huffman coding to associate each residual quantization index with a binary code, we can reduce the redundancies among CS measurements efficiently. Besides, the prediction and quantization process is designed to be layer-independent, which can save much of the encoding time. The proposed approach introduces a novel framework for using CS in the compression domain. The experimental results show that the proposed scheme can significantly outperform JPEG2000 and approach or reach the performance of HEVC-Intra on some test images.

To improve the performance of sketch-based image retrieval (SBIR) methods, most existing SBIR methods develop brand new SBIR methods. In fact, a re-ranking approach, which can refine the retrieval results of SBIR methods, is also beneficial. Inspired by this, in this paper, an SBIR re-ranking approach based on multi-clustering is proposed. In order to make the re-ranking approach invisible to users and adaptive to different types of image datasets, we made it an unsupervised method using blind feedback. Distinguished from the existing methods, this re-ranking approach uses the semantic information of three types of images: edge maps, object images (images with black background and natural images' foreground objects) and natural images themselves. With the initial retrieval results of an SBIR method, our approach first does the clustering operation for three types of images. Then, we utilize the clustering results to generate a cluster score for each initial retrieval result. Finally, the cluster score is used to calculate the final retrieval scores for the initial retrieval results. The experiments on different SBIR datasets are conducted. Experimental results demonstrate that, by implementing our re-ranking approach, the retrieval accuracy of a variety of SBIR methods is increased. Furthermore, the comparisons between our re-ranking method and the existing re-ranking methods are given.

This paper introduces a convolutional neural network (CNN) semantic re-ranking system to enhance the performance of sketch-based image retrieval (SBIR). Distinguished from the existing approaches, the proposed system can leverage category information brought by CNNs to support effective similarity measurement between the images. To achieve effective classification of query sketches and high-quality initial retrieval results, one CNN model is trained for classification of sketches, another for that of natural images. Through training dual CNN models, the semantic information of both the sketches and natural images is captured by deep learning. In order to measure the category similarity between images, a category similarity measurement method is proposed. Category information is then used for re-ranking. Re-ranking operation first infers the retrieval category of the query sketch and then uses the category similarity measurement to measure the category similarity between the query sketch and each initial retrieval result. Finally, the initial retrieval results are re-ranked. The experiments on different types of SBIR datasets demonstrate the effectiveness of the proposed re-ranking method. Comparisons with other re-ranking algorithms are also given to show the proposed method's superiority. Further, compared to the baseline systems, the proposed re-ranking approach achieves significantly higher precision in the top ten different SBIR methods and datasets.

Effective location recommendation is an important problem in both research and industry. Much research has focused on personalized recommendation for users. However, there are more uses such as site selection for firms and factories. In this study, we try to solve site selection problem by recommending some locations satisfying special requirements. There are many factors affecting it, including functions of architecture, building cost, pollution discharge etc. We focus on the specific site selection of meteorological observation stations in this paper with leveraging the factors of functions of architecture and building cost from multi-source urban big data. We consider not only recommending the locations that can provide more accurate prediction and cover more areas, but also minimizing the cost of building new stations. We design an extensible two-stage framework for the station placing including prediction model and recommendation model. It is very convenient for executives to add more real-life factors into our approach. We have some empirical findings and evaluate the proposed approach using the real meteorological data of Shaanxi province, China. Experiment results show the better performance of our approach than existing commonly used methods.

Next POI recommendation has been studied extensively in recent years. The goal is to recommend next POI for users at specific time given users' historical check-in data. Therefore, it is crucial to model users' general taste and recent sequential behavior. Moreover, the context information such as the category and check-in time is also important to capture user preference. To this end, we propose a long- and short-term preference learning model (LSPL) considering the sequential and context information. In long-term module, we learn the contextual features of POIs and leverage attention mechanism to capture users' preference. In the short-term module, we utilize LSTM to learn the sequential behavior of users. Specifically, to better learn the different influence of location and category of POIs, we train two LSTM models for location-based sequence and category-based sequence, respectively. Then we combine the long and short-term results to recommend next POI for users. At last, we evaluate the proposed model on two real-world datasets. The experiment results demonstrate that our method outperforms the state-of-art approaches for next POI recommendation.

Relationships encode the interactions among individual instances and play a critical role in deep visual scene understanding. Suffering from the high predictability with non-visual information, relationship models tend to fit the statistical bias rather than ``learning" to infer the relationships from images. To encourage further development in visual relationships, we propose a novel method to mine more valuable relationships by automatically pruning visually-irrelevant relationships. We construct a new scene graph dataset named Visually-Relevant Relationships Dataset (VrR-VG) based on Visual Genome. Compared with existing datasets, the performance gap between learnable and statistical method is more significant in VrR-VG, and frequency-based analysis does not work anymore. Moreover, we propose to learn a relationship-aware representation by jointly considering instances, attributes and relationships. By applying the representation-aware feature learned on VrR-VG, the performances of image captioning and visual question answering are systematically improved, which demonstrates the effectiveness of both our dataset and features embedding schema. Both our VrR-VG dataset and representation-aware features will be made publicly available soon.

Image-text matching tasks have recently attracted a lot of attention in the computer vision field. The key point of this cross-domain problem is how to accurately measure the similarity between the visual and the textual contents, which demands a fine understanding of both modalities. In this paper, we propose a novel position focused attention network (PFAN) to investigate the relation between the visual and the textual views. In this work, we integrate the object position clue to enhance the visual-text joint-embedding learning. We first split the images into blocks, by which we infer the relative position of region in the image. Then, an attention mechanism is proposed to model the relations between the image region and blocks and generate the valuable position feature, which will be further utilized to enhance the region expression and model a more reliable relationship between the visual image and the textual sentence. Experiments on the popular datasets Flickr30K and MS-COCO show the effectiveness of the proposed method. Besides the public datasets, we also conduct experiments on our collected practical news dataset (Tencent-News) to validate the practical application value of proposed method. As far as we know, this is the first attempt to test the performance on the practical application. Our method can achieve the state-of-art performance on all of these three datasets.

Personalized recommendation has received a lot of attention as a highly practical research topic. However, existing recommender systems provide the recommendations with a generic statement such as “Customers who bought this item also bought…”. Explainable recommendation, which makes a user aware of why such items are recommended, is in demand. The goal of our research is to make the users feel as if they are receiving recommendations from their friends. To this end, we formulate a new challenging problem called personalized reason generation for explainable recommendation for songs in conversation applications and propose a solution that generates a natural language explanation of the reason for recommending a song to that particular user. For example, if the user is a student, our method can generate an output such as “Campus radio plays this song at noon every day, and I think it sounds wonderful,” which the student may find easy to relate to. In the offline experiments, through manual assessments, the gain of our method is statistically significant on the relevance to songs and personalization to users comparing with baselines. Large-scale online experiments show that our method outperforms manually selected reasons by 8.2% in terms of click-through rate. Evaluation results indicate that our generated reasons are relevant to songs and personalized to users, and they attract users to click the recommendations.

In recent years, content-based image retrieval has been of concern because of practical needs on Internet services, especially methods that can improve retrieving speed and accuracy. The SIFT feature is a well-designed local feature. It has mature applications in feature matching and retrieval, whereas the raw SIFT feature is high dimensional, with high storage cost as well as computational cost in feature similarity measurements. Thus, we propose a hashing scheme for fast SIFT feature-based image matching and retrieval. First, a training process of the hashing function involves geometric and topological information being introduced; second, a geometry-enhanced similarity evaluation that considers both the global and details of images in evaluation is explained. Compared with state-of-the-art methods, our method achieves better performance.

In recent years, we have witnessed a flourish of social review websites. Internet users can easily share their experiences on some products and services with their friends. Therefore, measuring interpersonal influence becomes a popular method for recommender systems. However, traditional works are all based on external tangible activities, such as following, retweeting, mentioning, etc. In this paper, we explore user internal factors to measure his/her influence on a specific domain, namely, the social network on local businesses. The proposed user internal factors include user sentimental deviations and the review's reliability. The internal factors are not from explicit behavior but could help us to understand users. In addition, we utilize an attention mechanism that could auto-learn the weights of different factors. Through a case study on the Yelp dataset, we found that the proposed user internal factors on influence, that is, the proposed user sentimental deviations and the review's reliability, are effective in improving the accuracy of rating predictions.

Aggregating deep convolutional features into a global image vector has attracted sustained attention in image retrieval. In this paper, we propose an efficient unsupervised aggregation method that uses an adaptive Gaussian filter and an element-value sensitive vector to co-weight deep features. Specifically, the Gaussian filter assigns large weights to features of region-of-interests (RoI) by adaptively determining the RoI's center, while the element-value sensitive channel vector suppresses burstiness phenomenon by assigning small weights to feature maps with large sum values of all locations. Experimental results on benchmark datasets validate the proposed two weighting schemes both effectively improve the discrimination power of image vectors. Furthermore, with the same experimental setting, our method outperforms other very recent aggregation approaches by a considerable margin.

Place-of-Interest (POI) summarization by aesthetics evaluation can recommend a set of POI images to the user and it is significant in image retrieval. In this paper, we propose a system that summarizes a collection of POI images regarding both aesthetics and diversity of the distribution of cameras. First, we generate visual albums by a coarse-to-fine POI clustering approach and then generate 3D models for each album by the collected images from social media. Second, based on the 3D to 2D projection relationship, we select candidate photos in terms of the proposed crowd source saliency model. Third, in order to improve the performance of aesthetic measurement model, we propose a crowd-sourced saliency detection approach by exploring the distribution of salient regions in the 3D model. Then, we measure the composition aesthetics of each image and we explore crowd source salient feature to yield saliency map, based on which, we propose an adaptive image adoption approach. Finally, we combine the diversity and the aesthetics to recommend aesthetic pictures. Experimental results show that the proposed POI summarization approach can return images with diverse camera distributions and aesthetics.

As the image sharing websites like Flickr become more and more popular, extensive scholars concentrate on tag-based image retrieval. It is one of the important ways to find images contributed by social users. In this research field, tag information and diverse visual features have been investigated. However, most existing methods use these visual features separately or sequentially. In this paper, we propose a global and local visual features fusion approach to learn the relevance of images by hypergraph approach. A hypergraph is constructed first by utilizing global, local visual features, and tag information. Then, we propose a pseudo-relevance feedback mechanism to obtain the pseudo-positive images. Finally, with the hypergraph and pseudo relevance feedback, we adopt the hypergraph learning algorithm to calculate the relevance score of each image to the query. Experimental results demonstrate the effectiveness of the proposed approach.

Image transmission in a wireless visual sensor network (WVSN) with limited resources over an unreliable and bandwidth-limited wireless channel is challenging. This paper presents a highly efficient and robust image coding and transmission scheme with a simple encoder based on compressive sensing (CS) for WVSNs. First, an image measurement based on scrambled block compressive sampling with a separable sensing operator is proposed to simplify the encoder. Second, a progressive nonuniform quantization, which exploits the measurement distribution at the encoder side and the measurement dependencies at the decoder side, is designed to improve the rate-distortion (R-D) performance while maintaining low complexity at the encoder. Third, to further improve the R-D performance, a progressive non-local low-rank reconstruction is designed at the decoder. The experimental results show that the proposed scheme can achieve higher R-D performance compared with the benchmark CS-based image coding and transmission schemes. Higher robustness can be achieved compared with the traditional source-channel coding, such as Consultative Committee for Space Data Systems$-$Image Data Compression (CCSDS-IDC) with Raptor codes under a time-varying packet loss channel, and the encoding time can be significantly reduced compared with the traditional image coding schemes. The experimental results also show that the proposed scheme achieves state-of-the-art coding efficiency with lower computational complexity at the encoder while still supporting error resilience.

Mobile videos on smartphones have been widely used and enjoyed by increasing numbers of people in recent years; however, due to the high bit rates of video streams, viewers must pay high data communication costs when watching videos over cellular networks. Thus, the viewers' quality of experience (QoE) for streaming videos is influenced by their psychological states under this cost pressure. This paper evaluates the cost-constrained video quality. First, a cost-constrained QoE framework that considers technical factors, content types and cost pressure is developed. Then, a cost-constrained video quality satisfaction (CVQS) model is proposed to assess users' video quality satisfaction (QS) considering the three aspects. Finally, the proposed model is verified by empirical analysis, mathematical fitting and experimental simulations. The proposed CVQS model maximizes the QS of videos by considering the cost to users, the users' personality profiles and objective video quality. An example application that employs the CVQS model is provided to demonstrate the applicability and effectiveness of the model.

Visual saliency detection has become an active research direction in recent years. A large number of saliency models, which can automatically locate objects of interest in images, have been developed. As these models take advantage of different kinds of prior assumptions, image features, and computational methodologies, they have their own strengths and weaknesses and may cope with only one or a few types of images well. Inspired by these facts, this paper proposes a novel salient object detection approach with the idea of inferring a superior model from a variety of previous imperfect saliency models via optimally leveraging the complementary information among them. The proposed approach mainly consists of three steps. First, a number of existing unsupervised saliency models are adopted to provide weak/imperfect saliency predictions for each region in the image. Then, a fusion strategy is used to fuse each image region's weak saliency predictions into a strong one by simultaneously considering the performance differences among various weak predictions and various characteristics of different image regions. Finally, a local spatial consistency constraint that ensures high similarity of the saliency labels for neighboring image regions with similar features is proposed to refine the results. Comprehensive experiments on five public benchmark datasets and comparisons with a number of state-of-the-art approaches can demonstrate the effectiveness of the proposed work.

The ability to recognize a vehicle and estimate its pose is important for to a wide range of practical scenarios. Recently, it has been shown that several vehicle components could be a discriminative visual pattern during recognition and detection. To detect these components, our observation is that it is more reliable to build probabilistic prediction maps rather than simply relying on region proposals. Another key insight here is that the spatial relationship between these vehicle components shows some consistent patterns, and is able to provide strong semantic cues for vehicle pose estimation. In this paper, we propose a supervised learning framework with Fully Convolutional Network (FCN) to build probabilistic prediction maps of vehicle components, then vehicle pose is estimated based on the absence/presence and the spatial relationship (e.g. the relative distance, scales, angles) of these components. The proposed method is evaluated with three state-of-the-art approaches on public vehicle image datasets and achieve superior performance.

With fast development of information engineering and social network, people's locations can be conveniently sensed by spatial technology, such as global positioning systems (GPS), base stations, Wi-Fi access points and even from the appearances of the photos they have taken. The social networks and the online shopping platforms have been gathering billions of users, who share a large amount of images taken in places they live in and visit. We can leverage the social networks to express our opinions about the services and places of interest (POIs). The interactions among users, and user and POIs or services generate big social media data, which have rich information for user, location, and service cognition. Many real-time network applications rely heavily on the accurate social users' locations. How to sense the locations from multisource social media data is very important and challenging. Thus, in this paper, we give a systematic review of the works that combine social media and spatial technology for POI cognition and image localization.

Recently, advances in intelligent mobile device and positioning techniques have fundamentally enhanced social networks, which allows users to share their experiences, reviews, ratings, photos, check-ins, etc. The geographical information located by smart phone bridges the gap between physical and digital worlds. Location data functions as the connection between user's physical behaviors and virtual social networks structured by the smart phone or web services. We refer to these social networks involving geographical information as location-based social networks (LBSNs). Such information brings opportunities and challenges for recommender systems to solve the cold start, sparsity problem of datasets and rating prediction. In this paper, we make full use of the mobile users' location sensitive characteristics to carry out rating prediction. We mine: 1) the relevance between user's ratings and user-item geographical location distances, called as user-item geographical connection, 2) the relevance between users' rating differences and user-user geographical location distances, called as user-user geographical connection. It is discovered that humans' rating behaviors are affected by geographical location significantly. Moreover, three factors: user-item geographical connection, user-user geographical connection, and interpersonal interest similarity, are fused into a unified rating prediction model. We conduct a series of experiments on a real social rating network dataset Yelp. Experimental results demonstrate that the proposed approach outperforms existing models.

Social media sharing Websites allow users to annotate images with free tags, which significantly contribute to the development of the web image retrieval. Tag-based image search is an important method to find images shared by users in social networks. However, how to make the top ranked result relevant and with diversity is challenging. In this paper, we propose a topic diverse ranking approach for tag-based image retrieval with the consideration of promoting the topic coverage performance. First, we construct a tag graph based on the similarity between each tag. Then, the community detection method is conducted to mine the topic community of each tag. After that, inter-community and intra-community ranking are introduced to obtain the final retrieved results. In the inter-community ranking process, an adaptive random walk model is employed to rank the community based on the multi-information of each topic community. Besides, we build an inverted index structure for images to accelerate the searching process. Experimental results on Flickr data set and NUS-Wide data sets show the effectiveness of the proposed approach.

Locations of images have been widely used in many application scenarios for large geotagged image corpora. As to images that are not geographically tagged, we estimate their locations with the help of the large geotagged image set by content-based image retrieval. Bag-of-words image representation has been utilized widely. However, the individual visual word-based image retrieval approach is not effective in expressing the salient relationships of image region. In this paper, we present an image location estimation approach by multisaliency enhancement. We first extract region-of-interests (ROIs) by mean-shift clustering on the visual words and salient map of the image based on which we further determine the importance of the ROI. Then, we describe each ROI by the spatial descriptors of visual words. Finally, region-based visual phrases are generated to further enhance the saliency in image location estimation. Experiments show the effectiveness of our proposed approach.

Big data increasingly benefit both research and industrial area such as health care, finance service and commercial recommendation. This paper presents a personalized travel sequence recommendation from both travelogues and community contributed photos and the heterogeneous metadata (e.g., tags, geo-location, and date taken) associated with these photos. Unlike most existing travel recommendation approaches, our approach is not only personalized to user's travel interest but also able to recommend a travel sequence rather than individual Points of Interest (POIs). Topical package space including representative tags, the distributions of cost, visiting time and visiting season of each topic, is mined to bridge the vocabulary gap between user travel preference and travel routes. We take advantage of the complementary of two kinds of social media: travelogue and community contributed photos. We map both user's and routes' textual descriptions to the topical package space to get user topical package model and route topical package model (i.e., topical interest, cost, time and season). To recommend personalized POI sequence, first, famous routes are ranked according to the similarity between user package and route package. Then top ranked routes are further optimized by social similar users' travel records. Representative images with viewpoint and seasonal diversity of POIs are shown to offer a more comprehensive impression. We evaluate our recommendation system on a collection of 7 million Flickr images uploaded by 7,387 users and 24,008 travelogues covering 864 travel POIs in nine famous cities, and show its effectiveness. We also contribute a new dataset with more than 200 K photos with heterogeneous metadata in nine famous cities.

Due to the popularity of online video sharing Web sites such as YouTube, millions of users have treated online video as a source of information and entertainment. Therefore, video annotation has evoked great interest in the past few years. In this paper, we propose a four-step approach to automatically annotate video shots with sentences. The first step is video preprocessing, converting video shot into a sequence of frame images. The second step is to find related candidate elements of the sentence about the video contents. The main elements in the sentence are objects, events, scenes, and modifiers. These candidate elements are gained by searching for similar images with the video frames in our collected image data sets instead of video data sets. The third step is to select the best elements among these candidate ones by a weighted scoring algorithm. The final step is to construct a sentence with the help of a correlation graph algorithm to analyze the relationships among the best elements. The experimental results indicate that our method is effective to annotate videos with sentences. What is more, the weighted scoring algorithm and the correlation graph algorithm that we propose are efficient in developing the experimental performance.

In this paper, we focus on tackling the problem of automatic semantic annotation of high resolution (HR) optical satellite images, which aims to assign one or several predefined semantic concepts to an image according to its content. The main challenges arise from the difficulty of characterizing complex and ambiguous contents of the satellite images and the high human labor cost caused by preparing a large amount of training examples with high-quality pixel-level labels in fully supervised annotation methods. To address these challenges, we propose a unified annotation framework by combining discriminative high-level feature learning and weakly supervised feature transferring. Specifically, an efficient stacked discriminative sparse autoencoder (SDSAE) is first proposed to learn high-level features on an auxiliary satellite image data set for the land-use classification task. Inspired by the motivation that the encoder of the prelearned SDSAE can be regarded as a generic high-level feature extractor for HR optical satellite images, we then transfer the learned high-level features to semantic annotation. To compensate the difference between the auxiliary data set and the annotation data set, the transferred high-level features are further fine-tuned in a weakly supervised scheme by using the tile-level annotated training data. Finally, the fine-tuning process is formulated as an ultimate optimization problem, which can be solved efficiently with our proposed alternate iterative optimization method. Comprehensive experiments on a publicly available land-use classification data set and an annotation data set demonstrate the superiority of our SDSAE-based high-level feature learning method and the effectiveness of our weakly supervised semantic annotation framework compared with state-of-the-art fully supervised annotation methods.

A sketch-based image retrieval often needs to optimize the tradeoff between efficiency and precision. Index structures are typically applied to large-scale databases to realize efficient retrievals. However, the performance can be affected by quantization errors. Moreover, the ambiguousness of user-provided examples may also degrade the performance, when compared with traditional image retrieval methods. Sketch-based image retrieval systems that preserve the index structure are challenging. In this paper, we propose an effective sketch-based image retrieval approach with re-ranking and relevance feedback schemes. Our approach makes full use of the semantics in query sketches and the top ranked images of the initial results. We also apply relevance feedback to find more relevant images for the input query sketch. The integration of the two schemes results in mutual benefits and improves the performance of the sketch-based image retrieval.

Nowadays, with the boom of social media and e-commerce, more and more people prefer to share their consumption experiences and rate services on review sites. Much research has focused on personalized recommendation. However, quality of service also plays an important role in recommender systems, and it is the main concern of this paper. An overall rating that indicates the popular view usually represents the evaluation. There are some challenges when we do not have enough review information to extract public opinion. Take, for example, a movie for which one user rates a two star rating, and another rates a five star rating. In this case, it is difficult to conduct a quality evaluation fairly. However, it is possible to be improved with the help of big social users' contextual information. In this paper, we propose a model to conduct service quality evaluation by improving overall rating of services using an empirical methodology. We use the concept of user rating's confidence, which denotes the trustworthiness of user ratings. First, entropy is utilized to calculate user ratings' confidence. Second, we further explore spatial-temporal features and review sentimental features of user ratings to constrain their confidences. Last, we fuse them into a unified model to calculate an overall confidence, which is utilized to perform service quality evaluation. Extensive experiments implemented on Yelp and Douban Movie datasets demonstrate the effectiveness of our model.

In recent years, we have witnessed a flourish of review websites. It presents a great opportunity to share our viewpoints for various products we purchase. However, we face an information overloading problem. How to mine valuable information from reviews to understand a user's preferences and make an accurate recommendation is crucial. Traditional recommender systems (RS) consider some factors, such as user's purchase records, product category, and geographic location. In this work, we propose a sentiment-based rating prediction method (RPS) to improve prediction accuracy in recommender systems. Firstly, we propose a social user sentimental measurement approach and calculate each user's sentiment on items/products. Secondly, we not only consider a user's own sentimental attributes but also take interpersonal sentimental influence into consideration. Then, we consider product reputation, which can be inferred by the sentimental distributions of a user set that reflect customers' comprehensive evaluation. At last, we fuse three factors-user sentiment similarity, interpersonal sentimental influence, and item's reputation similarity-into our recommender system to make an accurate rating prediction. We conduct a performance evaluation of the three sentimental factors on a real-world dataset collected from Yelp. Our experimental results show the sentiment can well characterize user preferences, which helps to improve the recommendation performance.

Social media sharing websites like Flickr allow users to annotate images with free tags, which significantly contribute to the development of the web image retrieval and organization. Tag-based image search is an important method to find images contributed by social users in such social websites. However, how to make the top ranked result relevant and, with diversity, is challenging. In this paper, we propose a social re-ranking system for tag-based image retrieval with the consideration of an image's relevance and diversity. We aim at re-ranking images according to their visual information, semantic information, and social clues. The initial results include images contributed by different social users. Usually each user contributes several images. First, we sort these images by inter-user re-ranking. Users that have higher contribution to the given query rank higher. Then we sequentially implement intra-user re-ranking on the ranked user's image set, and only the most relevant image from each user's image set is selected. These selected images compose the final retrieved results. We build an inverted index structure for the social image dataset to accelerate the searching process. Experimental results on a Flickr dataset show that our social re-ranking method is effective and efficient.

The paper presents a sketch-based image retrieval algorithm. One of the main challenges in sketch-based image retrieval (SBIR) is to measure the similarity between a sketch and an image. To tackle this problem, we propose an SBIR-based approach by salient contour reinforcement. In our approach, we divide the image contour into two types. The first is the global contour map. The second, called the salient contour map, is helpful to find out the object in images similar to the query. In addition, based on the two contour maps, we propose a new descriptor, namely an angular radial orientation partitioning (AROP) feature. It fully utilizes the edge pixels' orientation information in contour maps to identify the spatial relationships. Our AROP feature based on the two candidate contour maps is both efficient and effective to discover false matches of local features between sketches and images, and can greatly improve the retrieval performance. The application of the retrieval system based on this algorithm is established. The experiments on the image dataset with 0.3 million images show the effectiveness of the proposed method and comparisons with other algorithms are also given. Compared to baseline performance, the proposed method achieves 10% higher precision in top 5.

With the boom of social media, it is a very popular trend for people to share what they are doing with friends across various social networking platforms. Nowadays, we have a vast amount of descriptions, comments, and ratings for local services. The information is valuable for new users to judge whether the services meet their requirements before partaking. In this paper, we propose a user-service rating prediction approach by exploring social users' rating behaviors. In order to predict user-service ratings, we focus on users' rating behaviors. In our opinion, the rating behavior in recommender system could be embodied in these aspects: 1) when user rated the item, 2) what the rating is, 3) what the item is, 4) what the user interest that we could dig from his/her rating records is, and 5) how the user's rating behavior diffuses among his/her social friends. Therefore, we propose a concept of the rating schedule to represent users' daily rating behaviors. In addition, we propose the factor of interpersonal rating behavior diffusion to deep understand users' rating behaviors. In the proposed user-service rating prediction approach, we fuse four factors-user personal interest (related to user and the item's topics), interpersonal interest similarity (related to user interest), interpersonal rating behavior similarity (related to users' rating behavior habits), and interpersonal rating behavior diffusion (related to users' behavior diffusions)-into a unified matrix-factorized framework. We conduct a series of experiments in the Yelp dataset and Douban Movie dataset. Experimental results show the effectiveness of our approach.

Tag-based image search is an important method to process images contributed by social users in social media sharing websites like Flickr. However, existing ranking methods for tag-based image search frequently return results that are irrelevant, low-diversity or time-consuming. In this paper, we propose a user-oriented image ranking system with the consideration of image relevance, diversity and computation complexity, aiming to automatically rank images according to their visual information, semantic information and social clues. When you input a query in the user-oriented image search engine, images tagged with query are obtained as the initial results. The initial results include images contributed by different social users. Usually each user contributes several images. First we sort these users by inter-user ranking. Users that have a higher contribution to the given query rank higher. Then we sequentially implement intra-user ranking on the ranked user's image set, and only the most relevant image in each user's image set is selected. These selected images compose the final retrieval results. Experimental results on Flickr dataset show that our user-oriented ranking method is effective and efficient.

Nowadays, image location has been widely used in many application scenarios for large geo-tagged image corpora. As to images which are not geographically tagged, we can estimate their locations with the help of the large geo-tagged image set by content based image retrieval. In this paper, we propose a global feature clustering and local feature refinement based image location estimation approach. We exploit spatial information by processing useful visual words. In this process, visual word groups are generated. Moreover to improve the retrieval performance, spatial constraint is utilized to code the relative position of visual words. Here we generate a position descriptor for each visual word. Experiments show the effectiveness of our proposed approach.

With the explosion of e-commerce, it presents a great opportunity for people to share their consumption experience in review websites. However, at the same time we face the information overloading problem. How to mine valuable information from these reviews and make an accurate recommendation is crucial for us. Traditional recommender systems (RS) consider many factors, such as product category, geographic location, user's purchase records, and the other social network factors. In this paper, we firstly propose a social user's reviews sentiment measurement approach and calculate each user's sentiment score on items/services. Secondly, we consider service reputation, which reflects the customers' comprehensive evaluation. At last, we fuse service reputation factor into our recommender system to make an accurate rating prediction, which is based on probabilistic matrix factorization. We conduct a series of experiments on Yelp dataset, and experimental results show the proposed approach outperforms the existing RS approaches.

With the rapid development of social networks, more and more users choose to share their own photos with their friends. Especially, users prefer to share the photos they took during traveling, thus there emerges many user generated content for place-of-interests (POIs). So based on the user contributed photos, we can summarize each POI by mining location-of-interest (LOI, which represents the attractive viewpoints of POI) and selecting some representative images from them. It is important for scheduling a traveling, and in this paper, an effective POI summarization approach is proposed by an improved geo-clustering with visual and views verification, which helps us to have a representative and comprehensive perception for POI. In our approach, we firstly collect POI related photos from social media, and filter the raw data by the combination of tags and geo-locations. Secondly, we mine LOIs for each POI by the improved geo-location clustering method. Finally, we employ visual and views verification to select images from LOIs to summarize the POI. We conduct a series of experiments based on Flcoickr dataset. Experimental results demonstrate the effectiveness of our proposed method.

The paper presents a sketch-based image retrieval algorithm. One of the main challenges in sketch-based image retrieval (SBIR) is to measure the similarity between a sketch and an image in contour with high precision. To tackle this problem, we divided the contour of image into two types: the first is global contour, suggesting that we can use it to reduce the similarity between the images with complex background. The second, called salient contour, is helpful to retrieve images with objects similar to the query. Besides, we propose a new descriptor, namely angular radial orientation partitioning (AROP) feature, which makes full use of the gradient orientation information to decrease the gap between sketch and image. Using the two contours as candidate contours for feature extraction could increase the retrieval rate dramatically. Finally an application of retrieval system based on this algorithm is established. The experiment on 0.42 million image dataset shows excellent retrieval performance of the proposed method and comparisons with other algorithms are also given.

Landmark summarization with diverse viewpoints is very important in landmark retrieval, as it can create a comprehensive description of a landmark for users. In this paper we present an approach for summarizing a collection of landmark images from diverse viewpoints. First, we group landmark images with content overlap by viewpoint album (VA) generation. Second, we model the relative viewpoint of each image within the VA based on the spatial layout of distinctive descriptors of a landmark. Third, we express the relative viewpoint of an image with a 4-D viewpoint vector, including horizontal, vertical, scale, and rotation. Finally, we summarize the landmarks in terms of viewpoints. Experimental results show the effectiveness of the proposed landmark summarization approach.

This paper presents a robust framework to solve the face hallucination problem across multiple factors, i.e., different expressions, head poses, and illuminations. It proposes a redundant transformation with diagonal loading for modeling the mappings among different new face factors, and a local reconstruction with geometry and position constraints for incorporating image details in the new factor spaces. Our proposed redundant and sparse strategies are discussed, and the experiments indicate that it is not necessary to adopt sparse representation in the proposed framework. The experimental results demonstrate that the proposed framework offers robustness when dealing with the inputs that have different expressions, head poses, and illuminations compared with the state-of-the-art methods, can generate high-resolution face images with better image qualities than the hierarchical tensor-based method, and improves the state of the art from single one output to multiple outputs with new factors.

Automatic image annotation is an attractive service for users and administrators of online photo sharing websites. In this paper, we propose an image annotation approach that exploits latent semantic community of labels and multikernel learning (LCMKL). First, a concept graph is constructed for labels indicating the relationship between the concepts. Based on the concept graph, semantic communities are explored using an automatic community detection method. For an image to be annotated, a multikernel support vector machine is used to determine the image's latent community from its visual features. Then, a candidate label ranking based approach is determined by intracommunity and intercommunity ranking. Experiments on the NUS-WIDE database and IAPR TC-12 data set demonstrate that LCMKL outperforms some state-of-the-art approaches.

Nowadays, locations of images have been widely used in many application scenarios for large geo-tagged image corpora. As to images which are not geographically tagged, we estimate their locations with the help of the large geo-tagged image set by content-based image retrieval. In this paper, we exploit spatial information of useful visual words to improve image location estimation (or content-based image retrieval performances). We proposed to generate visual word groups by mean-shift clustering. To improve the retrieval performance, spatial constraint is utilized to code the relative position of visual words. We proposed to generate a position descriptor for each visual word and build fast indexing structure for visual word groups. Experiments show the effectiveness of our proposed approach.

Nowadays, it is very convenient to capture photos by a smart phone. As using, the smart phone is a convenient way to share what users experienced anytime and anywhere through social networks, it is very possible that we capture multiple photos to make sure the content is well photographed. In this paper, an effective scalable mobile image retrieval approach is proposed by exploring contextual salient information for the input query image. Our goal is to explore the high-level semantic information of an image by finding the contextual saliency from multiple relevant photos rather than solely using the input image. Thus, the proposed mobile image retrieval approach first determines the relevant photos according to visual similarity, then mines salient features by exploring contextual saliency from multiple relevant images, and finally determines contributions of salient features for scalable retrieval. Compared with the existing mobile-based image retrieval approaches, our approach requires less bandwidth and has better retrieval performance. We can carry out retrieval with <200-B data, which is <5% of existing approaches. Most importantly, when the bandwidth is limited, we can rank the transmitted features according to their contributions to retrieval. Experimental results show the effectiveness of the proposed approach.

From social media has emerged continuous needs for automatic travel recommendations. Collaborative filtering (CF) is the most well-known approach. However, existing approaches generally suffer from various weaknesses. For example , sparsity can significantly degrade the performance of traditional CF. If a user only visits very few locations, accurate similar user identification becomes very challenging due to lack of sufficient information for effective inference. Moreover, existing recommendation approaches often ignore rich user information like textual descriptions of photos which can reflect users' travel preferences. The topic model (TM) method is an effective way to solve the “sparsity problem,” but is still far from satisfactory. In this paper, an author topic model-based collaborative filtering (ATCF) method is proposed to facilitate comprehensive points of interest (POIs) recommendations for social users. In our approach, user preference topics, such as cultural, cityscape, or landmark, are extracted from the geo-tag constrained textual description of photos via the author topic model instead of only from the geo-tags (GPS locations). Advantages and superior performance of our approach are demonstrated by extensive experiments on a large collection of data.

Owing to the portable and excellent phone camera, people now prefer to take photos and upload them by mobile phone. Content based image retrieval is effective for users to obtain relevant information about a photo. Taking the limited bandwidth and instability into account, we propose an effective scalable mobile image retrieval approach in this paper. The proposed mobile image retrieval algorithm first determines the relevant photos according to visual similarity in mobile end, then mines salient visual words by exploring saliency from multiple relevant images, and finally we determine the contribution order of salient visual words for scalable retrieval. In server terminal, spatial verification is performed to re-rank the results. Compared to the existing approaches of mobile image retrieval, our approach transmits less data and reduces the computational cost of spatial verification. Most importantly, when the bandwidth is limited, we can transmit a part of features according their contributions to retrieval. Experimental results show the effectiveness of the proposed approach.

Social image share websites such as Flickr allow users to manually annotate their images with their own words, which can be used to facilitating image retrieval and other image applications. For the vast number of online images contributed by social users, existing methods on tag recommendation haven't taken users' characteristics and tagging habits into consideration. In this paper, we propose a personalized tag recommendation system for Flickr users. It can recommend users personalized tags for their newly uploaded photos based on the history information in their social communities. We carry out the personalized tag recommendation from three aspects. First, the tags we recommend to users are users' own vocabularies. Second, different recommendation methods are implemented to different users. Third, different users are recommended with different number of tags based on their tagging habits. The experimental results indicate that our personalized tag recommendation is effective.

Mobile visual search is attracting much research attention recently. Existing works focus on addressing the limited capacity of wireless channel yet overlook its instability, thus is not adaptive to the change of channel capacity. In this paper, a novel image retrieval algorithm that is scalable to various channel condition is proposed. The proposed algorithm contains three contributions: (1) to achieve instant retrieval under various channel capacity, we adjust transmission load by sparseness instead of codebook size; (2) we introduce hierarchical sparse coding into our retrieval workflow, where original codebook is transformed into a tree-structured dictionary which implies elements' priority; (3) we propose transmission priority ranking schemes that is adaptive to specific query. Experiment results show that the proposed algorithm outperforms BoW and Lasso based algorithm under different parameter settings. Retrieval results under different channel limitation validate the scalability of our method.

We have witnessed the popularity of image-sharing websites for sharing personal experiences through photos on the Web. These websites allow users describing the content of their uploaded images with a set of tags. Those user-annotated tags are often noisy and biased. Social image tagging aims at removing noisy tags and suggests new relevant tags. However, most existing tag enrichment approaches predominantly focus on tag relevance and overlook tag diversity problem. How to make the top-ranked tags covering a wide range of semantic is still an opening, yet challenging, issue. In this paper, we propose an approach to retag social images with diverse semantics. Both the relevance of a tag to image as well as its semantic compensations to the already determined tags are fused to determine the final tag list for a given image. Different from existing image tagging approaches, the top-ranked tags are not only highly relevant to the image but also have significant semantic compensations with each other. Experiments show the effectiveness of the proposed approach.

With the advent and popularity of social network, more and more users like to share their experiences, such as ratings, reviews, and blogs. The new factors of social network like interpersonal influence and interest based on circles of friends bring opportunities and challenges for recommender system (RS) to solve the cold start and sparsity problem of datasets. Some of the social factors have been used in RS, but have not been fully considered. In this paper, three social factors, personal interest, interpersonal interest similarity, and interpersonal influence, fuse into a unified personalized recommendation model based on probabilistic matrix factorization. The factor of personal interest can make the RS recommend items to meet users' individualities, especially for experienced users. Moreover, for cold start users, the interpersonal interest similarity and interpersonal influence can enhance the intrinsic link among features in the latent space. We conduct a series of experiments on three rating datasets: Yelp, MovieLens, and Douban Movie. Experimental results show the proposed approach outperforms the existing RS approaches.

With the advent and popularity of social network, more and more users like to share their experiences, such as ratings, reviews, and blogs. The new factors of social network like interpersonal influence and interest based on circles of friends bring opportunities and challenges for recommender system (RS) to solve the cold start and sparsity problem of datasets. Some of the social factors have been used in RS, but have not been fully considered. In this paper, three social factors, personal interest, interpersonal interest similarity and interpersonal influence, fuse into a unified personalized recommendation model based on probabilistic matrix factorization. The factor of personal interest can make the RS recommend items to meet users' individualities, especially for experienced users. Moreover, for cold start users, the interpersonal interest similarity and interpersonal influence can enhance the intrinsic link among features in the latent space. We conduct a series of experiments on real rating datasets. Experimental results show the proposed approach outperforms the existing RS approaches .

Automatic image annotation is an important function for online photo sharing service. The concurrence of labels is pretty common in multi-label annotation. In this paper, we propose a novel approach called latent-community and multi-kernel learning (LCMKL). The established graph of labels is regarded as a semantic network. Community detection method is introduced that treats the label set as communities. Multi-kernel learning SVM is adopted for specifying communities and settling difficulty of extracting semantically meaningful entities with some simple features. Experiments on NUS-WIDE database demonstrate that LCMKL outperforms other state-of-the-art approaches.

Traveling experience sharing has become pervasive. In this paper, we present a system which automatically generates a multimedia travelogue for mobile users. Multimedia travelogue shows user footprint with photos on the corresponding location on the map, which could offer the space distribution inner or between the locations. Meanwhile, location overviews with representative tags and images show comprehensive knowledge of the landmarks. The key challenges are: 1) when mapping footprint, some photos are without geo-tags; 2)user-contributed photos may be not clear and graceful; 3) to detect which landmark is actually on the photo to offer location overview. We solve these challenges by combining both geo-tags and tags to estimate the location. First we use the relationship of timestamp and geo-tag in the same trip group to estimate the footprints. Second we localize the landmark on the photo by tag match method which matching user tags with representative tags of the landmark. Experimental results on a Flickr image collection of nearly 2 million images of 6,581 users demonstrate the effectiveness of our approach.

We propose two synthetic aperture radar (SAR) complex image compression schemes based on DLWT_IQ and DLWT_FFT. DLWT_IQ encodes the real parts and imaginary parts of the images using directional lifting wavelet transform (DLWT) and bit plane encoder (BPE), while DLWT_FFT encodes the real images converted by fast Fourier transform (FFT). Compared with discrete wavelet transform-IQ (DWT_IQ), DLWT_IQ improves the peak signal-to-noise ratio (PSNR) up to 1.28 dB and reduces the mean phase error (MPE) up to 21.74%; and compared with DWT_FFT, DLWT_FFT improves the PSNR up to 1.22 dB and reduces the MPE up to 20.32%. Moreover, the proposed schemes increase the PSNR up to 3.34 dB and decrease the MPE up to 50.43% as compared with the set partitioning in hierarchical trees (SPIHT) algorithm. In addition to this, we observe a novel phenomenon, that is, DLWT with direction prediction achieves a higher clustering capability for complex SAR images than DWT. Then, coding algorithm based on DLWT requires fewer coding bits than DWT for the same number of coding coefficients, and DLWT outperforms DWT in terms of rate-distortion performance even if the K-term nonlinear approximation of DWT is better than that of DLWT.

Social media has become a very popular way for people to share their photos with friends. Because most of the social images are attached with GPS (geo-tags), a photo's GPS information can be estimated with the help of the large geo-tagged image set while using a visual searching based approach. This paper proposes an unsupervised image GPS location estimation approach with hierarchical global feature clustering and local feature refinement. It consists of two parts: an offline system and an online system. In the offline system, a hierarchical structure is constructed for a large-scale offline social image set with GPS information. Representative images are selected for each GPS location refined cluster, and an inverted file structure is proposed. In the online system, when given an input image, its GPS information can be estimated by hierarchical global clusters selection and local feature refinement in the online system. Both the computational cost and GPS estimation performance demonstrates the effectiveness of the proposed hierarchical structure and inverted file structure in our approach.

In social image sharing websites, users provide several descriptive tags to annotate their shared images. Usually, the raw tags are noisy, biased and incomplete. How to filter the tags is important for tag based applications. In this paper, a similar compatible principle based tag filtering approach is proposed. We classify tags into two sets. One is relevant to image content. The other is irrelevant to image content. We filter the tags by ranking high relevant tags ahead of the tags with low relevance by the similar compatible principle. This approach determines the ranks of user annotated tags by maximizing the compatible value of changing the labels of the tags from irrelevant to relevant at each step. Experiments on crawled Flickr dataset demonstrate the effectiveness of the proposed approach.

In this paper, we describe an approach for visually summarizing a landmark by recommending images with diverse viewpoints (e.g. front-side viewpoint, bottom-top viewpoint, close-distant viewpoint, etc). Our approach models an image's viewpoint using a 4-D viewpoint vector, which describes viewpoint in horizontal, vertical, scale and orientation aspects. To construct the viewpoint vector for an image, we select Identical Semantic Points (ISPs) from hundreds to thousands SIFT points of the image to captures some major and unique parts of a landmark. Then a four dimensional viewpoint vector is utilized to measure on the position coordinate, scale and orientation of the ISPs in an image. After that, we perform viewpoint clustering to finally summarize landmarks. We evaluate our approach on 5K Oxford building image set and provide final summarization results for some famous landmarks in Oxford.

In this paper, a graph cut based tag enrichment approach is proposed. We build a graph for each image with its initial tags. The graph is with two terminals. Nodes of the graph are full connected with each other. Min-cut/max-flow algorithm is utilized to find the relevant tags for the image. Experiments on Flickr dataset demonstrate the effectiveness of the proposed graph-cut based tag enrichment approach.

In H.264/AVC, the encoders have to try all the block types (16×16, 16×8, 8×16, 8×8, 8×4, 4×8 and 4×4) for each macro-block motion estimation and inter mode decision. All motion vectors and residuals are passed into Lagrange rate distortion (RD) cost model to calculate all RD cost values exhaustively, which introduce considerable computation complexity and resource demanding. In this paper, we introduce a JND (Just Noticeable Difference) profile and analyze how this profile characterizes rate and distortion. Based on JND profile, we devise a perceptual rate distortion (PRD) model with low-complexity to replace the Lagrange RD cost model. We scheme a novel inter macro-block partitioning method based on this PRD model. Our model and method first adopt perceptual filters to find the best mode of every macro-block and then start the motion estimation. Simulation results demonstrated that the inter mode decision based on PRD model won about 80% encoding time reduction and better quality than the inter mode decision based on the RD cost value decision.

Scale and rotation invariant texture classification is a challenging and significant topic in texture analysis. This paper presents a scale and rotation invariant Gabor Texture Descriptor (GTD) for texture classification. Firstly, Gabor filters with different orientations and scales are carried out on an image. Secondly, to obtain a scale invariant GTD, different scale energies of these Gabor filtered images are calculated, and the optimal matching scales of the GTD are chosen by the sliding window filter with the scale energies. Thirdly, to extract a rotation invariant GTD, a new method using Discrete Fourier Transform is employed on the scale invariant GTD. At last, the distances between the GTDs of a query image and the target images are calculated for texture classification. Five different GTDs are employed for texture classification to demonstrate the effectiveness of the proposed scale and rotation invariant GTD.

Information of text in videos and images plays an important role in semantic analysis. In this paper, we propose an effective method for text detection and localization in noisy background. The algorithm is based on corner response. Compared to non-text regions, there often exist dense edges and corners in text regions. So we can get relatively strong responses from text regions and low responses from non-text regions. These responses provide us useful cues for text detection and localization in images. Then using a simple block based threshold scheme, we get candidate regions for text. These regions are further verified by combining other features such as color and size range of connected component. Finally, text line is located accurately by the projection of corner response. The experimental results show the effectiveness of our methods.

Object categorization plays an important role in computer vision, semantic based image content understanding, and image retrieval. Wavelet packet transform provides a very good observation for the images by sub-band filtering. Different objects have distinctive characteristics in the sub-bands of wavelet packets, which should be discriminative for objects classification. In this paper, an object categorization method using hierarchical wavelet packet texture descriptors is proposed. Comparisons between Gabor texture descriptor, pyramid of histograms of orientation gradients (PHOG) and the proposed hierarchical wavelet packet texture descriptors on the widely used OT, Scene-13 and Sport event datasets are also given. Experimental results show that object categorization performances of the proposed texture descriptors are better than that of Gabor texture descriptor and as good as that of PHOG shape descriptor. Object categorization performances of the texture descriptors under various decomposition levels and wavelet bases are discussed. Performances of texture descriptors of global and local images with different partition patterns are also analyzed.

Parts of compressed video streams may be lost or corrupted when being transmitted over bandwidth limited networks and wireless communication networks with error-prone channels. Error concealment (EC) techniques are often adopted at the decoder side to improve the quality of the reconstructed video. Under the conditions of a high rate of data packets that arrives at the decoder corrupted, it is likely that the incorrectly decoded macro-blocks (MBs) are concentrated in a connected region, where important spatial reference information is lost. The conventional EC methods usually carry out the block concealment following a lexicographic scan (from top to bottom and from left to right of the image), which would make the methods ineffective for the case that the corrupted blocks are grouped in a connected region. In this paper, a temporal error concealment method, adaptive error concealment order determination (AECOD), is proposed to recover connected corrupted regions. The processing order of an MB in a connected corrupted region is adaptively determined by analyzing the external boundary patterns of the MBs in its neighborhood. The performances, on several video sequences, of the proposed EC scheme have been compared with those obtained by using other error concealment methods reported in the literature. Experimental results show that the AECOD algorithm can improve the recovery performance with respect to the other considered EC methods.

The starting and ending (S & E) frames of each text that embeds in a scene or is graphically added to video, provide not only important clues for highlight events detection in semantic-based video analysis, indexing and retrieval but also hint for decoding the videos structure and classification. In this paper, a fast text tracking method of determining S & E frames is proposed, which consists of two modules: text matching, where a mean absolute difference (MAD) based line against line matching (LALM) criterion is proposed to determine whether the detected texts in two frames are similar or not; and S & E frames determination, where an MAD based line matching algorithm is introduced in the direct current (DC) domain to determine the accurate positions of the S & E frame. Experiments conducted with a variety of video sources show that the proposed text tracking method is fast and robust.

Error concealment (EC) techniques are often adopted at decoder side to improve the reconstructed video qualities. In this paper, a texture based selective boundary matching algorithm (TSBMA) is proposed to conceal erroneous macro-blocks (EMBs). Different from the conventional boundary matching algorithms which are carried out progressively in the raster scan mode, the processing order of the TSBMA for an EMB in a connected EMBs region is determined by the texture intensity information of its neighboring blocks, which are correct or recovered. The EMB with maximum texture intensity in its neighbors are firstly recovered. It is likely that the EMB which has more corrected MBs in its neighbors with stronger texture intensity has higher possibility to be recovered accurately, which will provide valuable information to recover the left EMBs in its neighbors. Thus, the proposed TSBMA can avoid the dependent recovery problem in the conventional BMAs. Moreover, the influence of boundary width and confidences of boundaries on the recovery results are discussed. Comparison results with different error concealment methods on several video sequences show the effectiveness of the proposed TSBMA.

Video text information plays an important role in semantic-based video analysis, indexing and retrieval. Video texts are closely related to the content of a video. Text-based video analysis, browsing and retrieval are usually carried out in the following for steps: video text detection, localization, segmentation and recognition. Videos are commonly stored in compressed formats where MPEG coding techniques are adopted. In this paper, a DCT coefficient based multilingual video text detection and localization scheme for compressed videos is proposed. Candidate text blocks are detected in terms of block texture constraint. An adaptive method for the horizontal and vertical aligned text lines determination is then designed according to the run length of the horizontal and vertical block numbers. The remaining block regions are further verified by local block texture constraints. And the text block region can be localized by virtue of the horizontal and vertical block texture projections. Finally, a foreground and background integrated (FBI) video text segmentation approach is adopted in this paper to eliminate the complex background in text regions. The final experimental results show the effectiveness of our methods

Scene change detection is a fundamental step in automatic video indexing, browsing and retrieval. Fade in and fade out are two kinds of gradually changing scenes which are difficult to be detected in comparison with the abruptly changing scenes. The salient character of flashlight effect is the luminance change, which is caused by abrupt appearance or disappearance of the illumination source. Performance of shot boundary detection is not satisfactory for the video sequences containing flashlights, if no flashlight discrimination strategy is adopted. In this paper, an effective fades and flashlight detection method is proposed for both the compressed and uncompressed videos, based on the accumulating histogram difference (AHD). This fades detection method is proposed in terms of their mathematical models. AHDs of all the two consecutive frames during fades transitions can be classified into six cases. The flashlight detection method is proposed based on the AHD and the energy variation characters. AHD and energy variation characters for the starting and ending frames of a flashlight have certain regularities, which can also be expressed by cases. Thus the fades and flashlight detection problems are converted into cases matching ones. Experimental results on several test video sequences with different bit rates show the effectiveness of the proposed AHD based fades and flashlight detection method