Obtaining the precise 3D motion of a table tennis ball from standard monocular videos is a challenging problem, as existing methods trained on synthetic data struggle to generalize to the noisy, imperfect ball and table detections of the real world. This is primarily due to the inherent lack of 3D ground truth trajectories and spin annotations for real-world video. To overcome this, we propose a novel two-stage pipeline that divides the problem into a front-end perception task and a back-end 2D-to-3D uplifting task. This separation allows us to train the front-end components with abundant 2D supervision from our newly created TTHQ dataset, while the back-end uplifting network is trained exclusively on physically-correct synthetic data. We specifically re-engineer the uplifting model to be robust to common real-world artifacts, such as missing detections and varying frame rates. By integrating a ball detector and a table keypoint detector, our approach transforms a proof-of-concept uplifting method into a practical, robust, and high-performing end-to-end application for 3D table tennis trajectory and spin analysis.

In content-based image retrieval (CBIR), aligning user queries with database images is challenging, especially when the queries are ‘impossible queries’ i.e. queries that seek items that are non-existent in the database. These gaps in retrieval systems result in user dissatisfaction. This paper introduces a novel task, query refinement for non-existent items (QRNE), an approach that reformulates impossible queries to better match available content, while preserving the user’s original intent. We have found limitations in current encoding systems, particularly the limitations of CLIP embeddings. Due to CLIP’s contrastive learning-based training mechanism, it may struggle to identify similarities. To address this challenge, this paper proposes a embedding remapping network for the CLIP encoder and a QRNE baseline algorithm leveraging the remapping network. Experiments show that the remapped embeddings are able to overcome the limitations of CLIP embeddings.

Bhuiyan Sanjid Shafique, Ashmal Vayani, Muhammad Maaz, Hanoona Abdul Rasheed, Dinura Dissanayake, Mohammed Irfan Kurpath, Yahya Hmaiti, Go Inoue, Jean Lahoud, Md. Safirur Rashid, Shadid Intisar Quasem, Maheen Fatima, Franco Vidal, Mykola Maslych, Ketan Pravin More, Sanoojan Baliah, Hasindri Watawana, Yuhao Li, Fabian Farestam, Leon Schaller, Roman Tymtsiv, Simon Weber, Hisham Cholakkal, Ivan Laptev, Shin’ichi Satoh, Michael Felsberg, Mubarak Shah, Salman Khan, Fahad Shahbaz Khan. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

In many countries, studies have highlighted the under-representation of women in the media. But beyond quantitative imbalance is the question of the qualitative asymmetry of men and women portrayals. How to help the evaluation of content and salient features specific to male and female discourse? We propose in this study to leverage the knowledge acquired by a classification model trained for gender detection on automatic transcripts, to highlight patterns distinctive of male or female speech. Results show the relevance of coupling explainable AI with classifier confidence to compute consistent attributions.

Cross-modal retrieval models have shown impressive performance on the image-to-recipe retrieval task, a common benchmark in the multimedia field. However, the task assumes that an exact recipe match for a query image exists in the target database—an assumption that rarely holds true in real-world scenarios. When excluding exact matches from the target domain, our analysis revealed that relying solely on visual and textual similarity between recipes is insufficient to achieve good retrieval results. Other similarities should also be considered. Since ingredient similarity aligns with human intuition and nutritional similarity is crucial for health-conscious applications, we propose a model that incorporates ingredient and nutritional relevance into the retrieval process. We measured the similarity of unpaired recipes using three new metrics: mean absolute scaled error (MASE) for assessing nutritional similarity and IOU and weighted IOU (WIOU) for measuring ingredient overlap. Our proposed method can also be applied with existing image-recipe retrieval models and improved top-1 MASE, IOU, and WIOU by up to 18.13%, 9.91%, and 6.88%.

One-shot subset selection serves as an effective tool to reduce deep learning training costs by identifying an informative data subset based on the information extracted by an information extractor (IE). Traditional IEs, typically pre-trained on the target dataset, are inherently dataset-dependent. Foundation models (FMs) offer a promising alternative, potentially mitigating this limitation. This work investigates two key questions: (1) Can FM-based subset selection outperform traditional IE-based methods across diverse datasets? (2) Do all FMs perform equally well as IEs for subset selection? Extensive experiments uncovered surprising insights: FMs consistently outperform traditional IEs on fine-grained datasets, whereas their advantage diminishes on coarse-grained datasets with noisy labels. Motivated by these finding, we propose RAM-APL (RAnking Mean-Accuracy of Pseudo-class Labels), a method tailored for fine-grained image datasets. RAM-APL leverages multiple FMs to enhance subset selection by exploiting their complementary strengths. Our approach achieves state-of-the-art performance on fine-grained datasets, including Oxford-IIIT Pet, Food-101, and Caltech-UCSD Birds-200-2011.

Action recognition aims to identify action categories in trimmed videos captured by multimedia devices, which often suffer from jitter, especially in uncrewed aerial vehicle (UAV) applications. Existing methods typically ignore the effect of jitter on actor motion or rely on external stabilization tools trained on large-scale unstable video datasets that may not be tailored to specific tasks. To address this, we propose the Stabilization-enhanced Recognition Network (StaRNet), an end-to-end framework that integrates video stabilization and contrastive learning. Inspired by traditional stabilizers, StaRNet’s Motion-aware Stabilization Module (MSM) constructs positive and negative video pairs to model instability: positive pairs use optical flow to estimate frame motion and refine rigid motion via keyframe estimation for motion-aware stabilization, while negative pairs assess temporal consistency using motion cues to boost classification. Moreover, we introduce a Motion-aware Constraint (MC) that regulates dynamic stabilization to adapt to varying motion patterns and enrich action representations. Experiments on UAV benchmarks show that StaRNet outperforms state-of-the-art methods and substantially enhances video stabilization. The code is available athttps://github.com/lwxfight/-StaRNet

Previous incomplete multi-modal brain tumor segmentation technologies, while effective in integrating diverse modalities, commonly deliver under-expected performance gains. The reason lies in that the new modality may cause confused predictions due to uncertain and inconsistent patterns and quality in some positions, where the direct fusion consequently raises the negative gain for the final decision. In this paper, considering the potentially negative impacts within a modality, we propose multi-modal Positive-Negative impact region Double Calibration pipeline, called PNDC, to mitigate misinformation transfer of modality fusion. Concretely, PNDC involves two elaborate pipelines, Reverse Audit and Forward Checksum. The former is to identify negative regions impacts of each modality. The latter calibrates whether the fusion prediction is reliable in these regions by integrating the positive impacts regions of each modality. Finally, the negative impacts region from each modality and miss-match reliable fusion predictions are utilized to enhance the learning of individual modalities and fusion process. It is noted that PNDC adopts the standard training strategy without specific architectural choices and does not introduce any learning parameters, and thus can be easily plugged into existing network training for incomplete multi-modal brain tumor segmentation. Extensive experiments confirm that our PNDC greatly alleviates the performance degradation of current state-of-the-art incomplete medical multi-modal methods, arising from overlooking the positive/negative impacts regions of the modality. The code is released at PNDC.

Coreset selection seeks to choose a subset of crucial training samples for efficient learning. It has gained traction in deep learning, particularly with the surge in training dataset sizes. Sample selection hinges on two main aspects: a sample's representation in enhancing performance and the role of sample diversity in averting overfitting. Existing methods typically measure both the representation and diversity of data based on similarity metrics, such as L2-norm. They have capably tackled representation via distribution matching guided by the similarities of features, gradients, or other information between data. However, the results of effectively diverse sample selection are mired in sub-optimality. This is because the similarity metrics usually simply aggregate dimension similarities without acknowledging disparities among the dimensions that significantly contribute to the final similarity. As a result, they fall short of adequately capturing diversity. To address this, we propose a feature-based diversity constraint, compelling the chosen subset to exhibit maximum diversity. Our key lies in the introduction of a novel Contributing Dimension Structure (CDS) metric. Different from similarity metrics that measure the overall similarity of high-dimensional features, our CDS metric considers not only the reduction of redundancy in feature dimensions, but also the difference between dimensions that contribute significantly to the final similarity. We reveal that existing methods tend to favor samples with similar CDS, leading to a reduced variety of CDS types within the coreset and subsequently hindering model performance. In response, we enhance the performance of five classical selection methods by integrating the CDS constraint. Our experiments on three datasets demonstrate the general effectiveness of the proposed method in boosting existing methods.

In content-based image retrieval (CBIR), aligning user queries with database images is challenging, especially when the queries are “impossible queries,“ queries that seek items that are non-existent in the database. These gaps in retrieval systems result in user dissatisfaction. This paper introduces a novel task, query refinement for non-existent items (QRNE), an approach that reformulates impossible queries to better match available content, maintaining the user's original input. We have found limitations in current encoding systems, particularly the limitations of CLIP features. To achieve a QRNE algorithm that can address this challenge, this paper proposes a feature remapping network for the CLIP encoder and a QRNE baseline algorithm leveraging the remapping network. Experiments show that the remapped features are able to overcome the limitations of CLIP features.

Understanding student engagement in online education is crucial for optimizing learning outcomes. This paper introduces ECLIPSE dataset (Extended Classroom Learning Insights via Prolonged Student Engagement), comprising 10,110 annotated images from a 55-minutes , 30-minutes and 20-minutes online lecture. Annotations include four affective states: engagement, boredom, confusion, and frustration. ECLIPSE enables the investigation of learner attention dynamics over extended periods, overcoming the limitations of short-duration datasets. We establish benchmarks for ECLIPSE using models such as EfficientNet, Vision Transformer, Residual Attention Network, and GLAMOR-Net. We propose NeuralGaze, a novel framework integrating Neural Cellular Automata (NCA) with self-attention mechanisms, demonstrating superior accuracy in engagement level assessment compared to basic single-frame models. Furthermore, we introduce CG-SwT, a content-guided Swin Transformer model, which significantly outperforms the baseline ViT model on the ECLIPSE dataset (with F1-score improvements of 21.12%, 12.5%, 16.77%, and 15.41% for engagement, boredom, frustration, and confusion respectively). Our methods surpass existing single-frame engagement prediction baselines for both EngageNet and DAiSEE datasets by significant margins (7.4% and 6.2%, respectively). The code and dataset will be made publicly available.

Neural Architecture Search (NAS) methods seek optimal networks by exploring thousands of variants of a reference architecture.Yet, optimality is typically related to prediction performance, overlooking the environmental impacts of training.Thus, NAS search spaces are unfit for performance and energy consumption trade-offs.We contribute to energy-aware NAS with (i) a grammar-based Convolutional Neural Network generator (CN-NGen) producing diverse architectures not based on a reference one; (ii) 1,300 available architectures obtained via CNNGen with their implementation, energy consumption and performance measurements; (iii) Three state-of-the-art predictors releasing the need for trained models for performance and energy estimation. CNN Generator (CNNGen)CNNGen uses the Xtext context-free grammar framework [3] to generate CNN architectures.The sequence of grammar tokens describes the CNN's topology (i.e., the succession of layers).Our grammar captures the CNN domain knowledge to produce valid architectures.Thus, CNNGen differs from other NAS methods like NASBench [2] Indeed, CNNGen produces architectures from scratch and not as variants of existing ones.CNNGen also comes with an editor allowing to specify architectures.From a valid sequence of grammar tokens, 173

Video Tube Retrieval (VTR) has attracted wide attention in the multi-modal domain, aiming to accurately localize the spatial-temporal tube in videos based on the natural language description. Despite the remarkable progress, existing VTR models trained on a specific domain (source domain) often perform unsatisfactory in another domain (target domain), due to the domain gap. Toward this issue, we introduce the learning strategy, Unsupervised Domain Adaptation, into the VTR task (UDA-VTR), which enables the knowledge transfer from the labeled source domain to the unlabeled target domain without additional manual annotations. An intuitive solution is generating the pseudo labels for the target domain samples with the fully trained source model and fine-tuning the source model on the target domain with pseudo labels. However, the existing domain gap gives rise to two problems for this process: (1) The transfer of model parameters across domains may introduce source domain bias into target domain features, significantly impacting the feature-based prediction for target domain samples. (2) The pseudo labels tend to identify video tubes that are widely present in the source domain, rather than accurately localizing the correct video tubes specific to the target domain samples. To address the above issues, we propose the unsupervised domain adaptation model via Hierarchical dEbiAsing and noisy corRecTion (HEART) for cross-domain video tube retrieval, which contains two characteristic modules: Layered Feature Debiasing (including the adversarial feature alignment and the graph based alignment) and Pseudo Label Refinement. Extensive experiments prove the effectiveness of our HEART model by significantly surpassing the state-of-the-arts.

Despite the impressive achievements of Deep Neural Networks (DNNs) in computer vision, their vulnerability to adversarial attacks remains a critical concern. Extensive research has demonstrated that incorporating sophisticated perturbations into input images can lead to a catastrophic degradation in DNNs' performance. This perplexing phenomenon not only exists in the digital space but also in the physical world. Consequently, it becomes imperative to evaluate the security of DNNs-based systems to ensure their safe deployment in real-world scenarios, particularly in security-sensitive applications. To facilitate a profound understanding of this topic, this paper presents a comprehensive overview of physical adversarial attacks. First, we distill four general steps for launching physical adversarial attacks. Building upon this foundation, we uncover the pervasive role of artifacts carrying adversarial perturbations in the physical world. These artifacts influence each step. To denote them, we introduce a new term: adversarial medium. Then, we take the first step to systematically evaluate the performance of physical adversarial attacks, taking the adversarial medium as a first attempt. Our proposed evaluation metric, hiPAA, comprises six perspectives: Effectiveness, Stealthiness, Robustness, Practicability, Aesthetics, and Economics. We also provide comparative results across task categories, together with insightful observations and suggestions for future research directions.

Deep learning has been widely studied for processing and understanding multimedia data, and it does help improve performance. Recent research has shown that deep models are vulnerable to images containing adverse weather corruptions, leading to a safety risk for numerous safety-critical systems (e.g., autonomous driving systems). There are two problems with the current situation. First, collecting data under different weather scenarios is highly difficult in practice. Second, the performance degrades significantly when the training and test data are from different distributions, as exemplified by the weather corrupted test data. As a result, it is challenging to train a model without access to the images containing variations of various weather conditions, and it is difficult to make trained model generalized to unknown data under different weather conditions. In this paper, we introduce aCounterfactual Representation Learning(CRL) method to address these problems. Without access to training data including weather condition variations, our CRL makes the model resistant to unseen test data that has been corrupted by weather condition variations. Our basic idea is inspired by the perspective of counterfactual regularization. We build a causal model that introduces a counterfactual variable to eliminate the unobserved characteristics brought about by weather conditions. In particular, such a counterfactual variable is approximated by randomly shuffled features, echoing the previous empirical observation that the shuffling technique can perturb the shape details while preserving the local textures. We use information theoretic representation learning to encourage the neural networks to learn more powerful and robust features, which consist of two components. We conduct experiments on five benchmark datasets, namely, CIFAR-100-C, ImageNet-C, KITTI-C, BDD100 k, and CityScapes-C, all of which contain weather corruption. The results of our experiments show that our proposed method can not only be a plug-and-play technique but also work nicely for both object recognition and detection.

Not all semantics become confusing when deploying a semantic segmentation model for real-world scene understanding of adverse weather. The true semantics of most pixels have a high likelihood of appearing in the few top classes according to confidence ranking. In this paper, we replace the one-hot pseudo label with a candidate label set (CLS) that consists of only a few ambiguous classes and exploit its effects on self-training-based unsupervised domain adaptation. Specifically, we formulate the problem as a coarse-to-fine process. In the coarse-level process, adaptive CLS selection is proposed to pick a minimal set of confusing candidate labels based on the reliability of label predictions. Then, representation learning and label rectification are iteratively performed to facilitate feature clustering in an embedding space and to disambiguate the confusing semantics. Experimentally, our method outperforms the state-of-the-art methods on three realistic foggy benchmarks.

Adversarial attacks on thermal infrared imaging expose the risk of related applications. Estimating the security of these systems is essential for safely deploying them in the real world. In many cases, realizing the attacks in the physical space requires elaborate special perturbations. These solutions are often impractical and attention-grabbing. To address the need for a physically practical and stealthy adversarial attack, we introduce HotCold Block, a novel physical attack for infrared detectors that hide persons utilizing the wearable Warming Paste and Cooling Paste. By attaching these readily available temperature-controlled materials to the body, HotCold Block evades human eyes efficiently. Moreover, unlike existing methods that build adversarial patches with complex texture and structure features, HotCold Block utilizes an SSP-oriented adversarial optimization algorithm that enables attacks with pure color blocks and explores the influence of size, shape, and position on attack performance. Extensive experimental results in both digital and physical environments demonstrate the performance of our proposed HotCold Block. Code is available: https://github.com/weihui1308/HOTCOLDBlock.

Human evaluation is critical for validating the performance of text-to-image generative models, as this highly cognitive process requires deep comprehension of text and images. However, our survey of 37 recent papers reveals that many works rely solely on automatic measures (e.g., FID) or perform poorly described human evaluations that are not reliable or repeatable. This paper proposes a standardized and well-defined human evaluation protocol to facilitate verifiable and reproducible human evaluation in future works. In our pilot data collection, we experimentally show that the current automatic measures are incompatible with human perception in evaluating the performance of the text-to-image generation results. Furthermore, we provide insights for designing human evaluation experiments reliably and conclusively. Finally, we make several resources publicly available to the community to facilitate easy and fast implementations.

Referring image segmentation is a task that aims to predict pixel-wise masks corresponding to objects in an image described by natural language expressions.Previous methods for referring image segmentation employ a cascade framework to break down complex problems into multiple stages.However, its limitations are also apparent: existing methods within the cascade framework may encounter challenges in both maintaining a strong focus on the most relevant information during specific stages of the referring image segmentation process and rectifying errors propagated from early stages, which can ultimately result in sub-optimal performance.To address these limitations, we propose the Joint Mask Contextual Embedding Learning Network (JMCELN).JMCELN is designed to enhance the Cascade Framework by incorporating a Learnable Contextual Embedding and a Progressive Alignment Network (PAN).The Learnable Contextual Embedding module dynamically stores and utilizes reasoning information based on the current mask prediction results, enabling the network to adaptively capture and refine pertinent information for improved mask prediction accuracy.Furthermore, the Progressive Alignment Network (PAN) is introduced as an integral part of JMCELN.PAN leverages the output from the previous layer as a filter for the current output, effectively reducing inconsistencies between predictions from different stages.By iteratively aligning the predictions, PAN guides the Learnable Contextual Embedding to incorporate more discriminative information for reasoning, leading to enhanced prediction quality and a reduction in error propagation.With these methods, we achieved state-of-the-art results on three commonly used benchmarks, especially in more intricate datasets.

Image degradation often occurs during fast camera or object movements, regardless of the exposure modes: global shutter (GS) or rolling shutter (RS). Since these two exposure modes give rise to intrinsically different degradations, two restoration threads have been explored separately, i.e. motion deblurring of GS images and distortion correction of RS images, both of which are challenging restoration tasks, especially in the presence of a single input image. In this paper, we explore a novel in-between exposure mode, called global reset release (GRR) shutter, which produces GS-like blur but with row-dependent blur magnitude. We take advantage of this unique characteristic of GRR to explore the latent frames within a single image and restore a clear counterpart by only relying on these latent contexts. Specifically, we propose a residual spatially-compensated and spectrally-enhanced Transformer (RSS-T) block for row-dependent deblurring of a single GRR image. Its hierarchical positional encoding compensates global positional context of windows and enables order-awareness of the local pixel’s position, along with a novel feed-forward network that simultaneously uses spatial and spectral information for gaining mixed global context. Extensive experimental results demonstrate that our method outperforms the state-of-the-art GS deblurring and RS correction methods on single GRR input.

Person re-identification (re-ID) requires densely distributed cameras. In practice, the person of interest may not be captured by cameras and therefore need to be retrieved using subjective information (e.g., sketches from witnesses). Previous research defines this case using the sketch as sketch re-identification (Sketch re-ID) and focuses on eliminating the domain gap. Actually, subjectivity is another significant challenge. We model and investigate it by posing a new dataset with multi-witness descriptions. It features two aspects. 1) Large-scale. It contains over 4,763 sketches and 32,668 photos, making it the largest Sketch re-ID dataset. 2) Multi-perspective and multi-style. Our dataset offers multiple sketches for each identity. Witnesses' subjective cognition provides multiple perspectives on the same individual, while different artists' drawing styles provide variation in sketch styles. We further have two novel designs to alleviate the challenge of subjectivity. 1) Fusing subjectivity. We propose a non-local (NL) fusion module that gathers sketches from different witnesses for the same identity. 2) Introducing objectivity. An AttrAlign module utilizes attributes as an implicit mask to align cross-domain features. To push forward the advance of Sketch re-ID, we set three benchmarks (large-scale, multi-style, cross-style). Extensive experiments demonstrate our leading performance in these benchmarks. Dataset and Codes are publicly available at: https://github.com/Lin-Kayla/subjectivity-sketch-reid

Large ground-truth datasets and recent advances in deep learning techniques have been useful for layout detection. However, because of the restricted layout diversity of these datasets, training on them requires a sizable number of annotated instances, which is both expensive and time-consuming. As a result, differences between the source and target domains may significantly impact how well these models function. To solve this problem, domain adaptation approaches have been developed that use a small quantity of labeled data to adjust the model to the target domain. In this research, we introduced a synthetic document dataset called RanLayNet, enriched with automatically assigned labels denoting spatial positions, ranges, and types of layout elements. The primary aim of this endeavor is to develop a versatile dataset capable of training models with robustness and adaptability to diverse document formats. Through empirical experimentation, we demonstrate that a deep layout identification model trained on our dataset exhibits enhanced performance compared to a model trained solely on actual documents. Moreover, we conduct a comparative analysis by fine-tuning inference models using both PubLayNet and IIIT-AR-13K datasets on the Doclaynet dataset. Our findings emphasize that models enriched with our dataset are optimal for tasks such as achieving 0.398 and 0.588 mAP95 score in the scientific document domain for the TABLE class.

Previous studies have shown that optimizing the information bottleneck can significantly improve the robustness of deep neural networks. Our study closely examines the information bottleneck principle and proposes an Information Bottleneck Distillation approach. This specially designed, robust distillation technique utilizes prior knowledge obtained from a robust pre-trained model to boost information bottlenecks. Specifically, we propose two distillation strategies that align with the two optimization processes of the information bottleneck. Firstly, we use a robust soft-label distillation method to increase the mutual information between latent features and output prediction. Secondly, we introduce an adaptive feature distillation method that automatically transfers relevant knowledge from the teacher model to the student model, thereby reducing the mutual information between the input and latent features. We conduct extensive experiments to evaluate our approach's robustness against state-of-the-art adversarial attackers such as PGD-attack and AutoAttack. Our experimental results demonstrate the effectiveness of our approach in significantly improving adversarial robustness. Our code is available at https://github.com/SkyKuang/IBD.

Recent studies show that deep person re-identification (re-ID) models are vulnerable to adversarial examples, so it is critical to improving the robustness of re-ID models against attacks. To achieve this goal, we explore the strengths and weaknesses of existing re-ID models, i.e., designing learning-based attacks and training robust models by defending against the learned attacks. The contributions of this paper are three-fold: First, we build a holistic attack-defense framework to study the relationship between the attack and defense for person re-ID. Second, we introduce a combinatorial adversarial attack that is adaptive to unseen domains and unseen model types. It consists of distortions in pixel and color space (i.e., mimicking camera shifts). Third, we propose a novel virtual-guided meta-learning algorithm for our attack-defense system. We leverage a virtual dataset to conduct experiments under our meta-learning framework, which can explore the cross-domain constraints for enhancing the generalization of the attack and the robustness of the re-ID model. Comprehensive experiments on three large-scale re-ID benchmarks demonstrate that: 1) Our combinatorial attack is effective and highly universal in cross-model and cross-dataset scenarios; 2) Our meta-learning algorithm can be readily applied to different attack and defense approaches, which can reach consistent improvement; 3) The defense model trained on the learning-to-learn framework is robust to recent SOTA attacks that are not even used during training.

Recent person Re-IDentification (ReID) systems have been challenged by changes in personnel clothing, leading to the study of Cloth-Changing person ReID (CC-ReID). Commonly used techniques involve incorporating auxiliary information (e.g., body masks, gait, skeleton, and keypoints) to accurately identify the target pedestrian. However, the effectiveness of these methods heavily relies on the quality of auxiliary information and comes at the cost of additional computational resources, ultimately increasing system complexity. This paper focuses on achieving CC-ReID by effectively leveraging the information concealed within the image. To this end, we introduce an Auxiliary-free Competitive IDentification (ACID) model. It achieves a win-win situation by enriching the identity (ID)-preserving information conveyed by the appearance and structure features while maintaining holistic efficiency. In detail, we build a hierarchical competitive strategy that progressively accumulates meticulous ID cues with discriminating feature extraction at the global, channel, and pixel levels during model inference. After mining the hierarchical discriminative clues for appearance and structure features, these enhanced ID-relevant features are crosswise integrated to reconstruct images for reducing intra-class variations. Finally, by combing with self- and cross-ID penalties, the ACID is trained under a generative adversarial learning framework to effectively minimize the distribution discrepancy between the generated data and real-world data. Experimental results on four public cloth-changing datasets (i.e., PRCC-ReID, VC-Cloth, LTCC-ReID, and Celeb-ReID) demonstrate the proposed ACID can achieve superior performance over state-of-the-art methods. The code is available soon at: https://github.com/BoomShakaY/Win-CCReID.

Unsupervised domain adaptation for person re-identification (Re-ID) suffers severe domain discrepancies between source and target domains. To reduce the domain shift caused by the changes of context, camera style, or viewpoint, existing methods in this field fine-tune and adapt the Re-ID model with augmented samples, either translating source samples to the target style or assigning pseudo labels to the target. The former methods may lose identity details but keep redundant source background during translation. In contrast, the latter techniques may give noisy labels when the model meets the unseen background and person pose. We mitigate the domain shift in the former translation direction by cyclically decoupling environment and identity-related features. We propose a novel individual-preserving and environmental-switching cyclic generation network (IPES-GAN). Our network has the following distinct features: 1)Decoupled features instead of fused features:we encode the images into an individual part and an environmental part, which are proved beneficial to generation and adaptation; 2)Cyclic generation instead of one-step adaptive generation. We swap source and target environment features to generate cross-domain images with preserved identity-related features conditioned with source (target) background features and then changed again to generate back the input image so that cyclic generation runs in a self-supervised way. Experiments carried out on two significant benchmarks: Market-1501 and DukeMTMC-Reid, reveal state-of-the-art performance.

Video frame interpolation has made great progress in estimating advanced optical flow and synthesizing in-between frames sequentially. However, frame interpolation involving various resolutions and motions remains challenging due to limited or fixed pre-trained networks. Inspired by the success of the coarse-to-fine scheme for video frame interpolation, i.e., gradually interpolating frames of different resolutions, we propose a progressive boosting network (ProBoost-Net) based on a multi-scale framework to achieve flexible recurrent scales and then gradually optimize optical flow estimation and frame interpolation. Specifically, we designed a dense motion boosting (DMB) module to transfer features close to real motion to the decoded features from the later scales, which provides complementary information to refine the motion further. Furthermore, to ensure the accuracy of the estimated motion features at each scale, we propose a motion adaptive fusion (MAF) module that adaptively deals with motions with different receptive fields according to the motion conditions. Thanks to the framework's flexible recurrent scales, we can customize the number of scales and make trade-offs between computation and quality depending on the application scenario. Extensive experiments with various datasets demonstrated the superiority of our proposed method over state-of-the-art approaches in various scenarios.

The large variation of viewpoint and irrelevant content around the target always hinder accurate image retrieval and its subsequent tasks. In this paper, we investigate an extremely challenging task: given a ground-view image of a landmark, we aim to achieve cross-view geo-localization by searching out its corresponding satellite-view images. Specifically, the challenge comes from the gap between ground-view and satellite-view, which includes not only large viewpoint changes (some parts of the landmark may be invisible from front view to top view) but also highly irrelevant background (the target landmark tend to be hidden in other surrounding buildings), making it difficult to learn a common representation or a suitable mapping. To address this issue, we take advantage of drone-view information as a bridge between ground-view and satellite-view domains. We propose a Peer Learning and Cross Diffusion (PLCD) framework. PLCD consists of three parts: 1) a peer learning across ground-view and drone-view to find visible parts to benefit ground-drone cross-view representation learning; 2) a patch-based network for satellite-drone cross-view representation learning; 3) a cross diffusion between ground-drone space and satellite-drone space. Extensive experiments conducted on the University-Earth and University-Google datasets show that our method outperforms state-of-the-arts significantly.

Mean Average Precision (mAP) is the primary evaluation measure for object detection. Although object detection has a broad range of applications, mAP evaluates detectors in terms of the performance of ranked instance retrieval. Such the assumption for the evaluation task does not suit some downstream tasks. To alleviate the gap between downstream tasks and the evaluation scenario, we propose Optimal Correction Cost (OC-cost), which assesses detection accuracy at image level. OC-cost computes the cost of correcting detections to ground truths as a measure of accuracy. The cost is obtained by solving an optimal transportation problem between the detections and the ground truths. Unlike mAp, OC-cost is designed to penalize false positive and false negative detections properly, and every image in a dataset is treated equally. Our experimental result validates that OCscost has better agreement with human preference than a ranking-based measure, i.e., mAP for a single image. We also show that detectors' rankings by OC-cost are more consistent on different data splits than mAP. Our goal is not to replace mAP with OC-cost but provide an additional tool to evaluate detectors from another aspect. To help future researchers and developers choose a target measure, we provide a series of experiments to clarify how mAP and OC-cost differ.

Most computer vision systems assume distortion-free images as inputs. The widely used rolling-shutter (RS) image sensors, however, suffer from geometric distortion when the camera and object undergo motion during capture. Extensive researches have been conducted on correcting RS distortions. However, most of the existing work relies heavily on the prior assumptions of scenes or motions. Besides, the motion estimation steps are either oversimplified or computationally inefficient due to the heavy flow warping, limiting their applicability. In this paper, we investigate using rolling shutter with a global reset feature (RSGR) to restore clean global shutter (GS) videos. This feature enables us to turn the rectification problem into a deblur-like one, getting rid of inaccurate and costly explicit motion estimation. First, we build an optic system that captures paired RSGR/GS videos. Second, we develop a novel algorithm incorporating spatial and temporal designs to correct the spatial-varying RSGR distortion. Third, we demonstrate that existing image-to-image translation algorithms can recover clean GS videos from distorted RSGR inputs, yet our algorithm achieves the best performance with the specific designs. Our rendered results are not only visually appealing but also beneficial to downstream tasks. Compared to the state-of-the-art RS solution, our RSGR solution is superior in both effectiveness and efficiency. Considering it is easy to realize without changing the hardware, we believe our RSGR solution can potentially replace the RS solution in taking distortion-free videos with low noise and low budget.

Existing learning-based image inpainting methods are still in challenge when facing complex semantic environments and diverse hole patterns. The prior information learned from the large scale training data is still insufficient for these situations. Reference images captured covering the same scenes share similar texture and structure priors with the corrupted images, which offers new prospects for the image inpainting tasks. Inspired by this, we first build a benchmark dataset containing 10K pairs of input and reference images for reference-guided inpainting. Then we adopt an encoder-decoder structure to separately infer the texture and structure features of the input image considering their pattern discrepancy of texture and structure during inpainting. A feature alignment module is further designed to refine these features of the input image with the guidance of a reference image. Both quantitative and qualitative evaluations demonstrate the superiority of our method over the state-of-the-art methods in terms of completing complex holes. Code is available at https://github.com/Cameltr/RGTSI.

Very Important Person Localization (VIPLoc) aims at detecting certain individuals in a given image, who are more attractive than others in the image. Existing uncontrolled VIPLoc benchmark assumes that the image has one single VIP, which is not suitable for actual application scenarios when multiple VIPs or no VIPs appear in the image. In this paper, we re-built a complex uncontrolled conditions (CUC) dataset to make the VIPLoc closer to the actual situation, containing no, single, and multiple VIPs. Existing methods use the hand-designed and deep learning strategies to extract the features of persons and analyze the differences between VIPs and other persons from the perspective of statistics. They are not explainable as to why the VIP located this output for that input. Thus, there exist the severe performance degradation when we use these models in real-world VIPLoc. Specifically, we establish a causal inference framework that unpacks the causes of previous methods and derives a new principled solution for VIPLoc. It treats the scene as confounding factor, allowing the ever-elusive confounding effects to be eliminated and the essential determinants to be uncovered. Through extensive experiments, our method outperforms the state-of-the-art methods on public VIPLoc datasets and the re-built CUC dataset.

Interpolating video frames involving large motions remains an elusive challenge. In case that frames involve small and fast-moving objects, conventional feed-forward neural network-based approaches that estimate optical flow and synthesize in-between frames sequentially often result in loss of motion features and thus blurred boundaries. To address the problem, we propose a novel Recurrent Motion-Enhanced Interpolation Network (ReMEI-Net) by assigning attention to the motion features of small objects from both the intra-scale and inter-scale perspectives. Specifically, we add recurrent feedback blocks in the existing multi-scale autoencoder pipeline, aiming to iteratively enhance the motion information of small objects across different scales. Second, to further refine the motion features of the highly moving objects, we propose a Multi-Directional ConvLSTM (MD-ConvLSTM) block to capture the global spatial contextual information of motion from multiple directions. In this way, the coarse-scale features can be utilized to correct and enhance the fine-scale features through the feedback mechanism. Extensive experiments on various datasets demonstrate the superiority of our proposed method over state-of-the-art approaches in terms of clear locations and complete shape.

Vehicle counting is important for smart city applications such as logistics management, traffic estimation, and financial analysis. To perform vehicle counting using aerial images, researchers have proposed many algorithms, including detection-based, regression-based and density-based methods. However, most of these algorithms are only applicable to high-resolution images, which require clear vehicle outlines. For the reasons of acquisition difficulty, frequency and cost, it is necessary to explore methods for vehicle counting using low-resolution or even very low-resolution images. We build a cross-resolution vehicle counting (CRVC) dataset, including 192 very low-resolution images and 8 high-resolution images of a port from 2016 to 2019. For this task, we propose a novel vehicle counting via cross-resolution spatial consistency and intra-resolution time continuity constraints. The segmentation map is first obtained by semantic segmentation with the prior information above. The vehicle coverage rate relative to the located parking lot is calculated and then converted to vehicle area. Finally, the relationship between the area and the number of vehicles is established by regression. Experiments show that the vehicle counting results obtained by our method are highly consistent with the annotations and outperform other state-of-the-art methods. Our method is also applicable for images with a lower resolution of 10m and other locations. Code, data and pre-trained models are available online at https://github.com/hbsszq/Vehicle-Counting-in-Very-Low-Resolution-Aerial-Images.

Understanding foggy image sequence in driving scene is critical for autonomous driving, but it remains a challenging task due to the difficulty in collecting and annotating real-world images of adverse weather. Recently, self-training strategy has been considered as a powerful solution for unsupervised domain adaptation, which iteratively adapts the model from the source domain to the target domain by generating target pseudo labels and re-training the model. However, the selection of confident pseudo labels inevitably suffers from the conflict between sparsity and accuracy, both of which will lead to suboptimal models. To tackle this problem, we exploit the characteristics of the foggy image sequence of driving scenes to densify the confident pseudo labels. Specifically, based on the two discoveries of local spatial similarity and adjacent temporal correspondence of the sequential image data, we propose a novel Target-Domain driven pseudo label Diffusion (TDo-Dif) scheme. It employs superpixels and optical flows to identify the spatial similarity and temporal correspondence, respectively, and then diffuses the confident but sparse pseudo labels within a superpixel or a temporal corresponding pair linked by the flow. Moreover, to ensure the feature similarity of the diffused pixels, we introduce local spatial similarity loss and temporal contrastive loss in the model re-training stage. Experimental results show that our TDo-Dif scheme helps the adaptive model achieve 51.92% and 53.84% mean intersection-over-union (mIoU) on two publicly available natural foggy datasets (Foggy Zurich and Foggy Driving), which exceeds the state-of-the-art unsupervised domain adaptive semantic segmentation methods. The proposed method can also be applied to non-sequential images in the target domain by considering only spatial similarity.

There are many types of retinal disease, and accurately detecting these diseases is crucial for proper diagnosis. Convolutional neural networks (CNNs) typically perform well on detection tasks, and the attention module of CNNs can generate heatmaps as visual explanations of the model. However, the generated heatmap can only detect the most discriminative part, which is problematic because many object regions may exist in the region beside the heatmap in an area known as a complementary heatmap. In this study, we developed a method specifically designed multi-retinal diseases detection from fundus images with the complementary heatmap. The proposed CAM-based method is designed for 2D color images of the retina, rather than MRI images or other forms of data. Moreover, unlike other visual images for disease detection, fundus images of multiple retinal diseases have features such as distinguishable lesion region boundaries, overlapped lesion regions between diseases, and specific pathological structures (e.g. scattered blood spots) that lead to mis-classifications. Based on these considerations, we designed two new loss functions, attention-explore loss and attention-refine loss, to generate accurate heatmaps. We select both "bad" and "good" heatmaps based on the prediction score of ground truth and train them with the two loss functions. When the detection accuracy increases, the classification performance of the model is also improved. Experiments on a dataset consisting of five diseases showed that our approach improved both the detection accuracy and the classification accuracy, and the improved heatmaps were closer to the lesion regions than those of current state-of-the-art methods.

This special issue provides a premier forum for researchers in multimedia big data to share challenges and recent advancements in learning from noisy multimedia data. The multimedia age and its proliferation of devices and platforms is fueling exponential data growth. As computational power and deep learning algorithms rapidly evolve, the web has become a rich source of potential training data for robust machine learning, with search engines such as Google and Bing, Twitter, TikTok, Instagram, and short video sharing platforms offering large-scale data points in the hundreds of millions. The concurrent shift in the Internet to richer web data modalities such as text, audio, image, and video reveal further opportunities to leverage large-scale data for the automatic construction of a variety of datasets for model training and testing. However, the ubiquity of multimedia data means noise is a fundamental challenge, with ‘label noise’ and ‘domain mismatch’ the most critical issues in automatically collected datasets. Learning from noisy multimedia data tends towards poor performance, making it increasingly essential to address these challenges.

Recent advances in person re-identification (re-ID) have led to impressive retrieval accuracy. However, existing re-ID models are challenged by the adversarial examples crafted by adding quasi-imperceptible perturbations. Moreover, re-ID systems face the domain shift issue that training and testing domains are not consistent. In this study, we argue that learning powerful attackers with high universality that works well on unseen domains is an important step in promoting the robustness of re-ID systems. Therefore, we introduce a novel universal attack algorithm called ``MetaAttack'' for person re-ID. MetaAttack can mislead re-ID models on unseen domains by a universal adversarial perturbation. Specifically, to capture common patterns across different domains, we propose a meta-learning scheme to seek the universal perturbation via the gradient interaction between meta-train and meta-test formed by two datasets. We also take advantage of a virtual dataset (PersonX), instead of real ones, to conduct meta-test. This scheme not only enables us to learn with more comprehensive variation factors but also mitigates the negative effects caused by biased factors of real datasets. Experiments on three large-scale re-ID datasets demonstrate the effectiveness of our method in attacking re-ID models on unseen domains. Our final visualization results reveal some new properties of existing re-ID systems, which can guide us in designing a more robust re-ID model. Code and supplemental material are available at \url{https://github.com/FlyingRoastDuck/MetaAttack_AAAI21}.

Existing inpainting methods have achieved promising performance in recovering defective images of specific scenes. However, filling holes involving multiple semantic categories remains challenging due to the obscure se-mantic boundaries and the mixture of different semantic textures. In this paper, we introduce coherence priors between the semantics and textures which make it possible to concentrate on completing separate textures in a semantic-wise manner. Specifically, we adopt a multi-scale joint optimization framework to first model the coherence priors and then accordingly interleaving optimize image inpainting and semantic segmentation in a coarse-to-fine manner. A Semantic-Wise Attention Propagation (SWAP) module is devised to refine completed image textures across scales by exploring non-local semantic coherence, which effectively mitigates the mix-up of textures. We also propose two coherence losses to constrain the consistency between the semantics and the inpainted image in terms of the overall structure and detailed textures. Experimental results demonstrate the superiority of our proposed method for challenging cases with complex holes.

Although the unsupervised discovery and localization of common objects from within a set of images has received considerable attention, the difficulty of this task means that current methods are not sufficiently accurate. This paper describes an unsupervised method that more accurately discovers and localizes common particular objects within a set of images. First, our method constructs a match graph that precisely represents the relationship among particular object proposals extracted from images. The match graph necessarily contains communities, i.e., subgraphs where object proposals with identical subjects are densely connected by edges, indicating matches between local image features in the proposals. Our method determines the common particular objects by scoring the object proposals using a novel node centrality measure for graphs with a community structure, which we call the restrained random-walk centrality. Experiments demonstrate that our method is considerably more accurate than previous approaches on a particular object image set.

Creating a high-quality layout design from scratch is difficult for novices. Therefore, novices often consult the works of other skilled designers for ideas regarding layout designs. Researchers have previously investigated methods to support the layout design process; these works mainly focused on retrieval methods for similar layout designs, or refinement of existing layouts. To enhance user creativity in designing layouts, assistance is needed for exploring various designs. Herein, we propose a novel deep generative model that enables the generation of various layout designs and guarantees continuous and gradual changes in layouts, for effectively exploring graphic designs. Accordingly, we present an adversarial training method with dual critic networks; we trained our model by a public graphic design dataset. We developed another interaction method that allows the user to change the graphic designs between two different layout styles and categories parametrically. We demonstrated the efficacy of the proposed method in generating rich layout variations with representation of latent space by comparing the layout designs generated by our model with by an existing model.

In addition to video sharing services such as YouTube, the spread of video-based social networking services such as TikTok and Instagram have led to the accumulation of vast amounts of video data. In this situation, it has become important to develop a technology to efficiently retrieve necessary information from the accumulated video data. In this paper, we propose a method to retrieve similar videos by focusing on people and their actions in the videos. We apply this method to various datasets and TV videos to demonstrate its usefulness. Demonstration video: \urlhttps://youtu.be/y86api-FXpU

The central research question of the nearest neighbor search is how to reduce the memory cost while maintaining its accuracy. Instead of compressing each vector as is done in the existing methods, we propose a way to subsample unnecessary vectors to save memory. We empirically found that such unnecessary vectors have low hubness scores and thus can be easily identified beforehand. Such points are called anti-hubs in the data mining community. By removing anti-hubs, we achieved a memory-efficient search while preserving accuracy. In million-scale experiments, we showed that any vector compression method improves search accuracy by partial replacement with anti-hub removal under the same memory usage. A billion-scale benchmark showed that our data reduction combined with the best search method achieves higher accuracy under the assumption of fixed memory consumption. For example, our method had a much higher [email protected] (0.53) compared with the existing method (0.23) for the same memory consumption (6GB).

This paper focuses on camouflaged object detection (COD), which is a task to detect objects hidden in the background. Most of the current COD models aim to highlight the target object directly while outputting ambiguous camouflaged boundaries. On the other hand, the performance of the models considering edge information is not yet satisfactory. To this end, we propose a new framework that makes full use of multiple visual cues, i.e., saliency as well as edges, to refine the predicted camouflaged map. This framework consists of three key components, i.e., a pseudo-edge generator, a pseudo-map generator, and an uncertainty-aware refinement module. In particular, the pseudo-edge generator estimates the boundary that outputs the pseudo-edge label, and the conventional COD method serves as the pseudo-map generator that outputs the pseudo-map label. Then, we propose an uncertainty-based module to reduce the uncertainty and noise of such two pseudo labels, which takes both pseudo labels as input and outputs an edge-accurate camouflaged map. Experiments on various COD datasets demonstrate the effectiveness of our method with superior performance to the existing state-of-the-art methods.

Virtual try-on systems became popular for visualizing outfits, due to the importance of individual fashion in many communities. The objective of such a system is to transfer a piece of clothing to another person while preserving its detail and characteristics. To generate a realistic in-the-wild image, it needs visual optimization of the clothing, background and target person, making this task still very challenging. In this paper, we develop a method that generates realistic try-on images with unpaired images from in-the-wild datasets. Our proposed method starts with generating a mock-up paired image using geometric transfer. Then, the target’s pose information is adjusted using a modified pose-attention module. We combine a reconstruction and a content loss to preserve the detail and style of the transferred clothing, background and the target person. We evaluate the approach on the Fashionpedia dataset and can show a promising performance over a baseline approach.

Learning from implicit feedback is challenging because of the difficult nature of the one-class problem: we can observe only positive examples. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. However, such methods have two main drawbacks particularly in large-scale applications; (1) the pairwise approach is severely inefficient due to the quadratic computational cost; and (2) even recent model-based samplers (e.g. IRGAN) cannot achieve practical efficiency due to the training of an extra model.

Solving cold-start problems is indispensable to provide meaningful recommendation results for new users and items. Under sparsely observed data, unobserved user-item pairs are also a vital source for distilling latent users' information needs. Most present works leverage unobserved samples for extracting negative signals. However, such an optimisation strategy can lead to biased results toward already popular items by frequently handling new items as negative instances. In this study, we tackle the cold-start problems for new users/items by appropriately leveraging unobserved samples. We propose a knowledge graph (KG)-aware recommender based on graph neural networks, which augments labelled samples through pseudo-labelling. Our approach aggressively employs unobserved samples as positive instances and brings new items into the spotlight. To avoid exhaustive label assignments to all possible pairs of users and items, we exploit a KG for selecting probably positive items for each user. We also utilise an improved negative sampling strategy and thereby suppress the exacerbation of popularity biases. Through experiments, we demonstrate that our approach achieves improvements over the state-of-the-art KG-aware recommenders in a variety of scenarios; in particular, our methodology successfully improves recommendation performance for cold-start users/items.

Learning from implicit user feedback is challenging as we can only observe positive samples but never access negative ones. Most conventional methods cope with this issue by adopting a pairwise ranking approach with negative sampling. However, the pairwise ranking approach has a severe disadvantage in the convergence time owing to the quadratically increasing computational cost with respect to the sample size; it is problematic, particularly for large-scale datasets and complex models such as neural networks. By contrast, a pointwise approach does not directly solve a ranking problem, and is therefore inferior to a pairwise counterpart in top-K ranking tasks; however, it is generally advantageous in regards to the convergence time. This study aims to establish an approach to learn personalised ranking from implicit feedback, which reconciles the training efficiency of the pointwise approach and ranking effectiveness of the pairwise counterpart. The key idea is to estimate the ranking of items in a pointwise manner; we first reformulate the conventional pointwise approach based on density ratio estimation and then incorporate the essence of ranking-oriented approaches (e.g. the pairwise approach) into our formulation. Through experiments on three real-world datasets, we demonstrate that our approach dramatically reduces the convergence time (one to two orders of magnitude faster) and significantly improves the ranking performance.

BACKGROUND: Unsupervised learning can discover various unseen abnormalities, relying on large-scale unannotated medical images of healthy subjects. Towards this, unsupervised methods reconstruct a 2D/3D single medical image to detect outliers either in the learned feature space or from high reconstruction loss. However, without considering continuity between multiple adjacent slices, they cannot directly discriminate diseases composed of the accumulation of subtle anatomical anomalies, such as Alzheimer's disease (AD). Moreover, no study has shown how unsupervised anomaly detection is associated with either disease stages, various (i.e., more than two types of) diseases, or multi-sequence magnetic resonance imaging (MRI) scans. RESULTS: We propose unsupervised medical anomaly detection generative adversarial network (MADGAN), a novel two-step method using GAN-based multiple adjacent brain MRI slice reconstruction to detect brain anomalies at different stages on multi-sequence structural MRI: (Reconstruction) Wasserstein loss with Gradient Penalty + 100 [Formula: see text] loss-trained on 3 healthy brain axial MRI slices to reconstruct the next 3 ones-reconstructs unseen healthy/abnormal scans; (Diagnosis) Average [Formula: see text] loss per scan discriminates them, comparing the ground truth/reconstructed slices. For training, we use two different datasets composed of 1133 healthy T1-weighted (T1) and 135 healthy contrast-enhanced T1 (T1c) brain MRI scans for detecting AD and brain metastases/various diseases, respectively. Our self-attention MADGAN can detect AD on T1 scans at a very early stage, mild cognitive impairment (MCI), with area under the curve (AUC) 0.727, and AD at a late stage with AUC 0.894, while detecting brain metastases on T1c scans with AUC 0.921. CONCLUSIONS: Similar to physicians' way of performing a diagnosis, using massive healthy training data, our first multiple MRI slice reconstruction approach, MADGAN, can reliably predict the next 3 slices from the previous 3 ones only for unseen healthy images. As the first unsupervised various disease diagnosis, MADGAN can reliably detect the accumulation of subtle anatomical anomalies and hyper-intense enhancing lesions, such as (especially late-stage) AD and brain metastases on multi-sequence MRI scans.

In this paper, we propose a restrained random-walk similarity method for detecting the community structures of graphs. The basic premise of our method is that the starting vertices of finite-length random walks are judged to be in the same community if the walkers pass similar sets of vertices. This idea is based on our consideration that a random walker tends to move in the community including the walker's starting vertex for some time after starting the walk. Therefore, the sets of vertices passed by random walkers starting from vertices in the same community must be similar. The idea is reinforced with two conditions. First, we exclude abnormal random walks. Random walks that depart from each vertex are executed many times, and vertices that are rarely passed by the walkers are excluded from the set of vertices that the walkers may pass. Second, we forcibly restrain random walks to an appropriate length. In our method, a random walk is terminated when the walker repeatedly visits vertices that they have already passed. Experiments on real-world networks demonstrate that our method outperforms previous techniques in terms of accuracy.

Group re-identification (G-ReID) is an important yet less-studied task. Its challenges not only lie in appearance changes of individuals, but also involve group layout and membership changes. To address these issues, the key task of G-ReID is to learn group representations robust to such changes. Nevertheless, unlike ReID tasks, there still lacks comprehensive publicly available G-ReID datasets, making it difficult to learn effective representations using deep learning models. In this article, we propose a Domain-Transferred Single and Couple Representation Learning Network (DotSCN). Its merits are two aspects: 1) Owing to the lack of labelled training samples for G-ReID, existing G-ReID methods mainly rely on unsatisfactory hand-crafted features. To gain the power of deep learning models in representation learning, we first treat a group as a collection of multiple individuals and propose transferring the representation of individuals learned from an existing labeled ReID dataset to a target G-ReID domain without a suitable training dataset. 2) Taking into account the neighborhood relationship in a group, we further propose learning a novel couple representation between two group members, that achieves better discriminative power in G-ReID tasks. In addition, we propose a weight learning method to adaptively fuse the domain-transferred individual and couple representations based on an L-shape prior. Extensive experimental results demonstrate the effectiveness of our approach that significantly outperforms state-of-the-art methods by 11.7% CMC-1 on the Road Group dataset and by 39.0% CMC-1 on the DukeMCMT dataset.

Most recent approaches for the zero-shot cross-modal image retrieval map images from different modalities into a uniform feature space to exploit their relevance by using a pre-trained model. Based on the observation that manifolds of zero-shot images are usually deformed and incomplete, we argue that the manifolds of unseen classes are inevitably distorted during the training of a two-stream model that simply maps images from different modalities into a uniform space. This issue directly leads to poor cross-modal retrieval performance. We propose a bi-directional random walk scheme to mining more reliable relationships between images by traversing heterogeneous manifolds in the feature space of each modality. Our proposed method benefits from intra-modal distributions to alleviate the interference caused by noisy similarities in the cross-modal feature space. As a result, we achieved great improvement in the performance of the thermal v.s. visible image retrieval task. The code of this paper: https://github.com/fyang93/cross-modal-retrieval

Retinal diseases encompass a variety of types, including different diseases and severity levels. Training a model with different types of disease is impractical. Dynamically training a model is necessary when a patient with a new disease appears. Deep learning techniques have stood out in recent years, but they suffer from catastrophic forgetting, i.e., a dramatic decrease in performance when new training classes appear. We found that keeping the feature distribution of an old model helps maintain the performance of incremental learning. In this paper, we design a framework named ``Attribute Driven Incremental Network" (ADINet), a new architecture that integrates class label prediction and attribute prediction into an incremental learning framework to boost the classification performance. With image-level classification, we apply knowledge distillation (KD) to retain the knowledge of base classes. With attribute prediction, we calculate the weight of each attribute of an image and use these weights for more precise attribute prediction. We designed attribute distillation (AD) loss to retain the information of base class attributes as new classes appear. This incremental learning can be performed multiple times with a moderate drop in performance. The results of an experiment on our private retinal fundus image dataset demonstrate that our proposed method outperforms existing state-of-the-art methods. For demonstrating the generalization of our proposed method, we test it on the ImageNet-150K-sub dataset and show good performance.

This paper introduces a feedback-based approach to interpolate video frames involving small and fast-moving objects. Unlike the existing feedforward-based methods that estimate optical flow and synthesize in-between frames sequentially, we introduce a motion-oriented component that adds a feedback block to the existing multi-scale autoencoder pipeline, which feedbacks information of small objects shared between architectures of two different scales. We show that feeding this additional information enables more robust detection of optical flow caused by small objects in fast motion. Using experiments on various datasets, we show that the feedback mechanism allows our method to achieve state-of-the-art results, both qualitatively and quantitatively.

A smile is a representative expression of happiness or high quality-of-life; however, automatic recognition of a smile according to happiness remains a challenging task. Because expressions of happiness are strongly dependent upon physical condition and occurrence of other emotions, and similar facial expression often occur under different emotions, we consider that there is no absolute visual pattern of a smile corresponding to happiness. Therefore, in this study, we assumed that a “smile with happiness” is observed as the temporal ascent in the degree of smiling and attempted to recognize this by capturing changes in facial expression within temporally sequential images. As an implementation of this scheme, we proposed a Siamese-structure deep neural network to compare facial expressions in two input images and estimate the existence of smile ascension or descension. For primal analysis of the proposed network, we developed a unique smiling dataset containing image pairs with various changes in smiling degree, including slight changes. The results demonstrated that the proposed method achieved nearly perfect recognition with > 0.95 accuracy when recognizing changes in the degree of smiling that humans certainly recognize. Attention regions that contributed to the predicted labels were concentrated on the mouth, cheeks, and tail of the eyes, which indicates a reasonable function for recognizing changes in smiling degree was constructed by the proposed method.

Fashion analysis is an attractive domain for vision research due to its direct applications in e-commerce contexts. However, fashion datasets are commonly rather demanding, as both objects and attributes tend to be fine-grained and thus result in very long-tailed datasets. Furthermore, relationships between objects and attributes are often dense, but are crucial for the performance of fashion applications. In this paper, we propose to generate scene graphs for existing fashion datasets. By detecting relationships between fashion objects, their parts, and their attributes we gain a better understanding of the scenes. As no current fashion dataset provides scene graphs, we generate relationships between fashion objects from existing annotations. The output is post-processed and filtered to generate a meaningful scene graph for each image. In the experiments, we can show existing applications like image retrieval benefiting from the scene graph understanding. We first evaluate the accuracy of the generated scene graphs. Then, we employ scene graphs to fashion image retrieval in order to showcase their performance in real applications. The results show various benefits for fashion applications by exploiting scene graph knowledge. The resources and model for the proposed method are available on GitHub1.

Pedestrian detection at nighttime is a crucial and frontier problem in surveillance, but has not been well explored by the computer vision and artificial intelligence communities. Most of existing methods detect pedestrians under favorable lighting conditions (e.g. daytime) and achieve promising performances. In contrast, they often fail under unstable lighting conditions (e.g. nighttime). Night is a critical time for criminal suspects to act in the field of security. The existing nighttime pedestrian detection dataset is captured by a car camera, specially designed for autonomous driving scenarios. The dataset for nighttime surveillance scenario is still vacant. There are vast differences between autonomous driving and surveillance, including viewpoint and illumination. In this paper, we build a novel pedestrian detection dataset from the nighttime surveillance aspect: NightSurveillance1. As a benchmark dataset for pedestrian detection at nighttime, we compare the performances of state-of-the-art pedestrian detectors and the results reveal that the methods cannot solve all the challenging problems of NightSurveillance. We believe that NightSurveillance can further advance the research of pedestrian detection, especially in the field of surveillance security at nighttime.

An efficient and effective person re-identification (ReID) system relieves the users from painful and boring video watching and accelerates the process of video analysis. Recently, with the explosive demands of practical applications, a lot of research efforts have been dedicated to heterogeneous person re-identification (Hetero-ReID). In this paper, we provide a comprehensive review of state-of-the-art Hetero-ReID methods that address the challenge of inter-modality discrepancies. According to the application scenario, we classify the methods into four categories --- low-resolution, infrared, sketch, and text. We begin with an introduction of ReID, and make a comparison between Homogeneous ReID (Homo-ReID) and Hetero-ReID tasks. Then, we describe and compare existing datasets for performing evaluations, and survey the models that have been widely employed in Hetero-ReID. We also summarize and compare the representative approaches from two perspectives, i.e., the application scenario and the learning pipeline. We conclude by a discussion of some future research directions. Follow-up updates are available at https://github.com/lightChaserX/Awesome-Hetero-reID

Instance Re-identification (ReID) system facilitates various applications that require painful and boring video watching. Its efficiency and effectiveness accelerate the process of video analysis. In this tutorial, we summarize ReID technologies and provide an overview. We'll introduce fundamental technologies, existing challenges, trends, etc. This tutorial would be useful for multimedia content analysis and system-level multimedia retrieval, especially for an effective and efficient open-world ReID system for the practical, large-scale, and open-set domain.

Unsupervised crowd counting is a challenging yet not largely explored task. In this paper, we explore it in a transfer learning setting where we learn to detect and count persons in an unlabeled target set by transferring bi-knowledge learnt from regression- and detection-based models in a labeled source set. The dual source knowledge of the two models is heterogeneous and complementary as they capture different modalities of the crowd distribution. We formulate the mutual transformations between the outputs of regression- and detection-based models as two scene-agnostic transformers which enable knowledge distillation between the two models. Given the regression- and detection-based models and their mutual transformers learnt in the source, we introduce an iterative self-supervised learning scheme with regression-detection bi-knowledge transfer in the target. Extensive experiments on standard crowd counting benchmarks, ShanghaiTech, UCF_CC_50, and UCF_QNRF demonstrate a substantial improvement of our method over other state-of-the-arts in the transfer learning setting.

Although person Re-Identification (ReID) is widely applied in a variety of multimedia systems, most of its essentially multifaceted output is evaluated and visualized inflexibly only using a list of images ranked by the similarity of image content, while the correlations between samples of different IDs and the spatial-temporal features of the images are underinvestigated. As system operators need a comfortable access to these important elements, we introduce an interactive design of a person ReID system to visualize these quantities. We demonstrate that a system offering these visual representations can effectively expedite and improve a person re-identification analysis and make it a much user-friendly experience.

Person re-identification has received much attention in the last few years, as it enhances the retrieval effectiveness in the video surveillance networks and video archive management. In this paper, we demonstrate a guiding robot with person followers system, which recognizes the follower using a person re-identification technology. It first adopts existing face recognition and person tracking methods to generate person tracklets with different IDs. Then, a classic person re-identification model, pre-trained on the surveillance dataset, is adapted to the new robot vision condition incrementally. The demonstration showcases the quality of robot follower focusing.

Visible-infrared person re-identification (RGB-IR ReID) is extremely important for the surveillance applications under poor illumination conditions. Since the difference in the feature representations not only lies in the person' pose, viewpoint or illumination variations, but also comes from huge spectrum discrepancy, the task becomes practically very challenging. Existing RGB-IR ReID models focus on bridging the gap between RGB and IR images through shared feature embedding, subspace learning or via adversarial learning. However, these methods do not explicitly disregard the spectrum information which is otherwise irrelevant for ReID. Further, adversarial learning methods has less promising convergence. This motivates us to design a non-adversarial and fast disentanglement method to disentangle the spectrum information while learning the identity discriminative features. To extract these features, we propose a novel network with disentanglement loss which can distill identity features and dispel spectrum features. Our network has two branches, spectrum dispelling and spectrum distilling branch. On spectrum dispelling branch, we apply identification loss to learn the identity related and spectrum disentangled features. On spectrum distilling branch, we apply an identity-dispeller loss to fool the identity classifier so that it primarily learns spectrum related information. The entire network is trained in an end-to-end manner, which minimizes spectrum information and maximizes invariant identity relevant information at spectrum dispelling branch. Extensive experiments on existing datasets demonstrate the superior performance of our approach compared to the state-of-the-art.

Huge earth observatory video data (EOVD) and the limited transmission bandwidth from satellites to terrestrial devices pose serious challenges to compression efficiency for satellite video. In this work, we deeply explore long-term background redundancy caused by periodical satellite revisit, and make utilization of long-term background reference (LTBR) to design a high-efficiency coding method specific for EOVD. Firstly, we use data of Google Earth as prior background knowledge and construct LTBR from it. Then, two novel prediction methods are proposed to make full use of prior information of LTBR, including color and definition correction based inter prediction with LTBR (CDIP-LTBR), structure and texture constrained intra prediction with LTBR (STIP-LTBR). Lastly, the proposed two new prediction schemes are integrated into one unified coding framework along with HEVC to achieve improved coding performance, and an improved RDO method is designed for additional prediction modes selection problem in the framework to obtain a higher prediction efficiency. Extensive experiments on real-world EOVD show that the proposed coding scheme exhibit significant improvement over HEVC and H.264, in terms of BD-Rate, BD-PSNR and rate distortion comparison.

Person re-identification (Re-ID) aims at matching person images captured in non-overlapping camera views. To represent person appearance, low-level visual features are sensitive to environmental changes, while high-level semantic attributes, such as "short-hair" or "long-hair", are relatively stable. Hence, researches have started to design semantic attributes to reduce the visual ambiguity. However, to train a prediction model for semantic attributes, it requires plenty of annotations, which are hard to obtain in practical large-scale applications. To alleviate the reliance on annotation efforts, we propose to incrementally generate Deep Hidden Attribute (DHA) based on baseline deep network for newly uncovered annotations. In particular, we propose an auto-encoder model that can be plugged into any deep network to mine latent information in an unsupervised manner. To optimize the effectiveness of DHA, we reform the auto-encoder model with additional orthogonal generation module, along with identity-preserving and sparsity constraints. 1) Orthogonally generating: In order to make DHAs different from each other, Singular Vector Decomposition (SVD) is introduced to generate DHAs orthogonally. 2) Identity-preserving constraint: The generated DHAs should be distinct for telling different persons, so we associate DHAs with person identities. 3) Sparsity constraint: To enhance the discriminability of DHAs, we also introduce the sparsity constraint to restrict the number of effective DHAs for each person. Experiments conducted on public datasets have validated the effectiveness of the proposed network. On two large-scale datasets, i.e., Market-1501 and DukeMTMC-reID, the proposed method outperforms the state-of-the-art methods.

Image classification using convolutional neural networks (CNNs) outperforms other state-of-the-art methods. Moreover, attention can be visualized as a heatmap to improve the explainability of results of a CNN. We designed a framework that can generate heatmaps reflecting lesion regions precisely. We generated initial heatmaps by using a gradient-based classification activation map (Grad-CAM). We assume that these Grad-CAM heatmaps correctly reveal the lesion regions; then we apply the attention mining technique to these heatmaps to obtain integrated heatmaps. Moreover, we assume that these Grad-CAM heatmaps incorrectly reveal the lesion regions and design a dissimilarity loss to increase their discrepancy with the Grad-CAM heatmaps. In this study, we found that having professional ophthalmologists select 30% of the heatmaps covering the lesion regions led to better results, because this step integrates (prior) clinical knowledge into the system. Furthermore, we design a knowledge preservation loss that minimizes the discrepancy between heatmaps generated from the updated CNN model and the selected heatmaps. Experiments using fundus images revealed that our method improved classification accuracy and generated attention regions closer to the ground truth lesion regions in comparison with existing methods.

Most person re-identification (ReID) approaches assume that person images are captured under relatively similar illumination conditions. In reality, long-term person retrieval is common, and person images are often captured under different illumination conditions at different times across a day. In this situation, the performances of existing ReID models often degrade dramatically. This paper addresses the ReID problem with illumination variations and names it as Illumination-Adaptive Person Re-identification (IA-ReID). We propose an Illumination-Identity Disentanglement (IID) network to dispel different scales of illuminations away while preserving individuals' identity information. To demonstrate the illumination issue and to evaluate our model, we construct two large-scale simulated datasets with a wide range of illumination variations. Experimental results on the simulated datasets and real-world images demonstrate the effectiveness of the proposed framework.

Diffusion is commonly used as a ranking or re-ranking method in retrieval tasks to achieve higher retrieval performance, and has attracted lots of attention in recent years. A downside to diffusion is that it performs slowly in comparison to the naive k-NN search, which causes a non-trivial online computational cost on large datasets. To overcome this weakness, we propose a novel diffusion technique in this paper. In our work, instead of applying diffusion to the query, we precompute the diffusion results of each element in the database, making the online search a simple linear combination on top of the k-NN search process. Our proposed method becomes 10∼ times faster in terms of online search speed. Moreover, we propose to use late truncation instead of early truncation in previous works to achieve better retrieval performance.

Accurate Computer-Assisted Diagnosis, associated with proper data wrangling, can alleviate the risk of overlooking the diagnosis in a clinical environment. Towards this, as a Data Augmentation (DA) technique, Generative Adversarial Networks (GANs) can synthesize additional training data to handle the small/fragmented medical imaging datasets collected from various scanners; those images are realistic but completely different from the original ones, filling the data lack in the real image distribution. However, we cannot easily use them to locate disease areas, considering expert physicians' expensive annotation cost. Therefore, this paper proposes Conditional Progressive Growing of GANs (CPGGANs), incorporating highly-rough bounding box conditions incrementally into PGGANs to place brain metastases at desired positions/sizes on 256 X 256 Magnetic Resonance (MR) images, for Convolutional Neural Network-based tumor detection; this first GAN-based medical DA using automatic bounding box annotation improves the training robustness. The results show that CPGGAN-based DA can boost 10% sensitivity in diagnosis with clinically acceptable additional False Positives. Surprisingly, further tumor realism, achieved with additional normal brain MR images for CPGGAN training, does not contribute to detection performance, while even three physicians cannot accurately distinguish them from the real ones in Visual Turing Test.

Infrared-Visible person RE-IDentification (IV-REID) is a rising task. Compared to conventional person re-identification (re-ID), IV-REID concerns the additional modality discrepancy originated from the different imaging processes of spectrum cameras, in addition to the person's appearance discrepancy caused by viewpoint changes, pose variations and deformations presented in the conventional re-ID task. The co-existed discrepancies make IV-REID more difficult to solve. Previous methods attempt to reduce the appearance and modality discrepancies simultaneously using feature-level constraints. It is however difficult to eliminate the mixed discrepancies using only feature-level constraints. To address the problem, this paper introduces a novel Dual-level Discrepancy Reduction Learning (D$^2$RL) scheme which handles the two discrepancies separately. For reducing the modality discrepancy, an image-level sub-network is trained to translate an infrared image into its visible counterpart and a visible image to its infrared version. With the image-level sub-network, we can unify the representations for images with different modalities. With the help of the unified multi-spectral images, a feature-level sub-network is trained to reduce the remaining appearance discrepancy through feature embedding. By cascading the two sub-networks and training them jointly, the dual-level reductions take their responsibilities cooperatively and attentively. Extensive experiments demonstrate the proposed approach outperforms the state-of-the-art methods.

In this paper we present a novel approach to improve temporal matching kernel (TMK) for video retrieval tasks. TMK has the ability to align videos during retrieval, but provides little to none retrieval performance improvement over baseline methods. We discovered that TMK cannot discriminate between a true match case in which two videos have long, consecutive segments of similar frames and a false match case in which two videos contain non-consecutive segments of randomly similar frames. Our proposed burst-survive temporal matching kernel adopts a novel shuffle strategy to rule out false match cases, with the assistance of multiple periods selected from Fibonacci series. As a result, we achieved significant performance improvement on the EVVE dataset.

Most person re-identification (ReID) approaches focus on retrieving a person-of-interest from a database of collected individual images. In addition to the individual ReID task, matching a group of persons across different camera views also plays an important role in surveillance applications. This kind of Group Re-identification (GReID) task is very challenging since we face the obstacles not only from the appearance changes of individuals, but also from the group layout and membership changes. In order to obtain robust representation for the group image, we design a Domain-Transferred Graph Neural Network (DoT-GNN) method. The merits are three aspects: 1) Transferred Style. Due to the lack of training samples, we transfer the labeled ReID dataset to the G-ReID dataset style, and feed the transferred samples to the deep learning model. Taking the superiority of deep learning models, we achieve a discriminative individual feature model. 2) Graph Generation. We treat a group as a graph, where each node denotes the individual feature and each edge represents the relation of a couple of individuals. We propose a graph generation strategy to create sufficient graph samples. 3) Graph Neural Network. Employing the generated graph samples, we train the GNN so as to acquire graph features which are robust to large graph variations. The key to the success of DoT-GNN is that the transferred graph addresses the challenge of the appearance change, while the graph representation in GNN overcomes the challenge of the layout and membership change. Extensive experimental results demonstrate the effectiveness of our approach, outperforming the state-of-the-art method by 1.8% CMC-1 on Road Group dataset and 6.0% CMC-1 on DukeMCMT dataset respectively.

It is common that TV audiences want to quickly browse scenes with certain actors in TV series. Since 2016, the TREC Video Retrieval Evaluation (TRECVID) Instance Search (INS) task has started to focus on identifying a target person in a target scene simultaneously. In this paper, we name this kind of task as P-S INS (Person-Scene Instance Search). To find out P-S instances, most approaches search person and scene separately, and then directly combine the results together by addition or multiplication. However, we find that person and scene INS modules are not always effective at the same time, or they may suppress each other in some situations. Aggregating the results shot after shot is not a good choice. Luckily, for the TV series, video shots are arranged in chronological order. We extend our focus from time point (single video shot) to time slice (multiple consecutive video shots) in the time-line. Through detecting salient time slices, we prune the data. Through evaluating the importance of salient time slices, we boost the aggregation results. Extensive experiments on the large-scale TRECVID INS dataset demonstrate the effectiveness of the proposed method.

Person re-identification is widely applied in video surveillance and criminal investigation applications. To achieve better performance, an additional re-ranking step is often exploited. Related methods attempt to optimize the result according to every single query independently. However, in a practical scene, as the investigation process goes on, the other queries, in particular, the gradually accumulated logs, can be used to guide or regularize the current query. In this paper, we propose to optimize the result according to not only the current query itself but also the other queries and historical logs. We respectively investigate the cross-direction and the cross-ranking constraints among different queries. Based on the investigations, we propose a reciprocal optimization method to refine multiple ranking lists reciprocally. Experiments on the VIPeR, new-protocol CUHK03, and Market-1501 datasets confirm the effectiveness of our method. In particular, on the Market-1501 dataset, with full utilization of the other queries, the method achieves an accuracy rate of 94.66% at rank-1 and a very high mAP of 75.12%, and significantly outperforms the state-of-the-art methods.

Advances in multimedia analysis, enables access and indexing of huge and rich video content. However, delivering this heterogeneous content remains a challenge. We proposes the use of multiplex networks to combine both textual and visual semantic cues extracted from TV news videos for interactive search refinement on heterogeneous data. After preprocessing, the indexed videos can be queried, for which result space is summarized in a visual cloud helping query refinement with contextual information by combining visual and textual cues. We leverage on properties of multiplex networks to extract a hierarchy of layers, which not only contextualizes results, but also guides the construction of our visual cloud, and its interactions.

Thanks to the success of object detection technology, we can retrieve objects of the specified classes even from huge image collections.However, the current state-of-the-art object detectors (such as Faster R-CNN) can only handle pre-specified classes.In addition, large amounts of positive and negative visual samples are required for training.In this paper, we address the problem of open-vocabulary object retrieval and localization, where the target object is specified by a textual query (e.g., a word or phrase).We first propose Query-Adaptive R-CNN, a simple extension of Faster R-CNN adapted to open-vocabulary queries, by transforming the text embedding vector into an object classifier and localization regressor.Then, for discriminative training, we then propose negative phrase augmentation (NPA) to mine hard negative samples which are visually similar to the query and at the same time semantically mutually exclusive of the query.The proposed method can retrieve and localize objects specified by a textual query from one million images in only 0.5 seconds with high precision.

In this work we propose tracking as a generic addition to the instance search task. From video data perspective, much information that can be used is not taken into account in the traditional instance search approach. This work aims to provide insights on exploiting such existing information by means of tracking and the proper combination of the results, independently of the instance search system. We also present a study on the improvement of the system when using multiple independent instances (up to 4) of the same person. Experimental results show that our system improves substantially its performance when using tracking. Best configuration improves from mAP = 0.447 to mAP = 0.511 for a single example, and from mAP = 0.647 to mAP = 0.704 for multiple (4) given examples.

Matrix multiplication is a fundamental operation for data analysis. Memory-efficient matrix multiplication is always essential, especially for problems involving large-scale datasets. We present PQMM, a column-wise product quantization for memory-efficinet matrix multiplication. A simple idea of column-wise vector quantization is revisited. Each column of a data matrix is quantized into an index and the matrix product is approximated by the sum of the outer products obtained from the codewords fetched by the indices. The empirical evaluation illustrates that, with the same amount of memory, PQMM outperforms the state-of-the-art sketch-based method with respect to the error. We further show that a combination of the sketch-based method and PQMM results in a slightly less accurate data representation, but which is significantly more memory-efficient.

Person re-identification (REID) is an important task in video surveillance and forensics applications. Most of previous approaches are based on a key assumption that all person images have uniform and sufficiently high resolutions. Actually, various low-resolutions and scale mismatching always exist in open world REID. We name this kind of problem as Scale-Adaptive Low Resolution Person Re-identification (SALR-REID). The most intuitive way to address this problem is to increase various low-resolutions (not only low, but also with different scales) to a uniform high-resolution. SR-GAN is one of the most competitive image super-resolution deep networks, designed with a fixed upscaling factor. However, it is still not suitable for SALR-REID task, which requires a network not only synthesizing high-resolution images with different upscaling factors, but also extracting discriminative image feature for judging person’s identity. (1) To promote the ability of scale-adaptive upscaling, we cascade multiple SRGANs in series. (2) To supplement the ability of image feature representation, we plug-in a reidentification network. With a unified formulation, a Cascaded Super-Resolution GAN (CSR-GAN) framework is proposed. Extensive evaluations on two simulated datasets and one public dataset demonstrate the advantages of our method over related state-of-the-art methods.

Speechreading or lipreading is the technique of understanding and getting phonetic features from a speaker's visual features such as movement of lips, face, teeth and tongue. It has a wide range of multimedia applications such as in surveillance, Internet telephony, and as an aid to a person with hearing impairments. However, most of the work in speechreading has been limited to text generation from silent videos. Recently, research has started venturing into generating (audio) speech from silent video sequences but there have been no developments thus far in dealing with divergent views and poses of a speaker. Thus although, we have multiple camera feeds for the speech of a user, but we have failed in using these multiple video feeds for dealing with the different poses. To this end, this paper presents the world's first ever multi-view speech reading and reconstruction system. This work encompasses the boundaries of multimedia research by putting forth a model which leverages silent video feeds from multiple cameras recording the same subject to generate intelligent speech for a speaker. Initial results confirm the usefulness of exploiting multiple camera views in building an efficient speech reading and reconstruction system. It further shows the optimal placement of cameras which would lead to the maximum intelligibility of speech. Next, it lays out various innovative applications for the proposed system focusing on its potential prodigious impact in not just security arena but in many other multimedia analytics problems.

Existing approximate nearest neighbor search systems suffer from two fundamental problems that are of practical importance but have not received sufficient attention from the research community. First, although existing systems perform well for the whole database, it is difficult to run a search over a subset of the database. Second, there has been no discussion concerning the performance decrement after many items have been newly added to a system. We develop a reconfigurable inverted index (Rii) to resolve these two issues. Based on the standard IVFADC system, we design a data layout such that items are stored linearly. This enables us to efficiently run a subset search by switching the search method to a linear PQ scan if the size of a subset is small. Owing to the linear layout, the data structure can be dynamically adjusted after new items are added, maintaining the fast speed of the system. Extensive comparisons show that Rii achieves a comparable performance with state-of-the art systems such as Faiss.

One task key to the analysis of large multimedia archive over time is to dynamically monitor the activity of concepts and entities with their interactions. This is helpful to analyze threads of topics over news archives (how stories unfold), or to monitor evolutions and development of social groups. Dynamic graph modeling is a powerful tool to capture these interactions over time, while visualization and finding communities still remain difficult, especially with a high density of links. We propose to extract the backbone of dynamic graphs in order to ease community detection and guide the exploration of trends evolution. Through the graph structure, we interactively coordinate node-link diagrams, Sankey diagrams, time series, and animations in order to extract patterns and follow community behavior. We illustrate our system with the exploration of the role of soccer in 6 years of TV/radio magazines in France, and the role of North Korea in about 10 years of Japanese news.

Person re-identifcation is a key technique to match person images captured in non-overlapping camera views. Due to the sensitivity of visual features to environmental changes, semantic attributes, such as "short-hair" or "long-hair", begin to be investigated to represent person's appearance to improve the re-identifcation performance. Generally, training semantic attribute representations requires massive annotated samples, which limits the applicability on the large-scale practical applications. To alleviate the reliance on annotation efforts, we propose a new person representation with hidden attributes by mining latent information from visual feature in an unsupervised manner. In particular, an auto-encoder model is plugged-in to the deep learning network to compose a Deep Hidden Attribute Network (DHA-Net). The learnt hidden attribute representation preserves the robustness of semantic attributes and simultaneously inherits the discrimination ability of visual features. Experiments conducted on public datasets have validated the effectiveness of DHA-Net. On two large-scale datasets, i.e., Market-1501 and DukeMTMC-reID, the proposed method outperforms the state-of-the-art methods.

Person reidentification (re-id), as an important task in video surveillance and forensics applications, has been widely studied. Previous research efforts toward solving the person re-id problem have primarily focused on constructing robust vector description by exploiting appearance's characteristic, or learning discriminative distance metric by labeled vectors. Based on the cognition and identification process of human, we propose a new pattern, which transforms the feature description from characteristic vector to discrepancy matrix. In particular, in order to well identify a person, it converts the distance metric from vector metric to matrix metric, which consists of the intradiscrepancy projection and interdiscrepancy projection parts. We introduce a consistent term and a discriminative term to form the objective function. To solve it efficiently, we utilize a simple gradient-descent method under the alternating optimization process with respect to the two projections. Experimental results on public datasets demonstrate the effectiveness of the proposed pattern as compared with the state-of-the-art approaches.

Every day, vast amounts of data are uploaded to various social-sharing websites. Each social-sharing website has its own media dataset. Recently, mining media datasets has shown great potential for our daily lives, e.g., earthquake detection. Generally, different datasets have different characteristics. Combining different datasets is capable of achieving better performance than using any dataset independently, particularly if the datasets can compensate for each other. The resulting performance, however, depends on the fusion method. Effectively combining different datasets is challenging. As a solution to this challenge, this paper presents a generic two-stage framework for events of interest. Specifically, the first stage normalizes the contents of different datasets to make them comparable; then, the second stage combines the normalized contents for a ranked event list using graph-based algorithms. Practically, this paper unifies a flow-based media dataset and a check-in-based media dataset. Based on the precision for the top n events, the experimental results demonstrate that the proposed framework can achieve better performance in finding events associated with sports, local festivals, concerts, and exhibitions compared with a state-of-the-art approach that uses one dataset alone.

Recent years have witnessed the growing popularity of hashing for large-scale multimedia retrieval. Extensive hashing methods have been designed for data stored in a single machine, that is, centralized hashing . In many real-world applications, however, the large-scale data are often distributed across different locations, servers, or sites. Although hashing for distributed data can be implemented by assembling all distributed data together as a whole dataset in theory, it usually leads to prohibitive computation, communication, and storage costs in practice. Up to now, only a few methods were tailored for distributed hashing, which are all unsupervised approaches. In this paper, we propose an efficient and effective method called supervised distributed hashing (SupDisH), which learns discriminative hash functions by leveraging the semantic label information in a distributed manner. Specifically, we cast the distributed hashing problem into the framework of classification, where the learned binary codes are expected to be distinct enough for semantic retrieval. By introducing auxiliary variables, the distributed model is then separated into a set of decentralized subproblems with consistency constraints, which can be solved in parallel on each vertex of the distributed network. As such, we can obtain high-quality distinctive unbiased binary codes and consistent hash functions with low computational complexity, which facilitate tackling large-scale multimedia retrieval tasks involving distributed datasets. Experimental evaluations on three large-scale datasets show that SupDisH is competitive to centralized hashing methods and outperforms the state-of-the-art unsupervised distributed method significantly.

This paper addresses the problem of joint detection and recounting of abnormal events in videos. Recounting of abnormal events, i.e., explaining why they are judged to be abnormal, is an unexplored but critical task in video surveillance, because it helps human observers quickly judge if they are false alarms or not. To describe the events in the human-understandable form for event recounting, learning generic knowledge about visual concepts (e.g., object and action) is crucial. Although convolutional neural networks (CNNs) have achieved promising results in learning such concepts, it remains an open question as to how to effectively use CNNs for abnormal event detection, mainly due to the environment-dependent nature of the anomaly detection. In this paper, we tackle this problem by integrating a generic CNN model and environment-dependent anomaly detectors. Our approach first learns CNN with multiple visual tasks to exploit semantic information that is useful for detecting and recounting abnormal events. By appropriately plugging the model into anomaly detectors, we can detect and recount abnormal events while taking advantage of the discriminative power of CNNs. Our approach outperforms the state-of-the-art on Avenue and UCSD Ped2 benchmarks for abnormal event detection and also produces promising results of abnormal event recounting.

The technology used to detect community structures in graphs, or graph clustering technology, is important in a wide range of disciplines, such as sociology, biology, and computer science. Previously, many successful community detection methods have relied on the optimization of a quantity referred to as modularity, which is a quality index for the partition of a graph into communities. However, such methods suffer from a key drawback, namely, the inability to identify relatively small communities. To overcome this drawback, we propose a novel community detection method that can detect small communities. This is based on the property that a random walker will not readily leave a community even if it is small. The work presented in this paper demonstrates that our method detects both small and large communities in the practical application of clustering tourist attraction images obtained from Flickr.

We propose a fast event retrieval method in video databases based on temporal match kernel. The basic idea of the baseline is to maximize the score function for all possible relative timestamps. However, considering the stability and computational cost, we simplify the similarity score by calculating the energy of the score function. In this way, maximizing the score function which is time-consuming and sensitive to noise is avoided. We derive the simplified energy formulation by using Parseval's theorem, which describes the unitarity of Fourier transform. Further, by our formulation, the problem becomes a simple nearest neighbor search, and we can use product quantization (PQ) to accelerate the computation. We evaluate our method on EVVE dataset for event retrieval. The experimental results show that our approach is much faster than the baseline, with a higher mAP as well. Comparison with state of the art demonstrates the efficacy of our approach.

Politicians in the same political party often share the same views on social issues and legislative agendas. By mining patterns in TV news co-appearances and Twitter followers, in this paper we estimate political leanings (left / right) of unknown individuals, and detect outlier politicians who have views different from their colleagues in the same party, from a graph signal processing (GSP) perspective. Specifically, we first construct a similarity graph with politicians as nodes, where a positive edge connects two politicians with sizable shared Twitter followers, and a negative edge connects two politicians appearing in the same TV news segment (and thus likely take opposite stands on the same issue). Given a graph with both positive and negative edges, we propose a new graph-signal smoothness prior based on a constructed generalized graph Laplacian matrix that is guaranteed to be positive semi-definite. We formulate a graph-signal restoration problem that can be solved in closed form. Experimental results show that political leanings of unknown individuals can be reliably estimated and outlier politicians can be detected.

Kotenseki is a collection of classical and ancient Japanese literature. It is comprised of image books that express Japanese stories by using comic drawings of different characters, such as humans, nature, and animals. To effectively store them for posterity, a search system is important. We propose an efficient CBIR system to assist the users in easily accessing the information and have an enjoyable experience browsing Kotenseki images. There are two main functions comprising keyword-based and image-based queries. We also provide automatic detection of objects within the original images to create a database of feature vectors. Our study utilizes the benefits of deep-feature and object-detection techniques. We also present our fine-tuned model, which adds L2 normalization, dropout, and data augmentation to obtain better accuracy, which can reach 84% of mAP in our Kotenseki dataset.

This paper presents a method that embeds stability-sensitive filter in the temporal matching kernel with explicit feature maps (TE). The added filter improves the robustness of TE to noise for content-based video retrieval. Originally TE embeds temporal information of frame descriptors by using explicit feature mapping in a fixed length video vector by using a temporal invariant match kernel. TE matches relevant videos and at the same time estimates the temporal offset by referring only to these vectors. It has achieved state-of-the-art performance in video event retrieval. However, since TE defines video similarity as a sum of frame-wise similarities, we argue that TE is not so robust to noise. Usually matching videos have consecutive frames in correspondence, which result in series of high similarities. A simple sum of frame-wise similarities cannot differentiate this situation from that of non-consecutive high similarities which seems to lead to false matches. To circumvent this problem, we propose to apply a non-linear low-pass filter to the frame-wise similarities. In order to enjoy the benefit of the TE framework, we further develop a technique to embed the filter into frame descriptors. Consequently the video similarity can be evaluated as the dot product of video vectors with a stability-sensitive filter enabled. As the original TE, our method is seamlessly compatible with query expansion with temporal consistency checking. Our approach is evaluated on the EVVE dataset for particular event retrieval, it achieves state-of-the-art performance.

There has been a tremendous growth in video data the last decade. People are using mobile phones and tablets to take, share or watch videos more than ever before. Video cameras are around us almost everywhere in the public domain (e.g. stores, streets, public facilities, ...etc). Efficient and effective retrieval methods are critically needed in different applications. The goal of TRECVID is to encourage research in content-based video retrieval by providing large test collections, uniform scoring procedures, and a forum for organizations interested in comparing their results. In this tutorial, we present and discuss some of the most important and fundamental content-based video retrieval problems such as recognizing predefined visual concepts, searching in videos for complex ad-hoc user queries, searching by image/video examples in a video dataset to retrieve specific objects, persons, or locations, detecting events, and finally bridging the gap between vision and language by looking into how can systems automatically describe videos in a natural language. A review of the state of the art, current challenges, and future directions along with pointers to useful resources will be presented by different regular TRECVID participating teams. Each team will present one of the following tasks:

Significant progress has been made with deep neural networks recently. Sharing trained models of deep neural networks has been a very important in the rapid progress of research and development of these systems. At the same time, it is necessary to protect the rights to shared trained models. To this end, we propose to use digital watermarking technology to protect intellectual property and detect intellectual property infringement in the use of trained models. First, we formulate a new problem: embedding watermarks into deep neural networks. Second, we propose a general framework for embedding a watermark in model parameters, using a parameter regularizer. Our approach does not impair the performance of networks into which a watermark is placed because the watermark is embedded while training the host network. Finally, we perform comprehensive experiments to reveal the potential of watermarking deep neural networks as the basis of this new research effort. We show that our framework can embed a watermark during the training of a deep neural network from scratch, and during fine-tuning and distilling, without impairing its performance. The embedded watermark does not disappear even after fine-tuning or parameter pruning; the watermark remains complete even after 65% of parameters are pruned.

Region-based image retrieval (RBIR) technique is revisited. In early attempts at RBIR in the late 90s, researchers found many ways to specify region-based queries and spatial relationships; however, the way to characterize the regions, such as by using color histograms, were very poor at that time. Here, we revisit RBIR by incorporating semantic specification of objects and intuitive specification of spatial relationships. Our contributions are the following. First, to support multiple aspects of semantic object specification (category, instance, and attribute), we propose a multitask CNN feature that allows us to use deep learning technique and to jointly handle multi-aspect object specification. Second, to help users specify spatial relationships among objects in an intuitive way, we propose recommendation techniques of spatial relationships. In particular, by mining the search results, a system can recommend feasible spatial relationships among the objects. The system also can recommend likely spatial relationships by assigned object category names based on language prior. Moreover, object-level inverted indexing supports very fast shortlist generation, and re-ranking based on spatial constraints provides users with instant RBIR experiences.

Exploiting multimodal features has become a standard approach towards many video applications, including the video captioning task. One problem with the existing work is that it models the relevance of each type of features evenly, which neutralizes the impact of each individual modality to the word to be generated. In this paper, we propose a novel Modal Attention Network (MANet) to account for this issue. Our MANet extends the standard encoder-decoder network by adapting the attention mechanism to video modalities. As a result, MANet emphasizes the impact of each modality with respect to the word to be generated. Experimental results show that our MANet effectively utilizes multimodal features to generate better video descriptions. Especially, our MANet system was ranked among the top three systems at the 2nd Video to Language Challenge in both automatic metrics and human evaluations.

Multimedia data is by nature heterogeneous, conveying semantic information through multiple cues. Text analysis of closed captions already brought us understanding of the spoken information. Today's advances in computer vision now enable us to look for relevant semantic information from the visual content of real-world archives. Combining these two levels of extracted information to make sense of an archive still remains a challenge. Multiplex net- works, which model multiple families of interactions in a graph, can capture and combine both sources of semantics. We can leverage on these objects to extract hierarchies and integrate them in an interactive heterogeneous "visual cloud". Inspired by word clouds, these clouds allow to grasp visual and textual semantic information captured from a multimedia collection all at once. The interaction then enables direct access to the relevant video. We demonstrate our system with the exploration of a Japanese news archive.

Metric learning is an important issue in the person verification problem, which is to identify whether a pair of face or human body images is about the same person. Due to low running cost, the non-iterative statistical inference methods for metric learning show their efficiency and effectiveness to large scale datasets and on-line updating person verification applications. The KISSME method is a typical one that constructs the metric based on two assumptions that both of the discrepancy spaces of negative pairs and positive pairs should be Gaussian structures. However, we find that, in fact, the distribution of discrepancies of positive pairs might tend to the Laplace distribution rather than the Gaussian distribution. Based on this finding, we propose a metric learning method by exploiting Gaussian-Laplace distribution statistical inference, where the Gaussian distribution of negative discrepancies and the Laplace distribution of positive discrepancies are considered together. Experiments conducted on two human body datasets (VIPeR and Market-1501) and one face dataset (LFW) show its superiority in terms of effectiveness and efficiency as compared with the state-of-the-art approaches, no matter the appearance description is handcrafted or deep learned.

We adopt the generalized Pareto distribution for the information-based model and show that the parameters can be estimated based on the mean excess function. The proposed information retrieval model corresponds to the extension of the divergence from independence and is designed to be data-driven. The proposed model is then applied to the specific object search called the instance search and the effectiveness is experimentally confirmed.

Rushes videos consist of two types of content: the useless content and the redundant content (retakes). Then, automatic retake detection is more challenging due to the difficulty of eliminating repetitive takes, that are usually have different lengths and motion patterns. To overcome this challenge, previous approaches represent video segments using a longer string which is converted from SIFT matching, or a combination of different features. However, these require a large computational time and do not assist in improving a performance. In this work, we introduce a simple signature (global feature) to represent video segments because of its simplicity and effectiveness. The similarity between each pair of signature sequences was determined by using the Longest Common Subsequence algorithm (LCS). A simple retake detection was then used to detect a retake. This proposed was applied to the TRECVID BBC Rushed 2007 and 2008. The results showed that using a simple signature provides a high degree of accuracy, and reduces a computation time in feature extraction and LCS matching.

Huge video archives consisting of news programs, dramas, movies, and Web videos (e.g., YouTube) are available in our daily life. In all these videos, human is usually one of the most important subjects. Using state-of-the-art techniques, we can efficiently detect and track faces in the videos. In order to organize large-scale face tracks, containing sequences of (detected) consecutive faces in the videos, we propose an efficient method to retrieve human face tracks using bag-of-faces sparse representation (BoF-SR). Using the proposed method, a face track is encoded as a single BoF-SR, therefore allowing an efficient indexing method to handle large-scale data. To further consider the possible variations in face tracks, we generalize our method to find multiple SRs, in an unsupervised manner, to represent a bag of faces and balance the tradeoff between performance and retrieval time. The experimental results on two real-world (million-scale) data sets confirm that the proposed methods achieve significant performance gains compared with different state-of-the-art methods.

Video event detection is a challenging problem in information and multimedia retrieval. Different from single action detection, event detection requires a richer level of semantic information from video. In order to overcome this challenge, existing solutions often represent videos using high level features such as concepts. However, concept-based representation can be confusing because it does not encode the relationship between concepts. This issue can be addressed by exploiting the co-occurrences of the concepts, however, it often leads to a very huge number of possible combinations. In this paper, we propose a new approach to obtain the relationship between concepts by exploiting the syntactic dependencies between words in the image captions. The main advantage of this approach is that it significantly reduces the number of informative combinations between concepts. We conduct extensive experiments to analyze the effectiveness of using the new dependency representation for event detection on two large-scale TRECVID Multimedia Event Detection 2013 and 2014 datasets. Experimental results show that i) Dependency features are more discriminative than concept-based features. ii) Dependency features can be combined with our current event detection system to further improve the performance. For instance, the relative improvement can be as far as 8.6% on the MEDTEST14 10Ex setting.

As Internet users increasingly post images to express their daily sentiment and emotions, the analysis of sentiments in user-generated images is of increasing importance for developing several applications. Most conventional methods of image sentiment analysis focus on the design of visual features, and the use of text associated to the images has not been sufficiently investigated. This paper proposes a novel approach that exploits latent correlations among multiple views: visual and textual views, and a sentiment view constructed using SentiWordNet. In the proposed method, we find a latent embedding space in which correlations among the three views are maximized. The projected features in the latent space are used to train a sentiment classifier, which considers the complementary information from different views. Results of experiments conducted on Flickr and Instagram images show that our approach achieves better sentiment classification accuracy than methods that use a single modality only and the state-of-the art method that jointly uses multiple modalities.

Hierarchical classification is a computational efficient approach for large-scale image classification. The main challenging issue of this approach is to deal with error propagation. Irrelevant branching decision made at a parent node cannot be corrected at its child nodes in traversing the tree for classification. This paper presents a novel approach to reduce branching error at a node by taking its relative relationship into account. Given a node on the tree, we model each candidate branch by considering classification response of its child nodes, grandchild nodes and their differences with siblings. A maximum margin classifier is then applied to select the most discriminating candidate. Our proposed approach outperforms related approaches on Caltech-256, SUN-397 and ILSVRC2010-1K.

We present an algorithm for learning a feature representation for video segmentation. Standard video segmentation algorithms utilize similarity measurements in order to group related pixels. The contribution of our paper is an unsupervised method for learning the feature representation used for this similarity. The feature representation is defined over video supervoxels. An embedding framework learns a feature mapping for supervoxels in an unsupervised fashion such that supervoxels with similar context have similar embeddings. Based on the learned representation, we can merge similar supervoxels into spatio-temporal segments. Experimental results demonstrate the effectiveness of this learned supervoxel embedding on standard benchmark data.

Geometric verification with epipolar geometry often results in a high score for an incorrect image pair due to ambiguity in its geometric constraints. The ambiguity is caused by a high degree of freedom in the epipolar geometry and a weak constraint from the fitting between a point and a line. In order to mitigate the ambiguity, we propose to filter geometrically inconsistent components, namely correspondences, a sample, a model, and inliers in a RANSAC-based geometric verification. For the filtering, we introduce novel semi-2D constraints whose geometric constraint is weaker than full-2D constraint, but stronger than pure-epipolar constraint. Additionally, an advantage of the proposed approach is that it requires only an image pair instead of neither additional information nor prior learning. Experiments on the public dataset containing 3D object images show that the proposed approach improves the true positive rate when the false positive rate is low, and greatly reduces computational time for the geometric verification of both a correct image pair and an incorrect image pair.

TV ratings are a widely used indicator in the TV broadcasting field. While TV ratings are mainly used in advertising, they can also be used as a social sensor that reflects the interests of people. This paper presents a framework for discovering audience behavior through the mining of TV ratings. We have established a framework that enables discovery of numerous patterns of audience behavior from TV ratings. Used along with other multimedia contents such as video and text, it enables various types of knowledge to be semi-automatically found, such as what types of news programs are of most interest and what are the key visual features for acquiring high TV ratings. The discovery of audience behavior is achieved by focusing on the change points in the rating data, i.e., the points in time where many people switch the channel or turn the television on or off. Rich descriptions that characterize these points are extracted from multimedia contents, and then various filtering techniques are used to extract specific patterns of interest. Several applications of this framework for discovering knowledge demonstrated that it can effectively extract various types of audience behavior. To the best of our knowledge, this work is the first work to analyze the use of ratings data in combination with video and other multimedia data.

Video data is a sequence of video frames and their temporal continuity is an essential property of video stream. In this paper, we present a framework of constructing video continuity model from large-scale video archives with unsupervised learning. Our method estimates the similarity distribution of continuous frame pairs by applying simple assumption on the minimum duration of continuous video segments and then determines discontinuous frame pairs as outliers. The optimal estimation is pursued with measuring the separation between the estimated distribution of continuous frame pairs and that of discontinuous frame pairs. In order to verify the validity of the obtained model, the model is applied to the shot boundary detection. The results of experimental evaluation demonstrate the feasibility and the effectiveness of our method.

Visual analytics helps analytical reasoning and exploration of complex systems, for which it combines the means of interactive visualization, with the power of data analytics. The recent progress in computer vision techniques opens wide applications in real world video archives. Particularly, recent advances in face detection and recognition have been put under the spotlight. The applications of such techniques are often concern intelligence, or peer recognition in photo posted in social networks. We propose to combine those two domains by demonstrating a visual exploration of over a decade of the news program from the Japanese broadcaster NHK News 7. We derive social networks from face detection and tracking of this large dataset. With the help of a little domain knowledge, we monitor the activity of political public figures and explore the archive. This allows understanding and comparison of the politico-media scene presented by NHK under different Prime Minister's governance. The social networks are interactive, and also allow to explore the multimedia database and explore its video content.

Presents a listing of the new Associate Editors for this issue of the publication.

In image search re-ranking, besides the well-known semantic gap, intent gap, which is the gap between the representation of users' query/demand and the real intent of the users, is becoming a major problem restricting the development of image retrieval. To reduce human effects, in this paper, we use image click-through data, which can be viewed as the implicit feedback from users, to help overcome the intention gap, and further improve the image search performance. Generally, the hypothesis-visually similar images should be close in a ranking list-and the strategy-images with higher relevance should be ranked higher than others-are widely accepted. To obtain satisfying search results, thus, image similarity and the level of relevance typicality are determinate factors correspondingly. However, when measuring image similarity and typicality, conventional re-ranking approaches only consider visual information and initial ranks of images, while overlooking the influence of click-through data. This paper presents a novel re-ranking approach, named spectral clustering re-ranking with click-based similarity and typicality. First, to learn an appropriate similarity measurement, we propose click-based multi-feature similarity learning algorithm, which conducts metric learning based on click-based triplets selection, and integrates multiple features into a unified similarity space via multiple kernel learning. Then, based on the learnt click-based image similarity measure, we conduct spectral clustering to group visually and semantically similar images into same clusters, and get the final re-rank list by calculating click-based clusters typicality and within-clusters click-based image typicality in descending order. Our experiments conducted on two real-world query-image data sets with diverse representative queries show that our proposed re-ranking approach can significantly improve initial search results, and outperform several existing re-ranking approaches.

The rich nature of news makes it a classic subject of visual analytics research. Such analysis is often based on rich textual data. However, we want to test how much we can understand the news from video information through face detection and tracking. Towards this goal, we propose a visual analytics system and discuss its design and implementation to support media experts in understanding political interactions in an archive of 12 years of the Japanese public broadcaster NHK's News 7 program. After identifying the tasks and abstraction required for our analysis, we construct links from face detection and tracking to derive multiple political networks. Our proposed design embeds this rich data into a visual analytics framework that presents four levels of abstraction: time period, network, timeline, and face-tracks within video. We present how the exploration of the archive with our system results in good understanding of the Japanese politico-media scene during these 12 years while finding evidence of “presidentialization” of the media.

A complex event can be recognized by observing necessary evidences. In the real world scenarios, this is a difficult task because the evidences can happen anywhere in a video. A straightforward solution is to decompose the video into several segments and search for the evidences in each segment. This approach is based on the assumption that segment annotation can be assigned from its video label. However, this is a weak assumption because the importance of each segment is not considered. On the other hand, the importance of a segment to an event can be obtained by matching its detected concepts against the evidential description of that event. Leveraging this prior knowledge, we propose a new method, Event-driven Multiple Instance Learning (EDMIL), to learn the key evidences for event detection. We treat each segment as an instance and quantize the instance-event similarity into different levels of relatedness. Then the instance label is learned by jointly optimizing the instance classifier and its related level. The significant performance improvement on the TRECVID Multimedia Event Detection (MED) 2012 dataset proves the effectiveness of our approach.

This paper proposes a framework for content-based video retrieval that addresses various tasks as particular event retrieval, copy detection or video synchronization. Given a video query, the method is able to efficiently retrieve, from a large collection, similar video events or near-duplicates with temporarily consistent excerpts. As a byproduct of the representation, it provides a precise temporal alignment of the query and the detected video excerpts.

In this paper, a novel method for human action recognition from depth videos is proposed. We project 3D data on to multiple 2D-planes from which dense trajectories features are extracted. In the training stage, for each projection, a classifier is trained using the training data. In the testing stage, for each test video, the multiple trained classifiers are applied and the predicted scores are combined for final decision. We propose a greedy-based method to select a subset of the trained classifiers for optimal combination. Experiments on the MSR Action 3D dataset show that the proposed method outperforms the baseline method that does not use multi-projection-based features.

We study the problem of multi-class image classification with large number of classes, of which the one-vs-all based approach is prohibitive in practical applications. Recent state-of-the-art approaches rely on label tree to reduce classification complexity. However, building optimal tree structures and learning precise classifiers to optimize tree loss is challenging. In this paper, we introduce a novel approach using latent classifiers that can achieve comparable speed but better performance. The key idea is that instead of using C one-vs-all classifiers (C is the number of classes) to generate the score matrix for label prediction, a much smaller number of classifiers are used. These classifiers, called latent classifiers, are generated by analyzing the correlation among classes and removing redundancy. Experiments on several large datasets including ImageNet-1K, SUN-397, and Caltech-256 show the efficiency of our approach.

Bag-of-Word based model is one of the state-of-the-art approaches for object retrieval or also known as instance search problem. Although this model and its extensions are good for rich-textured objects, it is still unsolved for searching on textureless ones. In this paper, we propose to combine this model with Deformable Part Models object detector using late fusion technique to improve final result. To find the optimal weights for each type of query objects, we further propose to use a neural network to learn query features including object area, number of shared visual words to get optimal weights for each model. Experimental results on TRECVID Instance Search (INS) dataset with queries in INS2013 and INS2014 show that our proposed method significantly improves 18.48% and 14.63% in mAP respectively comparing to standard BOW model and outperform other state-of-the-art methods. This method opens a new way of adaptively combining DPM, an object detector, in a hybrid model for visual instance search.

TV news programs are one of major information sources for daily lives. Since the computer storage cost has reduced significantly, today we can digitally archive a huge amount of TV news programs. On the other hand, as the archive size grows larger, the cost for browsing and utilizing news video archives also increases significantly. To circumvent this problem, we devised a visualization method called "News Shot Cloud" in our former work. The news shot cloud displays TV news shots in the similar manner with the tag clouds. In this poster, we present an enhancement method of the news shot cloud by modifying the thumbnail layout from the conventional linear layout to the circular. The design and the advantages of the new layout are shown with experimental implementation. The new layout is more helpful to identify informative news shots rapidly, and at the same time, it also helps users to examine less-weighted news shots to find lower-ranked but unexpectedly informative ones.

To understand recent societal behavior, it is important to compare how multiple media are affected by real-world events and how each medium affects other media. This paper proposes a novel framework for inter-media comparison through visualizing images extracted from different types of media. We extract blog image clusters from our six-year blog archive and search for similar TV shots in each cluster from a broadcast news video archive by using image similarities. We then visualize such image flows on a timeline in 3D space to visually and interactively explore time sequential changes in influences among media resources and differences and/or similarities between them such as topics that become popular on only blogs or that become popular on blogs earlier than on TV.

We study how aggregating multiple images, on query or database side, impacts the performance of visual object retrieval in a Bag-of-Words framework. To this end, we first compare five different multi-image aggregation methods, and suggest selecting the average pooling method in most cases for its superior advantages in accuracy, speed, and memory footprint. Then we prove with experiments that more images generally yield better retrieval performance. What is more, we illustrate that simply aggregating query images without selection is far from optimal. Comprehensive experiments were conducted on three large-scale object retrieval datasets, and the new state-of the-art was achieved. This research can be leveraged in some real applications such as mobile search, where the retrieval performance will be improved once users snap multiple query images.

A video can be viewed as a layered structure where the lowest layer are frames, the top layer is the entire video, and the middle layers are the sequences of consecutive frames or the concatenation of lower layers. While it is easy to find local discriminative features in video from lower layers, it is non-trivial to aggregate these features into a discriminative video representation. In literature, people often use sum pooling to obtain reasonable recognition performance on artificial videos. However, the sum pooling technique does not work well on complex videos because the region of interests may reside within some middle layers. In this paper, we leverage the layered structure of video to propose a new pooling method, named sum-max video pooling, to handle this problem. Basically, we apply sum pooling at the low layer representation while using max pooling at the high layer representation. Sum pooling is used to keep sufficient relevant features at the low layer, while max pooling is used to retrieve the most relevant features at the high layer, therefore it can discard irrelevant features in the final video representation. Experimental results on the TRECVID Multimedia Event Detection 2010 dataset shows the effectiveness of our method.

Spatial re-ranking has proved to be successful in image retrieval. Yet no work on spatial re-ranking has been systematically reported for instance search from videos so far: As videos are composed of multiple frames, and frame-by-frame spatial verification is too prohibitive, till now we lack an efficient spatial re-ranking method designed for the purpose of video instance search. The effectiveness is unclear as well. This paper proposes a practical spatial re-ranking method for video instance search. We make two contributions to speed up the algorithm: One is to select the most representative image/frame for spatial verification, based on the available Bag-of-Words representation; the other is to efficiently build tentative matching feature pairs, which serve as the input of the RANSAC algorithm, by regarding features quantized to the same visual word as matches, thus avoid the costly nearest-neighbor search. These two modifications lead one order of magnitude speedup without compromising the performance. Another contribution is a ROI-originated RANSAC method, which improves the re-ranking performance significantly. Experiments were carried out on the TrecVid Instance Search 2013 dataset, and the new state-of-the-art performance was achieved by the proposed method, at a much faster speed.

An iPhone application called "Wonder Search" for application search has been developed under the hypothesis that a search interface included sense-related search facets gives a better chance of finding new applications for users. The search facets comprise five attributes: icon color, icon tone, application category, user evaluation, and release date. We enhanced a ring-structured graphical search interface for multi-faceted metadata to solve two reported usability problems by preparing dials for each facet and dividing the functions for ring rotation into two operations. Usability testing with 60 participants demonstrated that users can use this application not only for searching for applications effectively but also for analyzing the features of applications. The application can potentially be used for sensing the emotions of users if the hypothesis that the colors browsed or selected by users are related to their feelings is correct.

Content-based image retrieval (CBIR) technique is important for browsing the rapidly growing Web images. However, traditional CBIR methods usually fail when the query and database images are in different domains. Instead of focusing on a specific domain, we propose a method to solve the general cross-domain image retrieval problem. This method focuses on the structure of image content but not the detail at pixel level, so it is particularly useful for matching the images across different visual domains e.g., paintings, hand-drawn sketches or photographs. To provide an efficient and effective solution, we analyze the bag-of-words (BoW) matching procedure to find out what causing it to fail in the cross-domain setting. We observe that it is necessary to apply different matching constraints for cross-domain image retrieval; therefore, we propose a multi-level matching process which dynamically selects the most suitable matching constraint for feature matching and adopt a fast spatial verification to describe the structure similarity. Finally, our method is much faster than the state-of-the-art solution and achieves better performance in the experiments.

This paper deals with a novel concept of an exponential IDF in the BM25 formulation and compares the search accuracy with that of the BM25 with the original IDF in a content-based video retrieval (CBVR) task. Our video retrieval method is based on a bag of keypoints (local visual features) and the exponential IDF estimates the keypoint importance weights more accurately than the original IDF. The exponential IDF is capable of suppressing the keypoints from frequently occurring background objects in videos, and we found that this effect is essential for achieving improved search accuracy in CBVR. Our proposed method is especially designed to tackle instance video search, one of the CBVR tasks, and we demonstrate its effectiveness in significantly enhancing the instance search accuracy using the TRECVID2012 video retrieval dataset.

We propose a method for estimating the attentiveness of viewers watching TV on the basis of their emotional behaviors. Such behaviors are usually hard to perceive and have in the past required contact-type equipment to measure them. We devised a novel attentiveness estimator that uses three visual features that are correlated with emotional behaviors: the viewer's head motion, blink interval, and eye movement. The main contribution of this paper is that the method automatically obtains these features from ordinary noncontact-type sensors. In addition, the method considers not only the viewer's emotional behaviors but also content features which affect the behaviors. The temporal window pattern in a video sequence is used to describe temporally local features. Experimental results indicate the possibility of recognizing TV viewers' attentiveness from certain visible emotional behaviors measured by common home electronics.

Visual object retrieval aims at retrieving, from a collection of images, all those in which a given query object appears. It is inherently asymmetric: the query object is mostly included in the database image, while the converse is not necessarily true. However, existing approaches mostly compare the images with symmetrical measures, without considering the different roles of query and database. This paper first measure the extent of asymmetry on large-scale public datasets reflecting this task. Considering the standard bag-of-words representation, we then propose new asymmetrical dissimilarities accounting for the different inlier ratios associated with query and database images. These asymmetrical measures depend on the query, yet they are compatible with an inverted file structure, without noticeably impacting search efficiency. Our experiments show the benefit of our approach, and show that the visual object retrieval task is better treated asymmetrically, in the spirit of state-of-the-art text retrieval.

In this paper, we propose a biologically inspired approach to overcome the challenges of searching instances from large video databases. Specifically, we train sparse filters in subspaces from unlabelled natural images, then yield image feature for new image instances through pre-learned filters. Therefore, no traditional “hand-designed” features (e.g. colour histograms, interest point descriptors) are required in our system. Experiments on a challenging large video database containing 20982 videos show our approach outperforms traditional approaches such as Bag-of-Words using SURF, or the combination of SIFT, SURF, RGB and texture features.

Recently visual object retrieval has being widely studied for its vast application prospect, and the progress is usually tracked by the performance achieved on benchmark datasets. Therefore, in order to faithfully evaluate object retrieval algorithms, the quality of datasets must be ensured by some means. In this paper, we propose a method to evaluate the quality of benchmarking visual object retrieval on two highly cited object retrieval datasets, the Oxford datasets and TrecVid instance search datasets. Our evaluation method leverages the essential differences between object retrieval and other similar image search, and digs out some unrevealed and rather interesting features from those datasets. To the best of our knowledge, this research has never been touched before. Our method is believed to be beneficial to dataset collection and algorithm evaluation of object retrieval. More importantly, we hope this work can attract more attentions on this topic in the community.

Scalability to large numbers of classes is an important challenge for multi-class classification. It can often be computationally infeasible at test phase when class prediction is performed by using every possible classifier trained for each individual class. This paper proposes an attribute-based learning method to overcome this limitation. First is to define attributes and their associations with object classes automatically and simultaneously. Such associations are learned based on greedy strategy under certain conditions. Second is to learn a classifier for each attribute instead of each class. Then, these trained classifiers are used to predict classes based on their attribute representations. The proposed method also allows trade-off between test-time complexity (which grows linearly with the number of attributes) and accuracy. Experiments based on Animals-with-Attributes and ILSVRC2010 datasets have shown that the performance of our method is promising when compared with the state-of-the-art.

Broadcast TV program is a quite informative media resource which records our daily life over the time. While for emphasizing real-time reporting, those out-of-date video archives once were elaborately created with high quality are always left without being fully used. In this paper, many known state-of-the-art retrieval technologies are integrated into a commercial film retrieval system, which manages to index a huge commercial dataset archived from five TV channels within recent three years. The final purpose is to connect images queried by users with our archived broadcast video dataset via searching relevant commercials and accessing their broadcast information, such as air time and replay frequency. This system also serves as one part of our ongoing broadcast TV program reusing project.

We describe the graphical search interface we have developed for multi-faceted hierarchical metadata and a video navigation system implementing this 'Revolving Cube Show' interface. This interface can treat discrete, continuous, and hierarchical attributes, enabling users to search flexibly and intuitively by using simple operations to combine attributes. Testing using data on 5495 Japanese TV programs and five attributes (channel, time zone, airtime, genre, and performer) showed that users were able to easily combine the various attributes to create dynamic search hierarchies.

Rapid increase in video databases has forced the industry to have efficient and effective frameworks for video retrieval and indexing. Video segmentation into scenes is widely used for video summarization, partitioning, indexing and retrieval. In this paper, we propose a framework for scene detection mainly based on entropy and Speeded Up Robust Features (SURF) features. First, we detect the fade and abrupt boundaries based on frame entropy analysis and SURF features matching. Fade boundaries are smart indication of scenes beginning or ending in many videos and dramas, and are detected by frame entropy analysis. Before abrupt boundary detection, unnecessary frames which are obviously not abrupt boundaries, such as blank screens, high intensity influenced images, sliding credits, are removed. Candidate boundaries are detected to make SURF features efficient for abrupt boundary detection, and SURF features between candidate boundaries and their adjacent frames are used to detect the abrupt boundaries. Second, key frames are extracted from abrupt shots. We evaluate our key frame extraction with other famous algorithms and show the effectiveness of the key frames. Finally, scene boundaries are detected using sliding window of size K over the key frames in temporal order. In experimental evaluation on the TRECVID-2007 shot boundary test set, the algorithm for shot boundary achieves substantial improvements over state-of-the-art methods with the precision of 99% and the recall of 97.8%. Experimental results for video segmentation into scenes are also promising, compared to famous state-of-the-art techniques.

We describe a system based on exact-duplicate matching for detecting and localizing TV commercials in a video stream, clustering the exact duplicates, and detecting duplicate exact-duplicate clusters across video streams. A two-stage temporal recurrence hashing algorithm is used for the detection, localization, and clustering. The algorithm is fully unsupervised, generic, and ultrahigh speed. Another algorithm is used to integrate the video and audio streams to achieve higher performance extraction. Its sequence- and frame-level accuracies in testing were respectively 98.1% and 97.4%. A third algorithm uses a new bag-of-fingerprints model to detect duplicate exact-duplicate clusters across multiple streams. It is robust against decoding errors. Its contributions include: 1) fully unsupervised detection, extraction, and matching of exact duplicates; 2) more generic commercial detection than with the knowledge-based techniques; 3) ultrahigh-speed processing, which detected the TV commercials from a one-month video stream in less than 42 minutes, which is more than ten times faster than with state-of-the-art algorithms; and 4) more generic operation in terms of signal input, the performance of which is consistent between video and audio streams. Testing using a video database containing a ten-hour, a one-month, and a five-year video stream comprehensively demonstrates the effectiveness and efficiency of this system.

The practicality of large-scale image indexing and querying methods depends crucially upon the availability of semantic information. The manual tagging of images with semantic information is in general very labor intensive, and existing methods for automated image annotation may not always yield accurate results. The aim of this paper is to reduce to a minimum the amount of human intervention required in the semantic annotation of images, while preserving a high degree of accuracy. Ideally, only one copy of each object of interest would be labeled manually, and the labels would then be propagated automatically to all other occurrences of the objects in the database. To this end, we propose an influence propagation strategy, SW-KProp , that requires no human intervention beyond the initial labeling of a subset of the images. SW-KProp distributes semantic information within a similarity graph defined on all images in the database: each image iteratively transmits its current label information to its neighbors, and then readjusts its own label according to the combined influences of its neighbors. SW-KProp influence propagation can be efficiently performed by means of matrix computations, provided that pairwise similarities of images are available. We also propose a variant of SW-KProp which enhances the quality of the similarity graph by selecting a reduced feature set for each prelabeled image and rebuilding its neighborhood. The performances of the SW-KProp method and its variant were evaluated against several competing methods on classification tasks for three image datasets: a handwritten digit dataset, a face dataset and a web image dataset. For the digit images, SW-KProp and its variant performed consistently better than the other methods tested. For the face and web images, SW-KProp outperformed its competitors for the case when the number of prelabeled images was relatively small. The performance was seen to improve significantly when the feature selection strategy was applied.

We have improved our previously reported 'Concentric Ring View' search interface for multi-faceted metadata so that the display of the retrieved results is more meaningful. We constructed a country data exploration system as a teaching tool for geography education at the elementary- and middle-school levels. This system enables students to search for information flexibly and intuitively by combining attributes using simple operations. We defined seven attributes (five from textbooks and two related to national flags) and created a world map that helps the student place the retrieved results in context.

It is necessary to utilize visual information to improve the efficiency of retrieval in image-search engines that use textual information for indexing. One popular approach has been to learn visual consistency between images returned by these search engines. Most state-of-the-art methods of learning visual consistency usually learn one specific classifier for each query to re-rank the returned images. The main drawback with these query-specific based methods is that they require computational cost and processing time that are unsuitable for handling a large number of queries. Another approach has been to learn one generic classifier once and then use for all queries. Pursuing the generic classifier based approach, we study the problem of re-ranking faces returned by existing search engines to improve retrieval performance. Learning a generic classifier involves finding good query-dependent feature representation and collecting sufficient large number of training samples. Existing work [9, 15] studies query-dependent features for general objects rather than faces. In addition, training samples are usually collected manually. The key contribution of this research is to introduce a query-dependent feature for faces and an unsupervised method of automatically collecting training samples to learn the generic classifier. The experimental results demonstrated that the proposed method performed very well in various datasets.

A very promising application involving video collections is to search for relevant video segments from a video database when given few visual examples of the specific instance, e.g. a person, object, or place. However, this problem is difficult due to the lighting variations, different viewpoints, partial occlusion, and large changes in appearance. In this paper, we focus on a kind of restricted instance searching task, where the region of a specific instance to be searched for is manually labeled on each query image. We formulate this problem in a large vocabulary quantization based Bag-of-Words framework, while putting more research emphasis on investigating to what extent we can benefit from these labeled instance regions. The contribution of this paper mainly lies in two aspects: first, we proposed an algorithm for instance search that outperformed all submissions on the instance search dataset TRECVID 2011. Secondly, after thoroughly analyzing the experiment results, we show that our top performance is mainly due to similar scene retrieval, instead of the same instance search. This observation reveals that in the current dataset background is more dominated than instance, and it also suggests that a promising direction in which to further improve the current algorithm, which may also be the breakthrough for achieving this challenge, is to investigate more about how to truly take advantage of additional labeled instance regions. We believe our research opens a window for future new methods for searching instance.

We propose a dual-stage algorithm for fully-unsupervised and super fast TV commercial mining in this paper. The two stages involved in process include: 1) searching for recurring short segments, and 2) assembling these short segments into sets of long and complete commercial sequences. The first stage is achieved by frame hashing. Different from the related studies that depend on brute-force pairwise matching, we propose applying a second-stage hashing algorithm for the recurring segment assemblage, which is the key idea in this pa per. A large-scale archive containing a 10-hour and a 1-month stream was used for the experimentation. The algorithm mined commercials from the 1-month stream in less than 50 minutes, which was ten times faster than that of related studies, with a 98.05% sequence level and 97.39% frame-level accuracy. We demonstrate the performance consistency of the algorithm on both audio and video streams, and investigate the computational cost from both the theoretical and experimental viewpoints.

Spatial relationships between local features are thought to play a vital role in representing object categories. However, learning a compact set of higher-order spatial features based on visual words, e.g., doublets and triplets, remains a challenging problem as possible combinations of visual words grow exponentially. While the local pairwise codebook achieves a compact codebook of pairs of spatially close local features without feature selection, its formulation is not scale invariant and is only suitable for densely sampled local features. In contrast, the proximity distribution kernel is a scale-invariant and robust representation capturing rich spatial proximity information between local features, but its representation grows quadratically in the number of visual words. Inspired by the two abovementioned techniques, this paper presents the compact correlation coding that combines the strengths of the two. Our method achieves a compact representation that is scaleinvariant and robust against object deformation. In addition, we adopt sparse coding instead of k-means clustering during the codebook construction to increase the discriminative power of our method. We systematically evaluate our method against both the local pairwise codebook and proximity distribution kernel on several challenging object categorization datasets to show performance improvements.

Combining global scene classification with object detection has helped in improving the classification accuracy. However, training an object detector requires a large amount of manual annotation. The object detector may also fail when the object is occluded. Meanwhile, the presence of the object is not only indicated by the entire object region but any of its parts or its correlations with other regions in the image. To overcome these limitations, we propose using discriminative region localization instead of object detection in the combination. Our contribution is two-fold, a) a complete framework that combines global scene classification with discriminative region localization for image classification and b) a weakly supervised discriminative region localization approach that utilizes spatial context to improve the learning accuracy. Our experimental results on benchmark datasets demonstrated that the proposed discriminative region localization approach outperforms the state-of-the-art approach. In addition, the combination significantly increases the classification performance.

There have recently been many methods proposed for matching face sequences in the field of face retrieval. However, most of them have proven to be inefficient in large-scale video databases because they frequently require a huge amount of computational cost to obtain a high degree of accuracy. We present an efficient matching method that is based on the face sequences (called face tracks) in large-scale video databases. The key idea is how to capture the distribution of a face track in the fewest number of low-computational steps. In order to do that, each face track is represented by a vector that approximates the first principal component of the face track distribution and the similarity of face tracks bases on the similarity of these vectors. Our experimental results from a large-scale database of 457,320 human faces extracted from 370 hours of TRECVID videos from 2004-2006 show that the proposed method easily handles the scalability by maintaining a good balance between the speed and the accuracy.

We propose two novel algorithms for fully-unsupervised, super-fast, and cross-channel TV commercial mining in this paper. The tasks involved in the process include: 1) mining commercial clusters from streams of individual channels, and 2) grouping identical commercial clusters across multiple channels. The first process is achieved with a dual-stage hashing algorithm, which searches for recurring short segments by hashing frames, and it assembles these short segments into sets of commercial sequences by hashing temporal recurrences. The algorithm mined commercials from a one-month stream in less than 42 minutes, which was ten times faster than that in related studies. A new bag-of-fingerprints model is proposed for the second process to encode the temporal clues of local fingerprints. The model is abundantly robust against framing and fingerprinting errors in recurring sequences, and discovers false matches of local fingerprints. A five-month database was used for comprehensively demonstrating the effectiveness and efficiency of the model.

The local feature (e.g. SIFT) and Bag of Words (BOW) model play key roles in achieving a state-of-the-art performance for image classification. Although we realize that utilizing extra color information will undoubtedly boost the local feature, there still have not been any research that have carefully focused on how to efficiently transfer this color boosted local feature into a boosted BOW. In this paper, a channel-wise separate quantization is newly proposed to replace the traditional multi-channel combined quantization for color SIFT. Dozens of experiments conducted on common data sets have consistently proven that the latter is noticeably worse than the former. We attribute this to a larger quantization error induced by the unreasonable assumption of identical feature distributions in multiple channels. In addition, we compare the distinctness of the intensity and color components separately, and the experimental results are beyond our expectations. Our experiments also show that the accuracy is noticeably enhanced when we take the top N nearest neighbors in soft assignment into account.

The aim of this paper is to reduce to a minimum the level of human intervention in the semantic annotation process of images. Ideally, only one copy of each object of interest would be labeled manually, and the labels would then be propagated automatically to all other occurrences of the objects in the database. To that end, we propose a neighbor-based influence propagation approach KProp which builds a voting model and propagates the knowledge associated to some objects to similar objects. We show that KProp can perform efficiently through matrix computations and achieve better performance with fewer labeled examples per object.

Sparse coding, a method of explaining sensory data with as few dictionary bases as possible, has attracted much attention in computer vision. For visual object category recognition, L1 regularized sparse coding is combined with spatial pyramid representation to obtain state-of-the-art performance. However, because of its iterative optimization, applying sparse coding onto every local feature descriptor extracted from an image database can become a major bottleneck. To overcome this computational challenge, this paper presents "Generalized Lasso based Approximation of Sparse coding" (GLAS). By representing the distribution of sparse coefficients with slice transform, we fit a piece-wise linear mapping function with generalized lasso. We also propose an efficient post-refinement procedure to perform mutual inhibition between bases which is essential for an overcomplete setting. The experiments show that GLAS obtains comparable performance to L1 regularized sparse coding, yet achieves significant speed up demonstrating its effectiveness for large-scale visual recognition problems.

This paper proposes new low-dimensional image features that enable images to be very efficiently matched. Image matching is one of the key technologies for many vision-based applications, including template matching, block motion estimation, video compression, stereo vision, image/video near-duplicate detection, similarity join for image/video database, and so on. Normalized cross correlation (NCC) is one of widely used method for image matching with preferable characteristics such as robustness to intensity offsets and contrast changes, but it is computationally expensive. The proposed features, derived by the method of Lagrange multipliers, can provide upper-bounds of NCC as a simple dot product between two low-dimensional feature vectors. By using the proposed features, NCC-based image matching can be effectively accelerated. The matching performance with the proposed features is demonstrated using an image database obtained from actual broadcast videos. The new features are shown to outperform other methods: multilevel successive elimination algorithm (MSEA), discrete cosine transform (DCT) coefficients, and histograms, achieving very high precision while only slightly sacrificing recall.

This paper presents a unified framework for human action classification and localization in video using structured learning of local space-time features. Each human action class is represented by a set of its own compact set of local patches. In our approach, we first use a discriminative hierarchical Bayesian classifier to select those space-time interest points that are constructive for each particular action. Those concise local features are then passed to a Support Vector Machine with Principal Component Analysis projection for the classification task. Meanwhile, the action localization is done using Dynamic Conditional Random Fields developed to incorporate the spatial and temporal structure constraints of superpixels extracted around those features. Each superpixel in the video is defined by the shape and motion information of its corresponding feature region. Compelling results obtained from experiments on KTH [22], Weizmann [1], HOHA [13] and TRECVid [23] datasets have proven the efficiency and robustness of our framework for the task of human action recognition and localization in video.

Recently, sparse coding has been receiving much attention in object and scene recognition tasks because of its superiority in learning an effective codebook over k-means clustering. However, empirically, such codebook requires a relatively large number of visual words, essentially bases, to achieve high recognition accuracy. Therefore, due to the combinatorial explosion of visual words, it is infeasible to use this codebook to represent higher-order spatial features which are equally important in capturing distinct properties of scenes and objects. Contrasted with many previous techniques that exploit higher-order spatial features, Local Pairwise Codebook (LPC) is a simple and effective method to learn a compact set of clusters representing pairs of spatially close descriptors with k-means. Based on LPC, this paper proposes Directional Local Pairwise Bases (DLPB) that applies sparse coding to learn a compact set of bases capturing correlation between these descriptors, so to avoid the combinatorial explosion. Furthermore, such bases are learned for each quantized direction thereby explicitly adding directional information to the representation. We have evaluated DLPB with several challenging object and scene category datasets. Our experimental results show that DLPB outperforms the baselines across all datasets and achieves the state-of-the-art performance on some datasets.

In this paper, we present an image retrieval method based on feature tracking. Feature tracks are summarized into a compact discreet value and used for video indexing purpose. As opposed to existing space-time features, we do not make any assumption on the motion visible on the indexed videos. As a result, given an example query, our system is able to retrieve related videos from a large database. We evaluated our system with the copy detection benchmark MUSCLE-VCD-2007. We also ran retrieval experiment on hours of TV broadcast.

In this paper a particular issue is addressed: matching still images (screen shots) with videos. A content-based similarity search approach using image queries is proposed. A fast method based on local visual patterns both for matching and indexing is employed. But we argue that using every frames may limit the scalability of the approach. Therefore only keyframes are extracted and used. The main contribution of this paper is an investigation over the trade-off between accuracy and scalability using different keyframe rates for sampling the video database. This trade-off is evaluated on a ground truth using a large reference video database (1000 hours).

In this paper, we propose a method to detect and identify commercial films from broadcast videos by using Temporal Occurrence Pattern (TOP). Our method uses the characteristic of broadcast videos in Japan that each individual commercial film appears multiple times in broadcast stream and typically has the same duration (e.g., 15 seconds). Using this characteristic, the method can detect as well as identify individual commercial films within given video archive. Based on simple signature (global feature) for each frame image, the method first puts all frames into numbers of buckets where each bucket contains frames having the same signature, and thus they appear the same. For each bucket, TOP as a binary sequence representing the occurrence time within video archive is then generated. All buckets are then clustered using simple hierarchical clustering with similarity between TOPs allowing possible temporal offset. This clustering stage can stitch up all frames for each commercial film and identify multiple occurrence of the same commercial film at the same time. We tested our method using actual broadcast video archive and confirmed good performance in detecting and identifying commercial films.

In this paper, we present a robust framework for action recognition in video, that is able to perform competitively against the state-of-the-art methods, yet does not rely on sophisticated background subtraction preprocess to remove background features. In particular, we extend the Implicit Shape Modeling (ISM) of [10] for object recognition to 3D to integrate local spatiotemporal features, which are produced by a weakly supervised Bayesian kernel filter. Experiments on benchmark datasets (including KTH and Weizmann) verifies the effectiveness of our approach.

Our objective was to apply different kinds of database with our proposed graphical search interface, and to verify the effectiveness focused on user knowledge structure in searching because it allowed users to easily modify received information to suitable knowledge. This interface was for multi-faceted metadata named Concentric Ring View. The design concept of this interface is "result-oriented", which means users continue to search by evaluating the retrieved results.

PyCVF is an open source framework for computer vision and video-mining. It allows rapid development of applications and it provides standardized tools for common operations such as : browsing datasets, applying transformations to one dataset on-the-fly, computing features, indexing multimedia datasets, querying for nearest-neighbors, training a statistical model, or browsing the result in a 3d-space. PyCVF has a Python API, it also provides command line programs, QT Gui, and a web front-end. It can interacts nicely with other leading frameworks such as Weka, Orange, OpenCV, Django...

We investigate a method using support vector machines (SVMs) with walk-based graph kernels for high-level feature extraction from images. In this method, each image is first segmented into a finite set of homogeneous segments and then represented as a segmentation graph where each vertex is a segment and edges connect adjacent segments. Given a set of features associated with each segment, we then obtain a positive definite kernel between images by comparing walks in the respective segmentation graphs, and image classification is carried out with an SVM based on this kernel. In a benchmark experiment on the MediaMill challenge problem, the mean average precision increased from 0.216 (baseline) to 0.341 when our method was utilized.

Searching for images of people is an essential task for image and video search engines. However, current search engines have limited capabilities for this task since they rely on text associated with images and video, and such text is likely to return many irrelevant results. We propose a method for retrieving relevant faces of one person by learning the visual consistency among results retrieved from text correlation-based search engines. The method consists of two steps. In the first step, each candidate face obtained from a text-based search engine is ranked with a score that measures the distribution of visual similarities among the faces. Faces that are possibly very relevant or irrelevant are ranked at the top or bottom of the list, respectively. The second step improves this ranking by treating this problem as a classification problem in which input faces are classified as psilaperson-Xpsila or psilanon-person-Xpsila; and the faces are re-ranked according to their relevant score inferred from the classifierpsilas probability output. To train this classifier, we use a bagging-based framework to combine results from multiple weak classifiers trained using different subsets. These training subsets are extracted and labeled automatically from the rank list produced from the classifier trained from the previous step. In this way, the accuracy of the ranked list increases after a number of iterations. Experimental results on various face sets retrieved from captions of news photos show that the retrieval performance improved after each iteration, with the final performance being higher than those of the existing algorithms.

This paper describes our attempt to generate quizzes automatically from news programs, as novel method for content creation. Of the many types of possible quizzes we focus on the format of a multiple-choice quiz with accompanying images. To begin, we worked on formulating the engineering problem of quiz generation - selection of images suitable for a quiz, selection of text that explains the image, and the generation of alternative choices that are similar but different by selecting from subtitles. All the approaches were designed based on principal object in the image and text. As the suitable image for the quiz, an image which has a clear subject - we defined it as spotlight-image - is introduced. As for generation of the similar but different choices, categorizing the subject into person or other can make the correspondence between images and sentence much clear, and then appropriate choices are generated. By introducing these techniques, the accuracy has risen (35.7% up) and the effectiveness of our proposed method were confirmed.

The paper is aiming to detect and retrieve videos of the same scene (scene duplicates) from broadcast video archives. Scene duplicate is composed of different pieces of footage of the same scene, the same event, at the same time, but from the different viewpoints. Scene duplicate detection would be particularly useful to identify the same event reported in different programs from different broadcast stations. The approach should be invariant to viewpoint changes. We focused on object motion in videos and devised a video matching approach based on the temporal pattern of discontinuities obtained from feature point trajectories. We developed an acceleration method based on the discontinuity pattern, which is more robust to variations in camerawork and editing than conventional features, to dramatically reduce the computation burden. We compared our approach with an existing video matching method based on the local feature of keyframe. The spatial registration strategy of this method was also used with the proposed approach to cope with visually different unrelated video pairs. The performance and effectiveness of our approach was demonstrated on actual broadcasted videos.

Video shot boundary detection is one of the fundamental tasks of video indexing and retrieval applications. Although many methods have been proposed for this task, finding a general and robust shot boundary method that is able to handle the various transition types caused by photo flashes, rapid camera movement and object movement is still challenging. We present a novel approach for detecting video shot boundaries in which we cast the problem of shot boundary detection into the problem of text segmentation in natural language processing. This is possible by assuming that each frame is a word and then the shot boundaries are treated as text segment boundaries (e.g. topics). The text segmentation based approaches in natural language processing can be used. The experimental results from various long video sequences have proved the effectiveness of our approach.

We present a method to improve the performance of face retrieval in news videos by using the relevant-set correlation (RSC) clustering model. In this method, faces of a person are firstly retrieved by finding in shots whose associated transcripts contain that person's name. Then, by using the RSC clustering, these faces are organized into clusters and only representative faces of these clusters are introduced to users. As a result, the retrieval performance is significantly increased since only a small number of faces belonging to the clusters relevant to the target person are returned instead of all initial retrieved faces. The contribution of new RSC clustering model for this problem is two-fold: First, it can automatically determine an appropriate number of clusters and discards a large number of irrelevant faces. Second, since the precision of clusters can be controlled easily by a threshold, high precision clusters can be obtained and treated as labeled sets so that they can be used for feature selection by using the linear discriminant analysis (LDA) method. Experiments on the TRECVID 2004 dataset showed that the retrieval performance of the proposed method outperforms other existing methods.

This paper proposes a method for generating image-based quizzes from news video achieves. Although there are many types of quizzes, in this work we focus on matching quizzes in which an image is to be matched to one of several choices that are statements. The key to making a successful quiz of this type is to extract choice statements that are similar but nonidentical to the true statement regarding the image. For image selection, we need to select images that are suitable for image-based quizzes. In this paper, we report some preliminary work and highlight some interesting issues in making quizzes. We also propose a method for automatically generating quizzes using feature-based clustering analyses and assess the outcomes of trials actually using our method.

We introduce a video browsing interface 'mediaWalker' that lets users explore a news video archive based on a time-series semantic structure; the 'topic thread' structure. The interface lets users efficiently track up and down the development of news in an archive with more than 1,000 hours of video.

Automatic semantic classification of image databases is very useful for users searching and browsing but it is at the same time a very challenging research problem. Local features based image classification is one of the promising way to bridge the semantic gap in detecting concepts. This paper proposes a framework for incorporating contextual information into the concept detection process. The proposed method combines local and global classifiers (SVMs) with stacking. We studied the impact of topologic and semantic contexts in concept detection performance and proposed solutions to handle the large amount of dimensions involved in classified data. We conducted experiments on TRECVID'04 data set with 48104 images and 5 concepts. We found that the use of context yields a significant improvement both for the topologic and semantic contexts

This paper describes an efficient feature selection method that quickly selects a small subset out of a given huge feature set; for building robust object detection systems. In this filter-based method, features are selected not only to maximize their relevance with the target class but also to minimize their mutual dependency. As a result, the selected feature set contains only highly informative and non-redundant features, which significantly improve classification performance when combined. The relevance and mutual dependency of features are measured by using conditional mutual information (CMI) in which features and classes are treated as discrete random variables. Experiments on different huge feature sets have shown that the proposed CMI-based feature selection can both reduce the training time significantly and achieve high accuracy.

We propose a method that semi-automatically composes video stories by connecting individual stories in a news video archive along a topic-based semantic structure, namely the topic thread. We introduce the methods to realize the composition, namely, story segmentation, topic threading and clustering. We then evaluate the proposed approach based on preliminary tests. Since the thread structure reflects the development of topics in the real-world, we believe that the composed news video story should be effective for the user to gain a deeper understanding of the current topic of interest

A method has been developed to identify video shots of the same scene where camera flash lights are observed, and the method has been tested by using it to detect such shots from a large TV video archive. Camera flashes are often used in impressive scenes, such as interviews of important persons. Because such scenes are broadcasted repeatedly on various TV programs, a method for detecting them is a promising approach for semantic video indexing. The proposed identification method is invariant to the differences in viewpoint, illumination or any other visual environment because it depends on comparison between temporal occurrence patterns of flash lights. Furthermore, because each flash pattern is represented with a binary array, the comparison requires low computing cost. These advantages mean that the proposed method can be considered to provide semantic and efficient video analysis.

As the content of everyday news reports is unpredictable, keyword based news search engine can't provide effective services to audiences because the audiences may not be able to figure out proper keywords to search. In this paper, a novel framework is proposed to help audiences browse and retrieve news video clips without the need of keywords. Interesting keyframes and keywords are automatically extracted from news video clips and visually represented according to their interestingness and informativeness measurement. A computational approach is also developed to quantify the interestingness measurement of video clips. The keyframes and keywords are carefully organized so that the audiences can find news stories of interest at first glance.

We are developing a cooking navigation system, which helps even a novice user to cook several recipes in parallel without failure, while improving an advanced user's skill further. To realize this, the system optimizes the cooking procedure considering the following restrictions: (1) Duration of cooking, (2) Accuracy of cooking, and (3) Learning effect, by providing appropriate instructions to user's at the right timing, making full use of multimedia information. The users should be able to cook perfectly and comfortably just by following the text, video and audio provided by the system. According to the result of a preliminary experiment, all users from novice to experienced cooks could finish two dishes in parallel while enjoyeing the cooking very much. The result of a questionnaire shows the effectiveness of the multimedia navigation that we propose.

We evaluated the interactive retrieval functionality of the Concentric Ring View according to a series of usability studies. This is a ring structure-based graphical user interface, like a planisphere, for image retrieval with multi-faceted metadata. Attribute values for each facet are arranged on a ring, and retrieved images are displayed inside using search keys derived from the attribute values on the bottom part of the rings. By rotating the rings, users can browse retrieved images while adjusting search keys. The first usability test conducted with thirty six participants confirmed that: (i) novice users, even junior high school students, could use this interface; (ii) users could find images better than anticipated; and (iii) the interface was good at choosing the first relevant image, but users could not refine retrieval because they were unable to reuse retrieved results. To solve this problem, we added two functionalities, personal history for reuse and relevance feedback. With these improvements, we named the new version of the interface Concentric Ring View F+. A second usability test with seven participants confirmed the effectiveness of this newer interface.

We propose a topic-based inter-video news video corpus structuring method and a visual interface to efficiently browse through the structured corpus. Such inter-video structuring was not deeply sought in previous works. The topic-based structure is analyzed by closed-caption text analysis; topic segmentation and tracking. The visual interface provides the ability to 1) search and select a topic by query terms and 2) track a topic thread interactively referring to the text analysis results. Although topic retrieval is somewhat similar to conventional video retrieval methods, the combination with topic tracking makes it remarkably easy to narrow down the results that match a user's interest and moreover reveal underlying content-based structures, where the structure itself contains rich information.

Nearest neighbor (NN) search in high dimensional feature space is widely used for similarity retrieval of multimedia information. However recent research results in the database literature reveal that a curious problem happens in high dimensional space. Since high dimensional space has a high degree of freedom, points could be scattered so that every distance between them might yield no significant difference. In this case, we can say that the NN is indistinctive because many points exist at the similar distance. To make matters worse, indistinctive NNs require more search cost because search completes only after choosing the NN from plenty of strong candidates. In order to circumvent the handful effect of indistinctive NNs, the paper presents a new NN search algorithm which determines the distinctiveness of the NN during search operation. This enables us not only to cut down search cost but also to distinguish distinctive NNs from indistinctive ones. These advantages are especially beneficial to interactive retrieval systems.

The paper presents a method to automatically extract subject regions to be used as a key for image retrieval. If one wants to look up a particular object in an image database, a key image region of the object should be properly indicated to the system. However, this task is not trivial. In our approach, we assume that we have an actual object in the real world to be looked up in the image database. Taking advantage of disparity images using a commercially available stereo range finder, we automate subject region extraction. We assume that 3-D points in a dense disparity map yield a multimodal Gaussian probability distribution, and subject regions are extracted by properly selecting particular modes of Gaussian densities. We demonstrate that the system somehow extracts regions which correspond to "subjects." The system achieves adaptive and quick extraction, and it enables experimental image retrieval by a real object system which runs interactively.

The paper presents a comparative evaluation of matching methods of face sequences obtained from actual videos. Face information is quite important in videos, especially in news programs, dramas, and movies. Accurate face sequence matching enables many multimedia applications including content-based face retrieval, automated face annotation, video authoring, etc. However, face sequences in videos are subject to variation in lighting condition, pose, facial expression, etc., which cause difficulty in face matching. In order to cope with this problem, several face sequence matching methods are proposed by extending face still image matching, traditional pattern recognition, and recent pattern recognition techniques. They are expected to be applicable to face sequences extracted from actual videos. The performance of these methods is evaluated as the accuracy of face sequence annotation using the methods. The accuracy is evaluated using a considerable amount of actual drama videos. The evaluation results reveal merits and demerits of these methods, and indicate future research directions of face matching for videos.

This paper addresses to achieve an automated actor/actress annotation system for drama videos.As key technologies, face sequence extraction and matching methods applicable to actual drama videos are presented.The methods are then applied to realize a semi-automated actor/actress annotation system.After the system is trained using an episode of a drama series by given several tens of manual annotations to face sequences, it can automatically annotate face sequences not only of the trained episode, but of the other episodes of the drama series.Experimental results show that the system achieves around 70% accuracy in face sequence annotation using 40 manual annotations as the training set.

No abstract available.

This paper proposes a novel approach to extract meaningful content information from video by collaborative integration of image understanding and natural language processing. As an actual example, we developed a system that associates faces and names in videos, called Name-It, which is given news videos as a knowledge source, then automatically extracts face and name association as content information. The system can infer the name of a given unknown face image, or guess faces which are likely to have the name given to the system. This paper explains the method with several successful matching results which reveal effectiveness in integrating heterogeneous techniques as well as the importance of real content information extraction from video, especially face-name association.

Recently, similarity queries on feature vectors have been widely used to perform content-based retrieval of images. To apply this technique to large databases, it is required to develop multidimensional index structures supporting nearest neighbor queries efficiently. The SS-tree had been proposed for this purpose and is known to outperform other index structures such as the R*-tree and the K-D-B-tree. One of its most important features is that it employs bounding spheres rather than bounding rectangles for the shape of regions. However, we demonstrate in this paper that bounding spheres occupy much larger volume than bounding rectangles with high-dimensional data and that this reduces search efficiency. To overcome this drawback, we propose a new index structure called the SR-tree (Sphere/Rectangle-tree) which integrates bounding spheres and bounding rectangles. A region of the SR-tree is specified by the intersection of a bounding sphere and a bounding rectangle. Incorporating bounding rectangles permits neighborhoods to be partitioned into smaller regions than the SS-tree and improves the disjointness among regions. This enhances the performance on nearest neighbor queries especially for high-dimensional and non-uniform data which can be practical in actual image/video similarity indexing. We include the performance test results the verify this advantage of the SR-tree and show that the SR-tree outperforms both the SS-tree and the R*-tree.

The article describes a framework for electronic library systems incorporating automated generation of electronic library schema from paper printed materials, offering a hypertext style browsing interface. On our framework, document image understanding techniques are applied to table-of-content images of academic journals to automatically acquire bibliographic database schema and hypertext schema. The system should only provide typical hypertext links as implicit links in the form of "functions." Although the basic framework of automatic generation of hypertext schema has been explained before, we describe an overview of an experimental electronic library system CyberMagazine based on this framework to achieve basic electronic library facilities.

A syntactic rule learning method is presented for analyzing document images and constructing a database from them. This method is used in a digital library system named CyberMagazine, where document images are sequentially converted into database tuples by block segmentation, rough classification, and syntactic analysis. The syntactic rule has an ability to analyze symbols located in two dimensional plane, and has a syntax similar to an ordinal context free grammar except for the concatenation of symbols. In the presented learning method, the syntactic rules are generated from a set of parse trees by decomposing the trees according to non terminal symbols, generalizing the decomposed trees to a syntactic rule, and merging them.

A new method is presented for analyzing document images and constructing a database from them. This method is implemented in an electronic library system named CyberMagazine, where document images are sequentially converted into database tuples by block segmentation, rough classification, and syntactic analysis. CyberMagazine's image understanding method combines the use of decision tree classification and syntactic analysis using a newly presented matrix grammar. In order to combine the classification and the syntactic analysis steps, the authors propose an augmentation technique for carrying out decision tree classification. Experimental results subsequently demonstrate that high understanding accuracy is obtained by adequate augmentation.

A drawing understanding system using state transition models is described. This system performs flexible understanding following bottom-up and top down rules, which are easily derived. However, the system is sensitive to minor changes in drawings and has difficulty determining threshold values. A rule learning facility for overcoming these problems was added and evaluated. The system can generate rules from interactions with users using a machine learning method. It was confirmed that this system can generate proper rules for practical drawing understanding from plain interactions.>

A new method of collaborative unification between document image understanding and hypertext construction is presented. Document image understanding is indispensable to electronic library systems, but document understanding technologies are still immature. Moreover, hypertext links are difficult to acquire by hand. In the approach presented, document image understanding is taken as classification of text blocks to classes of bibliographic items which compose the understanding thesaurus. Hypertext links are obtained implicitly with functions corresponding to classes, and thus they are obtained automatically from understanding results. These classes include incompletely recognized classes distinctly to offer utilization of incompletely recognized text blocks as they are. Using this approach, large scale and practical electronic library systems can be offered.>

Recently, multi-purpose and generalized image/pattern recognition methods are required. To achieve this, it is desired to realize an automated recognition rule generation system. In this paper, a learning system of structural description rules which are used as models for the model based vision is presented. This system can generate structural description rules efficiently using partially structured example figure descriptions, and a proper inductive inference algorithm for generating structural description rules.>

A flexible drawing understanding system with state transition models is proposed. The drawing processor AI-Mudams (written in C) is used as the token extractor in the embodiment discussed. Given drawing images are converted efficiently to suitable geometrical primitives, such as contour vectors, core vectors, dots loops, or, in some cases, primitives with semantics (road line, or house etc.). The understanding system kernel is implemented in Prolog, and the geometrical evaluator is also prepared in C for checking basic geometrical situations, including shape, geometrical relations, and allocations. This understanding kernel accepts the individual state transition rules corresponding to individual drawing images and recognition targets and realizes understanding in the form of bottom-up and top-down state transition. Experiments on different types of drawings reveal that the framework is flexible and effective for various kinds of drawing image.>