Single Image Dehazing is a challenging ill-posed problem. While Transformers excel in long-range modeling, the low-pass characteristic of their self-attention further obscures high-frequency scene details attenuated by scattering. Furthermore, haze-degraded encoder features exhibit Mean-Elevation and Variance-Compression, leading to a significant statistical discrepancy with clear decoder features that hinders effective feature fusion. To address these limitations, this paper proposes the Frequency-Domain Stepwise and Moment Adjustment Network (FDSMA-Net). First, we design the High-Frequency Disentangled Attention (HFDA) mechanism, which explicitly decouples high-frequency components via attention inversion and utilizes a Dual Dynamic Frequency Modulation mechanism to calibrate and enhance details, effectively overcoming the spectral bias of standard attention. Second, the Moment-Driven Channel Fusion Attention (MDCFA) mechanism is introduced in the skip connections of the U-Net architecture. By explicitly aligning the first-order moment and second-order moment, MDCFA dynamically corrects the distribution shift caused by haze, ensuring statistically aligned feature fusion. Enabled by frequency disentanglement and moment alignment, FDSMA-Net excels on synthetic and real benchmarks, effectively recovering scene details and enhancing visual clarity. The source code of our proposed FDSMA-Net will be released at https://github.com/SWU-CS-MediaLab/FDSMA-Net.

Depth completion reconstructs dense depth maps from sparse LiDAR measurements and synchronized RGB images. Existing methods mainly face two challenges: the texture-geometry ambiguity and the topological rigidity propagation. To overcome these limitations, we propose a unified framework of YCbCr color space and the Dual-path Attention Network (YDANet), which treats the depth completion pipeline as a Disentanglement-Regression-Propagation paradigm. In the YDANet, we first introduce the YCbCr-Guided Adaptive Fusion (YGAF) module, which explicitly disentangles structural luminance from texture-rich chrominance and exploits intrinsic illumination-depth correlations to mitigate the texture-copying artifacts common in RGB-based methods. Then, to bridge the semantic gap between the backbone and refinement heads, we propose the Dual-Path Attention (DPA) module. Functioning as a lightweight Guidance-Refiner, DPA employs Spatial Reduction Attention (SRA) for global context and CNN-CBAM for local fidelity, utilizing a channel-wise fusion mechanism to distill clean affinity priors. Additionally, we incorporate the Gated Deformable Spatial Propagation Network (GD-SPN) to overcome the topological rigidity propagation. By employing a competitive gating mechanism to ensemble heterogeneous propagation kernels and sharing learned geometric parameters across iterations, GD-SPN achieves efficient non-rigid refinement. Extensive experiments demonstrate that YDANet achieves state-of-the-art performance among all 2D methods on the NYU Depth v2 dataset and ranks first on the KITTI depth completion online leaderboard among real-time approaches (< 80 ms) at the time of submission. The source code of our proposed YDANet will be released at https://github.com/SWU-CS-MediaLab/YDANet.

Many few-shot learning (FSL) approaches exploit Euclidean Distance as the metric, as it provides a stable and well-understood geometric structure for prototype-based classification. Despite these advantages, we observe that Euclidean Distance metric may suffer from a metric-level limitation under few-shot supervision. Meanwhile, with limited supervision, cosine-based scoring often leads to a metric over-smoothing phenomenon, where similarity scores among competing prototypes become overly compressed, producing weak effective margins and less stable decision boundaries, resulting in more ambiguous predictions when distinguishing fine-grained novel categories. In this work, we identify this metric-level limitation and propose a problem-driven prototype similarity framework, termed ProtoSimNet, for FSL which explicitly performs score-level distribution shaping of prototype similarities under data-scarce episodic supervision. ProtoSimNet introduces a cosine sharpening transformation, combined with an elastic regularization, which together enhance discriminability by amplifying the margins between high- and low-confidence prototype similarity scores while regularizing the encoder toward compact and less redundant feature embeddings. Our method strengthens the reliability of cosine-based classification and improves the robustness of prototype matching when only a few labeled samples are available. Experimental results on FSL benchmark datasets show that ProtoSimNet achieves consistent improvements over standard Prototypical Networks and competitive or superior performance compared with representative state-of-the-art methods.

Continuous Learning (CL) enables deep neural networks to adapt to evolving data while preserving performance on previously learned tasks, without access to all historical data. A significant challenge in Continuous Learning is the risk of over-fitting to limited new-class data and catastrophic forgetting of prior knowledge. Although recent methods have demonstrated that pre-trained Vision Transformers (ViTs) with prompt tuning can mitigate these issues, these methods rely on fixed, learnable prompts, which limit knowledge transfer and lead to sub-optimal performance. To address these limitations, we propose a ViT-Enhanced Prompts (VEP) framework for the tasks of continuous learning based on the pre-trained ViTs. Our VEP extracts highly distinctive global and local feature representations from the Multi-Head Self-Attention (MSA) and Multi-Layer Perceptron (MLP) modules of ViT, integrating them with dynamic prompts to enhance knowledge learning and transferring. Additionally, we introduce a diversity loss mechanism to reduce redundant information accumulation in the learned prompts and enhance their discrimination, aiming to mitigate catastrophic forgetting effectively. Extensive experiments on benchmark datasets, including ImageNet-R, CUB200, and CIFAR100, validate the contribution of each component in VEP and demonstrate that VEP achieves state-of-the-art performance on the task of Continuous Learning while maintaining low parameter overhead. The source code of our designed VEP is at https://github.com/SWU-CSMediaLab/VEP.

Most existing Visible-Infrared Person Re-Identification (VI-ReID) methods primarily rely on fixed datasets for training, which struggle to accommodate continuously evolving cross-domain data, thus significantly limiting the adaptation in real-world dynamic scenarios. The task of Lifelong Visible-Infrared Person Re-Identification (LVI-ReID) emerged and is required to overcome the challenge of the semantic gap caused by both cross-modality and cross-domain data. Drawing inspiration from complementary learning systems, we propose a prompt-based dynamic learning framework to address the challenges inherent in LVI-ReID. Specifically, we design a Prompt Pool (PP) module to encapsulate shared knowledge across tasks or domains. In addition, we propose an instance-level prompt generator (IPG) to further enhance the model's ability to capture domain-specific knowledge, overcoming the limitations of a fixed-size prompt pool. For task-agnostic inference during the LVI-ReID phase, we develop a query-key mechanism that adaptively selects the most relevant prompt by evaluating the similarity between query tokens and keys, thereby addressing the nuanced requirements of varying tasks. Extensive experimental evaluations demonstrate the superiority of our proposed prompt learning-based PP-IPG framework over state-of-the-art methods in both lifelong learnings, lifelong person re-identification (LReID), and LVI-ReID settings. These results underscore the efficacy and practicality of our framework for advancing LVI-ReID across dynamic cross-modality and cross-domains. The source code of our designed PP-IPG method is at https://github.com/SWU-CSMediaLab/PP-IPG.

The existing arbitrary style transfer methods mainly suffer two challenges. One is content integrity, as most methods focus too much on style, resulting in incomplete content information and missing details. The other is style consistency, which requires more exploration of style information to alleviate pattern and texture inconsistencies. Thus, a framework of Content Integrity and Style Consistency preserving arbitrary style transfer (CISC-ST) is proposed, which consists of a Dual Style Attention (DSA) mechanism and Frequency Domain Enhancement (FDE) architecture. The DSA aims to explore more style information and generate finer-grained style details. The FDE aims to enhance the overall content information to improve content preservation. Qualitative and quantitative experiments show that our method can produce stylized results that are more detailed in content and visually harmonious. The source code of our CISC-ST is available at https://github.com/SWU-CS-MediaLab/CISC-ST.

Since the diverse geometric appearances and densities of rain streaks, local-global information is equally essential for single image deraining. Balancing local-global information becomes a challenge. Thus, a Multi-Scale Fusion of Gated Neighborhood Attention Transformers (MSF-GNAT) for single image deraining is proposed in this paper. Firstly, a Gated Neighborhood Attention Transformer (GNAT) block is designed to achieve complex condition deraining by complementarily fusing global and local information. Secondly, a Multi-Scale Fusion (MSF) block is developed to fuse multi-scale information for richer representation. Moreover, a Simplified Gated Feed-forward Network (SGFN) is proposed to ensure that the convolutional results primarily attend to valid pixels. Evaluation experiments on synthetic and real-world datasets demonstrate the superiority of our MSF-GNAT over state-of-the-art methods. The source code is available at https://github.com/SWU-CS-MediaLab/MSF-GNAT.

Sketch-based 3D shape retrieval aims to retrieve similar 3D shapes given a 2D sketch query. Although this task has been studied for years, the inherent cross-modal gap and data imbalance between 2D sketches and 3D shapes remain challenging. To address the problems, we propose a simple and effective framework based on Multi-view Fusion Transformer. To be specific, we project 3D shapes into twelve distinct views, and their CNN features are combined with position embeddings, passing together into a transformer encoder to learn view weights. Then we process them through average pooling and MLPs to obtain the final 3D shape representation. Furthermore, to narrow the data imbalance between 2D sketches and 3D shapes, affine transformation and elastic deformation are fully utilized for sketch augmentation, so as to extract more comprehensive sketch features for feature matching with the multi-view 3D shape representation. Extensive experiments on SHREC13, SHREC14 and PART-SHREC14 datasets demonstrate our method achieves superior performance than previous competitive methods.

Visible-infrared person re-identification has been extensively explored, but it typically relies on stationary datasets for training. However, data is collected in a streaming manner in practical scenarios, necessitating the model's ability to continually learn without forgetting previous tasks. Existing methods focus simply on the lifelong single-modality person re-identification, but the visible images are sometimes unavailable, e.g., at night. To this end, this paper introduces a more challenging yet practical problem: Lifelong Visible-Infrared Person Re-identification (LVI-ReID). Inspired by the complementary learning systems, we propose a Tri-Token transformer with a Query-Key mechanism (TTQK) to tackle the LVI-ReID. Firstly, a general token is designed to capture robust domain-general features, shared across different domains, aiming to enhance the generalization capability. Subsequently, recognizing that different domains possess unique features like illumination and scenes, we allocate a specific token for each domain to extract significant domain-specific features, aiming to enhance the adaptability across domains. Furthermore, to prevent using the task identifier in the inference stage of LVI-ReID, we design a query-key mechanism to adaptively select the appropriate specific token based on the similarity between the query token and keys. Extensive experiments demonstrate that our method outperforms other lifelong learning and LReID methods. The source code of our designed LVI-ReID method is at https://github.com/SWU-CS-MediaLab/TTQK.

Few-Shot Class-Incremental Learning (FSCIL) aims to keep recognizing novel classes from a limited number of samples after training on abundant data from base classes while maintaining the performance of the old classes. The challenge, however, is that limited data from new classes not only leads to the issue of overfitting but also catastrophic forgetting. To address these two issues, we propose a causal inference strategy in the mainstream FSCIL framework, which encourages the model to learn significant knowledge in the base training session and enhance the model's ability to extract features to cope with the emergence of unseen classes in the incremental session, by improving the learning of causal relationships between features and predictions for perturbed samples. In addition, to improve the effectiveness of learning new tasks in the incremental sessions while preventing the model from overfitting to the novel class data, we freeze the feature extractor while adding a Fourier transform after the feature extractor in the incremental session. It can denoise the features, strengthen the features of the novel classes, and suppress the error in extracting the features of the limited number of samples directly from the feature extractor. Extensive experiments on CIFAR100, Caltech-USCD Birds-200-2011, and miniImageNet datasets show that our proposed framework achieves state-of-the-art performance on FSCIL. The source code of our designed framework is at https://github.com/SWU-CS-MediaLab/CIFSCIL.

Audio-Visual Question Answering (AVQA) is a sophisticated QA task, which aims at answering textual questions over given video-audio pairs with comprehensive multimodal reasoning. Through detailed causal-graph analyses and careful inspections of their learning processes, we reveal that AVQA models are not only prone to over-exploit prevalent language bias, but also suffer from additional joint-modal biases caused by the shortcut relations between textual-auditory/visual co-occurrences and dominated answers. In this paper, we propose a COllabrative CAusal (COCA) Regularization to remedy this more challenging issue of data biases. Specifically, a novel Bias-centered Causal Regularization (BCR) is proposed to alleviate specific shortcut biases by intervening bias-irrelevant causal effects, and further introspect the predictions of AVQA models in counterfactual and factual scenarios. Based on the fact that the dominated bias impairing model robustness for different samples tends to be different, we introduce a Multi-shortcut Collaborative Debiasing (MCD) to measure how each sample suffers from different biases, and dynamically adjust their debiasing concentration to different shortcut correlations. Extensive experiments demonstrate the effectiveness as well as backbone-agnostic ability of our COCA strategy, and it achieves state-of-the-art performance on the large-scale MUSIC-AVQA dataset.

Visual Question Answering (VQA) has achieved significant success over the last few years, while most studies focus on training a VQA model on a stationary domain (e.g., a given dataset). In real-world application scenarios, however, these methods are often inefficient because VQA systems are always supposed to extend their knowledge and meet the ever-changing demands of users. In this paper, we introduce a new and challenging multi-domain lifelong VQA task, dubbed MDL-VQA, which encourages the VQA model to continuously learn across multiple domains while mitigating the forgetting on previously-learned domains. Furthermore, we propose a novel replay-free Self-Critical Distillation (SCD) framework tailor-made for MDL-VQA, which alleviates forgetting issue via transferring previous-domain knowledge from teacher to student models. First, we propose to introspect the teacher's understanding over original and counterfactual samples, thereby creating informative instance-relevant and domain-relevant knowledge for logits-based distillation. Second, on the side of feature-based distillation, we propose to introspect the reasoning behavior of student model to establish the harmful domain-specific knowledge acquired in current domain, and further leverage the metric learning strategy to encourage student to learn useful knowledge in new domain. Extensive experiments demonstrate that SCD framework outperforms state-of-the-art competitors with different training orders.

This paper presents a new setting for visual question answering (VQA) called personalized federated VQA (FedVQA) that addresses the growing need for decentralization and data privacy protection. FedVQA is both practical and challenging, requiring clients to learn well-personalized models on scene-specific datasets with severe feature/label distribution skews. These models then collaborate to optimize a generic global model on a central server, which is desired to generalize well on both seen and unseen scenes without sharing raw data with the server and other clients. The primary challenge of FedVQA is that, client models tend to forget the global knowledge initialized from central server during the personalized training, which impairs their personalized capacity due to the potential overfitting issue on local data. This further leads to divergence issues when aggregating distinct personalized knowledge at the central server, resulting in an inferior generalization ability on unseen scenes. To address the challenge, we propose a novel federated pairwise preference preserving (FedP3) framework to improve personalized learning via preserving generic knowledge under FedVQA constraints. Specifically, we first design a differentiable pairwise preference (DPP) to improve knowledge preserving by formulating a flexible yet effective global knowledge. Then, we introduce a forgotten-knowledge filter (FKF) to encourage the client models to selectively consolidate easily-forgotten knowledge. By aggregating the DPP and the FKF, FedP3 coordinates the generic and the personalized knowledge to enhance the personalized ability of clients and generalizability of the server. Extensive experiments show that FedP3 consistently surpasses the state-of-the-art in FedVQA task.

Abstract Few-shot learning aims at recognizing novel visual categories from very few labelled examples. Different from the existing few-shot classification methods that are mainly based on metric learning or meta-learning, in this work we focus on improving the representation capacity of feature extractors. For this purpose, we propose a new two-stage dual selective knowledge transfer (DSKT) framework, to guide models towards better optimization. Specifically, we first exploit an improved multi-task learning approach to train a feature extractor with robust representation capability as a teacher model. Then, we design an effective dual selective knowledge distillation method, which enables the student model to selectively learn knowledge from the teacher model and current samples, thereby improving the student model’s ability to generalize on unseen classes. Extensive experimental results show that our DSKT achieves competitive performances on four well-known few-shot classification benchmarks.

In recent years a vast amount of visual content has been generated and shared from many fields, such as social media platforms, medical imaging, and robotics. This abundance of content creation and sharing has introduced new challenges, particularly that of searching databases for similar content - Content Based Image Retrieval (CBIR) - a long-established research area in which improved efficiency and accuracy are needed for real-time retrieval. Artificial intelligence has made progress in CBIR and has significantly facilitated the process of instance search. In this survey we review recent instance retrieval works that are developed based on deep learning algorithms and techniques, with the survey organized by deep feature extraction, feature embedding and aggregation methods, and network fine-tuning strategies. Our survey considers a wide variety of recent methods, whereby we identify milestone work, reveal connections among various methods and present the commonly used benchmarks, evaluation results, common challenges, and propose promising future directions.

In this paper, we introduce a challenging yet practical setting for person re-identification (ReID) task, named lifelong person re-identification (LReID), which aims to continuously train a ReID model across multiple domains and the trained model is required to generalize well on both seen and unseen domains. It is therefore critical to learn a ReID model that can learn a generalized representation without forgetting knowledge of seen domains. In this paper, we propose a new MEmorizing and GEneralizing framework (MEGE) for LReID, which can jointly prevent the model from forgetting and improve its generalization ability. Specifically, our MEGE is composed of two novel modules, i.e., Adaptive Knowledge Accumulation (AKA) and differentiable Ranking Consistency Distillation (RCD). Taking inspiration from the cognitive processes in the human brain, we endow AKA with two special capacities, knowledge representation and knowledge operation by graph convolution networks. AKA can effectively mitigate catastrophic forgetting on seen domains while improving the generalization ability to unseen domains. By considering the ranking factor that is specifically important in ReID, RCD is designed to distill the ranking knowledge in a differentiable manner, which can further prevent the catastrophic forgetting. To supporting the study of LReID, we build a new and large-scale benchmark with two practical evaluation protocols that consider the metrics of non-forgetting and generalization. Experiments demonstrate that 1) our MEGE framework can effectively improve the performance on seen and unseen domains under the domain-incremental learning constraint, and that 2) the proposed MEGE outperforms state-of-the-art competitors by large margins.

Fine-grained image retrieval has been extensively explored in a zero-shot manner. A deep model is trained on the seen part and then evaluated the generalization performance on the unseen part. However, this setting is infeasible for many real-world applications since (1) the retrieval dataset can be non-fixed so that new data are added constantly, and (2) data samples of the seen categories are also common in practice and are important for evaluation. In this paper, we explore lifelong fine-grained image retrieval (LFGIR), which learns continuously on a sequence of new tasks with data from different datasets. We first use knowledge distillation to minimize catastrophic forgetting on old tasks. Training continuously on different datasets causes large domain shifts between the old and new tasks while image retrieval is sensitive to even small shifts in the features. This tends to weaken the effectiveness of knowledge distillation by the frozen teacher. To mitigate the impact of domain shifts, we use the network inversion method to generate images of the old tasks. In addition, we design an on-the-fly teacher which transfers knowledge captured on a new task to the student to improve better generalization performance, thereby achieving a better balance between old and new tasks in the end. We name the whole framework as Dual Knowledge Distillation (DKD), whose efficacy is demonstrated by extensive experimental results on sequential tasks including seven datasets.

Current VQA models are suffering from the problem of overdependence on language bias, which severely reduces their robustness in real-world scenarios. In this paper, we analyze VQA models from the view of forward/backward chaining in the inference engine, and propose to enhance their robustness via a novel Bidirectional Chaining (VQA-BC) framework. Specifically, we introduce a backward chaining with hardnegative contrastive learning to reason from the consequence (answers) to generate crucial known facts (question-related visual region features). Furthermore, to alleviate the overconfident problem in answer prediction (forward chaining), we present a novel introspective regularization to connect forward and backward chaining with label smoothing. Extensive experiments verify that VQA-BC not only effectively overcomes language bias on out-of-distribution dataset, but also alleviates the over-correct problem caused by ensemble-based method on in-distribution dataset. Compared with competitive debiasing strategies, our method achieves state-of-the-art performance to reduce language bias on VQA-CP v2 dataset.

Lifelong person re-identification (LReID) is a challenging and emerging task, which concerns the ReID capability on both seen and unseen domains after learning across different domains continually. Existing works on LReID are devoted to introducing commonly-used lifelong learning approaches, while neglecting a serious side effect caused by using normalization layers in the context of domain-incremental learning. In this work, we aim to raise awareness of the importance of training proper batch normalization layers by proposing a new meta reconciliation normalization (MRN) method specifically designed for tackling LReID. Our MRN consists of grouped mixture standardization and additive rectified rescaling components, which are able to automatically maintain an optimal balance between domain-dependent and domain-independent statistics, and even adapt MRN for different testing instances. Furthermore, inspired by synaptic plasticity in human brain, we present a MRN-based meta-learning framework for mining the meta-knowledge shared across different domains, even without replaying any previous data, and further improve the model's LReID ability with theoretical analyses. Our method achieves new state-of-the-art performances on both balanced and imbalanced LReID benchmarks.

Deep learning for fine-grained image retrieval in an incremental context is less investigated. In this paper, we explore this task to realize the model’s continuous retrieval ability. That means, the model enables to perform well on new incoming data and reduce forgetting of the knowledge learned on preceding old tasks. For this purpose, we distill semantic correlations knowledge among the representations extracted from the new data only so as to regularize the parameters updates using the teacher-student framework. In particular, for the case of learning multiple tasks sequentially, aside from the correlations distilled from the penultimate model, we estimate the representations for all prior models and further their semantic correlations by using the representations extracted from the new data. To this end, the estimated correlations are used as an additional regularization and further prevent catastrophic forgetting over all previous tasks, and it is unnecessary to save the stream of models trained on these tasks. Extensive experiments demonstrate that the proposed method performs favorably for retaining performance on the already-trained old tasks and achieving good accuracy on the current task when new data are added at once or sequentially.

Person re-identification (ReID) methods always learn through a stationary domain that is fixed by the choice of a given dataset. In many contexts (e.g., lifelong learning), those methods are ineffective because the domain is continually changing in which case incremental learning over multiple domains is required potentially. In this work we explore a new and challenging ReID task, namely lifelong person re-identification (LReID), which enables to learn continuously across multiple domains and even generalise on new and unseen domains. Following the cognitive processes in the human brain, we design an Adaptive Knowledge Accumulation (AKA) framework that is endowed with two crucial abilities: knowledge representation and knowledge operation. Our method alleviates catastrophic forgetting on seen domains and demonstrates the ability to generalize to unseen domains. Correspondingly, we also provide a new and large-scale benchmark for LReID. Extensive experiments demonstrate our method outperforms other competitors by a margin of 5.8% mAP in generalising evaluation. The codes will be available at https://github.com/TPCD/LifelongReID.

According to current research, one of the major challenges in Visual Question Answering (VQA) models is the overdependence on language priors (and neglect of the visual modality). VQA models tend to predict answers only based on superficial correlations between the first few words in question and frequency of related answer candidates. To address this issue, we propose a novel Language Prior based Focal Loss (LP-Focal Loss) by rescaling the standard cross entropy loss. Specifically, we employ a question-only branch to capture the language biases for each answer candidate based on the corresponding question input. Then, the LP-Focal Loss dynamically assigns lower weights to biased answers when computing the training loss, thereby reducing the contribution of more-biased instances in the train split. Extensive experiments show that the LP-Focal Loss can be generally applied to common baseline VQA models, and achieves significantly better performance on the VQA-CP v2 dataset, with an overall 18% accuracy boost over benchmark models.

Most Visual Question Answering (VQA) models are faced with language bias when learning to answer a given question, thereby failing to understand multimodal knowledge simultaneously. Based on the fact that VQA samples with different levels of language bias contribute differently for answer prediction, in this paper, we overcome the language prior problem by proposing a novel Language Bias driven Curriculum Learning (LBCL) approach, which employs an easy-to-hard learning strategy with a novel difficulty metric Visual Sensitive Coefficient (VSC). Specifically, in the initial training stage, the VQA model mainly learns the superficial textual correlations between questions and answers (easy concept) from more-biased examples, and then progressively focuses on learning the multimodal reasoning (hard concept) from less-biased examples in the following stages. The curriculum selection of examples on different stages is according to our proposed difficulty metric VSC, which is to evaluate the difficulty driven by the language bias of each VQA sample. Furthermore, to avoid the catastrophic forgetting of the learned concept during the multi-stage learning procedure, we propose to integrate knowledge distillation into the curriculum learning framework. Extensive experiments show that our LBCL can be generally applied to common VQA baseline models, and achieves remarkably better performance on the VQA-CP v1 and v2 datasets, with an overall 20% accuracy boost over baseline models.

The focus of this survey is on the analysis of two modalities of multimodal deep learning: image and text. Unlike classic reviews of deep learning where monomodal image classifiers such as VGG, ResNet and Inception module are central topics, this paper will examine recent multimodal deep models and structures, including auto-encoders, generative adversarial nets and their variants. These models go beyond the simple image classifiers in which they can do uni-directional (e.g. image captioning, image generation) and bi-directional (e.g. cross-modal retrieval, visual question answering) multimodal tasks. Besides, we analyze two aspects of the challenge in terms of better content understanding in deep multimodal applications. We then introduce current ideas and trends in deep multimodal feature learning, such as feature embedding approaches and objective function design, which are crucial in overcoming the aforementioned challenges. Finally, we include several promising directions for future research.

Accurately matching visual and textual data in cross-modal retrieval has been widely studied in the multimedia community. To address these challenges posited by the heterogeneity gap and the semantic gap, we propose integrating Shannon information theory and adversarial learning. In terms of the heterogeneity gap, we integrate modality classification and information entropy maximization adversarially. For this purpose, a modality classifier (as a discriminator) is built to distinguish the text and image modalities according to their different statistical properties. This discriminator uses its output probabilities to compute Shannon information entropy, which measures the uncertainty of the modality classification it performs. Moreover, feature encoders (as a generator) project uni-modal features into a commonly shared space and attempt to fool the discriminator by maximizing its output information entropy. Thus, maximizing information entropy gradually reduces the distribution discrepancy of cross-modal features, thereby achieving a domain confusion state where the discriminator cannot classify two modalities confidently. To reduce the semantic gap, Kullback-Leibler (KL) divergence and bi-directional triplet loss are used to associate the intra- and inter-modality similarity between features in the shared space. Furthermore, a regularization term based on KL-divergence with temperature scaling is used to calibrate the biased label classifier caused by the data imbalance issue. Extensive experiments with four deep models on four benchmarks are conducted to demonstrate the effectiveness of the proposed approach.

Convolutional neural network training can suffer from diverse issues like exploding or vanishing gradients, scaling-based weight space symmetry and covariant-shift. In order to address these issues, researchers develop weight regularization methods and activation normalization methods. In this work we propose a weight soft-regularization method based on the Oblique manifold. The proposed method uses a loss function which pushes each weight vector to have a norm close to one, i.e. the weight matrix is smoothly steered toward the so-called Oblique manifold. We evaluate our method on the very popular CIFAR-10, CIFAR-100 and ImageNet 2012 datasets using two state-of-the-art architectures, namely the ResNet and wide-ResNet. Our method introduces negligible computational overhead and the results show that it is competitive to the state-of-the-art and in some cases superior to it. Additionally, the results are less sensitive to hyperparameter settings such as batch size and regularization factor.

Computational Fluid Dynamics (CFD) simulations are a very important tool for many industrial applications, such as aerodynamic optimization of engineering designs like cars shapes, airplanes parts etc. The output of such simulations, in particular the calculated flow fields, are usually very complex and hard to interpret for realistic three-dimensional real-world applications, especially if time-dependent simulations are investigated. Automated data analysis methods are warranted but a non-trivial obstacle is given by the very large dimensionality of the data. A flow field typically consists of six measurement values for each point of the computational grid in 3D space and time (velocity vector values, turbulent kinetic energy, pressure and viscosity). In this paper we address the task of extracting meaningful results in an automated manner from such high dimensional data sets. We propose deep learning methods which are capable of processing such data and which can be trained to solve relevant tasks on simulation data, i.e. predicting drag and lift forces applied on an airfoil. We also propose an adaptation of the classical hand crafted features known from computer vision to address the same problem and compare a large variety of descriptors and detectors. Finally, we compile a large dataset of 2D simulations of the flow field around airfoils which contains 16000 flow fields with which we tested and compared approaches. Our results show that the deep learning-based methods, as well as hand crafted feature based approaches, are well-capable to accurately describe the content of the CFD simulation output on the proposed dataset.

Visible-infrared person re-identification (VI-ReID) is a challenging and essential task in night-time intelligent surveillance systems. Except for the intra-modality variance that RGB-RGB person re-identification mainly overcomes, VI-ReID suffers from additional inter-modality variance caused by the inherent heterogeneous gap. To solve the problem, we present a carefully designed dual Gaussian-based variational auto-encoder (DG-VAE), which disentangles an identity-discriminable and an identity-ambiguous cross-modality feature subspace, following a mixture-of-Gaussians (MoG) prior and a standard Gaussian distribution prior, respectively. Disentangling cross-modality identity-discriminable features leads to more robust retrieval for VI-ReID. To achieve efficient optimization like conventional VAE, we theoretically derive two variational inference terms for the MoG prior under the supervised setting, which not only restricts the identity-discriminable subspace so that the model explicitly handles the cross-modality intra-identity variance, but also enables the MoG distribution to avoid posterior collapse. Furthermore, we propose a triplet swap reconstruction (TSR) strategy to promote the above disentangling process. Extensive experiments demonstrate that our method outperforms state-of-the-art methods on two VI-ReID datasets. Codes will be available at https://github.com/TPCD/DG-VAE.

Feature detectors and descriptors have been successfully used for various computer vision tasks, such as video object tracking and content-based image retrieval. Many methods use image gradients in different stages of the detection-description pipeline to describe local image structures. Recently, some, or all, of these stages have been replaced by convolutional neural networks (CNNs), in order to increase their performance. A detector is defined as a selection problem, which makes it more challenging to implement as a CNN. They are therefore generally defined as regressors, converting input images to score maps and keypoints can be selected with non-maximum suppression. This paper discusses and compares several recent methods that use CNNs for keypoint detection. Experiments are performed both on the CNN based approaches, as well as a selection of conventional methods. In addition to qualitative measures defined on keypoints and descriptors, the bag-of-words (BoW) model is used to implement an image retrieval application, in order to determine how the methods perform in practice. The results show that each type of features are best in different contexts.

Cross-modal hash retrieval has received considerable interest in the area of deep learning. Here hash codes of data of different modalities are learned where pair-wise loss functions control feature similarity in a shared embedding space. In this paper we improve on feature similarity by using Shannon's information entropy with respect to the modality information that is left in learning superior hash codes. We introduce a novel network for predicting the domain from the learned features while the protagonist network uses a loss function based on Shannon's information entropy to learn to maximize the domain uncertainty and therefore the information content. Additionally, according to the number of common labels between each similar image-text pair, we define a multi-level similarity matrix as supervisory information, which constrains all similar pairs with different weights. We show with extensive experiments that our novel approach to domain uncertainty leads to a cross-modal hash retrieval that outperforms the state-of-the-art.

Person re-identification (Re-ID) aims at matching images of the same person where images are captured by non-overlapping camera views distributed at different locations. To solve this problem, most recent works require a large pre-labeled dataset for training a deep model. These methods are not always suitable for real-world applications, because the latter often lack labeled data. In order to tackle this drawback, we proposed a novel Domain-Invariant Embedding Network (DIEN) to learn a domain-invariant embedding (DIE) feature by introducing a multi-loss joint learning with Recurrent Top- Down Attention (RTDA) mechanism. Due to the improvement in traditional triplet loss, our proposed model can benefit from both source-domain (labeled) data and target-domain (unlabeled) data. Furthermore, the resulting DIE feature not only has improved class discrimination but also robustness to domain shift. We compared our method with recent competitive algorithms and also evaluated the effectiveness of the proposed modules.

Fashion style transfer has attracted significant attention because it both has interesting scientific challenges and it is also important to the fashion industry. This paper focuses on addressing a practical problem in fashion style transfer, person-to-person clothing swapping, which aims to visualize what the person would look like with the target clothes worn on another person instead of dressing them physically. This problem remains challenging due to varying pose deformations between different person images. In contrast to traditional nonparametric methods that blend or warp the target clothes for the reference person, in this paper we propose a multistage deep generative approach named SwapGAN that exploits three generators and one discriminator in a unified framework to fulfill the task end-to-end. The first and second generators are conditioned on a human pose map and a segmentation map, respectively, so that we can simultaneously transfer the pose style and the clothes style. In addition, the third generator is used to preserve the human body shape during the image synthesis process. The discriminator needs to distinguish two fake image pairs from the real image pair. The entire SwapGAN is trained by integrating the adversarial loss and the mask-consistency loss. The experimental results on the DeepFashion dataset demonstrate the improvements of SwapGAN over other existing approaches through both quantitative and qualitative evaluations. Moreover, we conduct ablation studies on SwapGAN and provide a detailed analysis about its effectiveness.

General captioning practice involves a single forward prediction, with the aim of predicting the word in the next timestep given the word in the current timestep. In this paper, we present a novel captioning framework, namely Dual Prediction Network (DPN), which is end-to-end trainable and addresses the captioning problem with dual predictions. Specifically, the dual predictions consist of a forward prediction to generate the next word from the current input word, as well as a backward prediction to reconstruct the input word using the predicted word. DPN has two appealing properties: 1) By introducing an extra supervision signal on the prediction, DPN can better capture the interplay between the input and the target; 2) Utilizing the reconstructed input, DPN can make another new prediction. During the test phase, we average both predictions to formulate the final target sentence. Experimental results on the MS COCO dataset demonstrate that, benefiting from the reconstruction step, both generated predictions in DPN outperform the predictions of methods based on the general captioning practice (single forward prediction), and averaging them can bring a further accuracy boost. Overall, DPN achieves competitive results with state-of-the-art approaches, across multiple evaluation metrics.

Image-to-image translation between different domains has been an important research direction, with the aim of arbitrarily manipulating the source image content to become similar to a target image. Recently, cycle-consistent generative network (CycleGAN) has become a fundamental approach for general-purpose image-to-image translation, while almost no work has examined what factors may influence its performance. To provide more insights, we propose two new models roughly based on CycleGAN, namely Long CycleGAN and Nest CycleGAN. First, Long CycleGAN cascades several generators to perform the domain translation in a long cycle. It shows the benefit of stacking more generators on the generation quality. In addition to the long cycle, Nest CycleGAN develops new inner cycles to bridge intermediate generators directly, which can help constrain the unsupervised mappings. In the experiments, we conduct qualitative and quantitative comparisons for tasks including photo↔label, photo↔sketch, and photo colorization. The quantitative and qualitative results demonstrate the effectiveness of our two proposed models.

Computational Fluid Dynamics (CFD) simulations are able to produce complex and large outputs that accurately describe the physical properties of fluids and gases in various domains, such as air flow around a car, or the multi-phase flow inside an internal combustion engine. The simulation results, i.e. the flow fields, are often too complex to be analyzed directly. With the increasing number of simulations as well as their complexity, there is a need of automated processes that can analyze these complex outputs. In this paper, inspired by the success of convolutional neural networks (CNNs) in Computer Vision, we apply for the first time CNNs on CFD output. We show their capabilities in capturing and processing flow patterns. Furthermore, we design a novel CNN architecture tailored to the data produced by CFD simulations, as well as two conventional architectures and compare them. We propose and construct a new dataset of turbulent flow, within the application domain of steady flow around passenger cars. We use that dataset to evaluate and compare the proposed methods, on different tasks that depend on flow patterns. Finally, we compare our methods with a baseline k-nearest neighbor approach, tuned to be comparable to the state-of-the-art.

Objects are often organized in a semantic hierarchy of categories, where fine-level categories are grouped into coarse-level categories according to their semantic relations. While previous works usually only classify objects into the leaf categories, we argue that generating hierarchical labels can actually describe how the leaf categories evolved from higher level coarse-grained categories, thus can provide a better understanding of the objects. In this paper, we propose to utilize the CNN-RNN framework to address the hierarchical image classification task. CNN allows us to obtain discriminative features for the input images, and RNN enables us to jointly optimize the classification of coarse and fine labels. This framework can not only generate hierarchical labels for images, but also improve the traditional leaf-level classification performance due to incorporating the hierarchical information. Moreover, this framework can be built on top of any CNN architecture which is primarily designed for leaf-level classification. Accordingly, we build a high performance network based on the CNN-RNN paradigm which outperforms the original CNN (wider-ResNet) and also the current state-of-the-art. In addition, we investigate how to utilize the CNN-RNN framework to improve the fine category classification when a fraction of the training data is only annotated with coarse labels. Experimental results demonstrate that CNN-RNN can use the coarse-labeled training data to improve the classification of fine categories, and in some cases it even surpasses the performance achieved by fully annotated training data. This reveals that, CNN-RNN can alleviate the challenge of specialized and expensive annotation of fine labels.

In recent years, deep learning has been successfully applied to diverse multimedia research areas, with the aim of learning powerful and informative representations for a variety of visual recognition tasks. In this work, we propose convolutional fusion networks (CFN) to integrate multi-level deep features and fuse a richer visual representation. Despite recent advances in deep fusion networks, they still have limitations due to expensive parameters and weak fusion modules. Instead, CFN uses 1 × 1 convolutional layers and global average pooling to generate side branches with few parameters, and employs a locally-connected fusion module, which can learn adaptive weights for different side branches and form a better fused feature. Specifically, we introduce three key components of the proposed CFN, and discuss its differences from other deep models. Moreover, we propose fully convolutional fusion networks (FCFN) that are an extension of CFN for pixel-level classification applied to several tasks, such as semantic segmentation and edge detection. Our experiments demonstrate that CFN (and FCFN) can achieve promising performance by consistent improvements for both image-level and pixel-level classification tasks, compared to a plain CNN. We release our codes on https://github.com/yuLiu24/CFN . Also, we make a live demo ( goliath.liacs.nl ) using a CFN model trained on the ImageNet dataset.

Convolutional neural networks (CNNs) have attracted significant attention in visual recognition. Several recent studies have shown that, in addition to the fully connected layers, the features derived from the convolutional layers of CNNs can also achieve promising performance in image classification tasks. In this paper, we propose a new feature from the convolutional layers, called Bag of Surrogate Parts (BoSP), and its spatial variant, Spatial-BoSP (S-BoSP). The main idea is, we assume the feature maps in the convolutional layers as surrogate parts, and densely sample and assign image regions to these surrogate parts by observing the activation values. Together with BoSP/S-BoSP, we further propose another two schemes to enhance the performance: scale pooling and global-part prediction. Scale pooling aims to handle the objects with different scales and deformations, and global-part prediction combines the predictions of global and part features. By conducting extensive experiments on generic object, fine-grained object and scene datasets, we find the proposed scheme can not only achieve superior performance to the fully connected feature, but also produces competitive or, in some cases, remarkably better performance than the state of the art.

One challenging problem in semantic segmentation is due to the erroneous predictions between categorical foreground and cluttered background. To address it, we propose to utilize a fused loss function to train a fully convolutional network, which aims to enhance the discrimination between foreground and background in images. In addition, we propose a pixel objectness (POS) to measure the importance of pixels. POS is able to recover some missing foreground pixels from the background. Experimental results on the PASCAL VOC 2012 dataset demonstrate our approach can achieve considerable improvements compared with the baseline counterpart, while maintaining the ease of training deep networks.

We propose using relaxed deep supervision (RDS) within convolutional neural networks for edge detection. The conventional deep supervision utilizes the general groundtruth to guide intermediate predictions. Instead, we build hierarchical supervisory signals with additional relaxed labels to consider the diversities in deep neural networks. We begin by capturing the relaxed labels from simple detectors (e.g. Canny). Then we merge them with the general groundtruth to generate the RDS. Finally we employ the RDS to supervise the edge network following a coarse-to-fine paradigm. These relaxed labels can be seen as some false positives that are difficult to be classified. Weconsider these false positives in the supervision, and are able to achieve high performance for better edge detection. Wecompensate for the lack of training images by capturing coarse edge annotations from a large dataset of image segmentations to pretrain the model. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on the well-known BSDS500 dataset (ODS F-score of .792) and obtains superior cross-dataset generalization results on NYUD dataset.

In the well-known Bag-of-Words model, local features, such as the SIFT descriptor, are extracted and quantized into visual words. Then, an index is created to reduce computational burden. However, local clues serve as low-level representations that can not represent high-level semantic concepts. Recently, the success of deep features extracted from convolutional neural networks(CNN) has shown promising results toward bridging the semantic gap. Inspired by this, we attempt to introduce deep features into inverted index based image retrieval and thus propose the DeepIndex framework. Moreover, considering the compensation of different deep features, we incorporate multiple deep features from different fully connected layers, resulting in the multiple DeepIndex. We find the optimal integration of one midlevel deep feature and one high-level deep feature, from two different CNN architectures separately. This can be treated as an attempt to further reduce the semantic gap. Extensive experiments on three benchmark datasets demonstrate that, the proposed DeepIndex method is competitive with the state-of-the-art on Holidays(85:65% mAP), Paris(81:24% mAP), and UKB(3:76 score). In addition, our method is efficient in terms of both memory and time cost.

Image copy detection is a major challenge with regard to computational efficiency, memory requirements and accuracy. A popular approach from the literature is to use visual words (or Bag-of-Words) constructed from real value (SIFT and SURF) and binary string salient point descriptors (BRIEF, ORB, BRISK and FREAK). To accommodate large scale data sets, we used the approximate nearest neighbor (ANN) based cluster approach. Our results on several well-known test sets reveal that some of the recent binary string approaches outperformed notable descriptors such as SIFT and SURF.

In this paper, a method of browsing a large collection of images in a 3D environment is presented. Planetary bodies are used as a metaphor for browsing the image collection, where each planet represents a cluster of related images from the dataset. Salient point descriptors are used to cluster the imagery into planets. For each planet, maximum salient overlap is used to select a representative image. The user can move in 3D space through the planetary solar system. From a large distance, representative images painted on the planets are displayed. From a short distance, the images within the cluster are shown on the planet.

We present the Retina-inspired Invariant Fast Feature, RIFF, which was designed for invariance to scale, rotation, and affine image deformations. The feature descriptor is based on pair-wise comparisons over a sampling pattern loosely based on the human retina and introduces a method for improving accuracy by maximizing the discriminatory power of the point set. A performance evaluation with regard to bag of words based image retrieval on several well-known international datasets demonstrates that the RIFF descriptor has competitive performance to the state-of-the-art descriptors (e.g. SIFT, SURF, BRISK, and FREAK).

The world wide web is filled with billions of images and duplicates of images can frequently be found on many websites. These duplicates can be exact copies or differ slightly in their visual content. In this paper we provide a comparative study on how well content-based duplicate image detection methods are able to detect the duplicates of a query image. We conduct a survey to better understand in which ways such images on the internet differ from each other and use these observations to form a realistic and challenging duplicate image detection scenario. The methods we evaluate in our study are representative techniques from the research literature. In our evaluation, we target the performance of each method in relation to their descriptor size, description time and matching time, to assess their feasibility of application to large image collections (> 1 million).

Processing visual content in images and videos is a challenging task associated with the development of modern computer vision. Because salient point approaches can represent distinctive and affine invariant points in images, many approaches have been proposed over the past decade. Each method has particular advantages and limitations and may be appropriate in different contexts. In this paper we evaluate the performance of a wide set of salient point detectors and descriptors. We begin by comparing diverse salient point algorithms (SIFT, SURF, BRIEF, ORB, FREAK, BRISK, STAR, GFTT and FAST) with regard to repeatability, recall and precision and then move to accuracy and stability in real-time video tracking.

In this paper we present RetrievalLab, a content based retrieval tool that was designed for both educational and research purposes. It is a tool to facilitate the testing of new features, segmentations, machine learning approaches, and evaluation methods, by presenting a Matlab-like programming interface which illuminates the fundamental processes and algorithms in content based retrieval.

TOP-SURF is an image descriptor that combines interest points with visual words, resulting in a high performance yet compact descriptor that is designed with a wide range of content-based image retrieval applications in mind. TOP-SURF offers the flexibility to vary descriptor size and supports very fast image matching. In addition to the source code for the visual word extraction and comparisons, we also provide a high level API and very large pre-computed codebooks targeting web image content for both research and teaching purposes.

Experiential image retrieval systems aim to provide the user with a natural and intuitive search experience. The goal is to empower the user to navigate large collections based on his own needs and preferences, while simultaneously providing him with an accurate sense of what the database has to offer. In this paper we integrate a new browsing mechanism called deep exploration with the proven technique of retrieval by relevance feedback. In our approach, relevance feedback focuses the search on relevant regions, while deep exploration facilitates transparent navigation to promising regions of feature space that would normally remain unreachable. Optimal feature weights are determined automatically based on the evidential support for the relevance of each single feature. To achieve efficient refinement of the search space, images are ranked and presented to the user based on their likelihood of being useful for further exploration.

Our goal is to determine if artificially imagined or synthesized images can be beneficial to interactive visual search. We present a novel approach for using artificially imagined images in relevance feedback. Since the search engine constructs the synthetic images itself, any feedback given by the user on these images allows it to obtain a better understanding of what the user is looking for than it would from feedback on database images alone. We evaluated and compared our image synthesis approach with a normal Rocchio-based system on a well-known texture database with real users.

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100+ recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.

In the current state-of-the-art in multimedia content analysis (MCA), the fundamental techniques are typically derived from core pattern recognition and computer vision algorithms. It is well known that completely automatic pattern recognition and computer vision approaches have not been successful in being robust and domain independent so we should not expect more from MCA algorithms. The exception to this would naturally be methods which are human-interactive or not automatic. In this paper, we describe some of the recent work we have done in multimedia content analysis across multiple domains where the fundamental technique is founded in interactive search. Our novel algorithm integrates our previous work from wavelet based salient points and genetic algorithms and shows that the main contribution and improvement is from the user feedback provided by the interactive search.

Recognizing human facial expression and emotion by computer is an interesting and challenging problem. We propose a method for recognizing emotions through facial expressions displayed in video sequences. We introduce a tree-augmented naive-Bayes (TAN) classifier that learns the dependencies between facial features; we also provide an algorithm for finding the best TAN structure. Our person-dependent and person-independent experiments show that using this TAN structure provides significantly better results than using simpler NB-classifiers.

The translated function with any integer multiple of the sampling period is completely represented in the wavelet space by one of the overcomplete discrete wavelet transform (ODWT) members. This theoretical result leads to a new motion estimation and motion compensation scheme working in the wavelet transform domain. Our experiments, performed on real image sequences, show high quality and low bit rate performances. Moreover, by performing the motion estimation in the wavelet space a major reduction of the computational cost is achieved.

In image retrieval, global features related to color or texture are commonly used to describe the image content. The use of interest points in contentbased image retrieval allows an image index to represent local properties of images. In this paper, we present a wavelet-based salient point extraction algorithm and we show that extracting the color and texture information in the locations given by these points provides significantly improved results in terms of retrieval accuracy, computational complexity, and storage space of feature vectors as compared to the global feature approaches. 1

The use of salient points in content-based retrieval allows an image index to represent local properties of the image. Classic corner detectors can also be used for this purpose but they have drawbacks when are applied to various natural images mainly because visual features do not need to be corners and corners may gather in small regions. In this paper, we present a salient point detector using wavelet transform and we compare it with two corner detectors using two criteria: repeatability rate and information content. We determine which detector gives the best results and show that it satisfies the criteria well.

Imagine a professional Web site designer who constantly has to come up with innovative looks for the client's homepages? How does he find the new content? The major search engines such as Hotbot (http://www.hotbot.com) allow us to find text on the Web, but typically have few or no capabilities for finding visual media. In this article we discuss methods for finding visual media on the WWW. The emphasis is on iconic queries which are essentially drag-and-drop visual concepts or simple semantics. We describe our method for finding static feature sets and then describe a novel method called active feature sets, which chooses a feature set based on the context in the image.

Many visual matching algorithms can be described in terms of the features and the inter-feature distance or metric. The most commonly used metric is the sum of squared differences (SSD), which is valid from a maximum likelihood perspective when the real noise distribution is Gaussian. Based on real noise distributions measured from international test sets, we have found experimentally that the Gaussian noise distribution assumption is often invalid. This implies that other metrics, which have distributions closer to the real noise distribution, should be used. In this paper we considered two different visual matching applications: content-based retrieval in image databases and stereo matching. Towards broadening the results, we also implemented several sophisticated algorithms from the research literature. In each algorithm we compared the efficacy of the SSD metric, the SAD (sum of the absolute differences) metric, the Cauchy metric, and the Kullback relative information. Furthermore, in the case where sufficient training data is available, we discussed and experimentally tested a new metric based directly on the real noise distribution, which we denoted the maximum likelihood metric.

In the research literature, maximum likelihood principles were applied to stereo matching by altering the stereo pair so that the difference would have a Gaussian distribution. In this paper we present a novel method of applying maximum likelihood to stereo matching. In our approach, we measure the real noise distribution from a training set, and then construct a new metric which we denote the maximum likelihood metric for comparing the stereo pair. The maximum likelihood metric is optimal in the sense that it maximizes the probability of similarity. In our experiments and discussion, we compared the maximum likelihood metric to other promising algorithms from the research literature using international stereo data sets. Furthermore, we showed that the algorithms from the research literature could be improved by using the maximum likelihood metric instead of the sum of squared differences.

Texture are one of the basic features in visual searching and computational vision. In the literature most of the attention has been focussed on the texture features with minimal consideration of the noise models. In this paper, we investigate the problem of texture classification from a maximum likelihood perspective. We take into account the texture model, the noise distribution, and the inter-dependence of the texture features. Our investigation shows that the real noise distribution is closer to an exponential than a Gaussian distribution, and that the L/sub 1/ metric has a better retrieval rate than L/sub 2/. We also propose the Cauchy metric as an alternative for both the L/sub 1/ and L/sub 2/ metrics. Furthermore, we provide a direct method for deriving an optimal distortion measure from the real noise distribution, which experimentally provides consistently improved results over the other metrics. We conclude with results and discussions on an international texture database.

In many computer vision algorithms, a metric or similarity measure is used to determine the distance between two features. The Euclidean or SSD (sum of the squared differences) metric is prevalent and justified from a maximum likelihood perspective when the additive noise distribution is Gaussian. Based on real noise distributions measured from international test sets, we have found that the Gaussian noise distribution assumption is often invalid. This implies that other metrics, which have distributions closer to the real noise distribution, should be used. In this paper, we consider three different applications: content-based retrieval in image databases, stereo matching, and motion tracking. In each of them, we experiment with different modeling functions for the noise distribution and compute the accuracy of the methods using the corresponding distance measures. In our experiments, we compared the SSD metric, the SAD (sum of the absolute differences) metric, the Cauchy metric, and the Kullback relative information. For several algorithms from the research literature which used the SSD or SAD, we showed that greater accuracy could be obtained by using the Cauchy metric instead.

The coarse-to-fine search strategy is extensively used in current reported research. However, it has the same problem as any hill climbing algorithm, most importantly, it often finds local instead of global minima. Drawing upon the artificial intelligence literature, we applied an optimal graph search, namely A*, to the problem. Using real stereo and video test sets, we compared the A* method to both template and hill climbing. Our results show that A* has greater accuracy than the ubiquitous coarse-to-fine hill climbing pyramidal search algorithm in both stereo matching and motion tracking.

Article Learning visual concepts Share on Authors: Jean M. Buijs Leiden Institute for Advanced Computer Science, Leiden University, Postbus 9512, 2300 RA Leiden, The Netherlands Leiden Institute for Advanced Computer Science, Leiden University, Postbus 9512, 2300 RA Leiden, The NetherlandsView Profile , Michael S. Lew Leiden Institute for Advanced Computer Science, Leiden University, Postbus 9512, 2300 RA Leiden, The Netherlands Leiden Institute for Advanced Computer Science, Leiden University, Postbus 9512, 2300 RA Leiden, The NetherlandsView Profile Authors Info & Claims MULTIMEDIA '99: Proceedings of the seventh ACM international conference on Multimedia (Part 2)October 1999 Pages 5–7https://doi.org/10.1145/319878.319880Online:01 October 1999Publication History 4citation377DownloadsMetricsTotal Citations4Total Downloads377Last 12 Months1Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In content based image retrieval, color indexing is one of the most prevalent retrieval methods. In literature, most of the attention has been focussed on the color model with little or no consideration of the noise models. In this paper we investigate the problem of color indexing from a maximum likelihood perspective. We take into account the color model, the noise distribution, and the quantization of the color features. Furthermore, from the real noise distribution we derive a distortion measure, which consistently provides improved accuracy. Our investigation concludes with results on a real stock photography database, consisting of 11,000 color images.

If an image should be retrieved by its subregions from a large image database, an immense number of possible queries will appear.Therefore, the index which encodes the spatial information of an image, should make only few assumptions about possible queries.In addition, this index has to consider different scales of objects in the image.In this paper, we propose a novel approach using a hierarchical index encoding image regions, gained by a fixed partition.The suggested index uses color features and is easy to implement.The index is tested on a database with more than 12,000 images.

This paper describes an algorithm for deriving minimal 3D triangulated models from dense 3D range data. It also addresses an important problem in 3D real time rendering: how to minimize the number of triangles in the model while retaining the quality of the rendered images. In a typical rendering engine, the hidden surfaces are removed at run time. In our approach. We assert that hidden surfaces should be removed if possible prior to run time. Consequently, there would be fewer triangles to render at run time. We accomplish this goal by integrating into the vertex removal criterion a weight associated with each vertex, which reflects the probability of the vertex being visible. This weight is used to bias the vertex removal decision towards removing hidden surfaces before frequently visible surfaces. We tested our algorithm on face models derived from a laser range finder, and a complex business office model. Our results include a function of decimation rate with respect to distortion.

Euclidean metric is frequently used in computer vision, mostly ad-hoc without any justification. However we have found that other metrics like a double exponential or Cauchy metric provide better results, in accordance with the maximum likelihood approach. In this paper we experiment with different modeling functions for similarity noise and compute the accuracy of different methods using these modeling functions in two kinds of applications: content-based image retrieval from a large database and stereo matching. We provide a way to determine the modeling distribution which fits best the similarity noise distribution according to the ground truth. In the optimum case, when one has chosen the best modeling distribution, its corresponding metric will give the best ranking results for the ground truth provided.

Automatic human face detection in digital images with a complex environment is still an unsolved problem in computer vision and pattern recognition. It has several uses, such as human face recognition, content based image retrieval and model based video coding. We present an automatic human face detection system where several methods are tested and compared. The underlying principle of the system is to compare subimages of the image pyramid, spanned by the input image, with a set of 'nose-eye' templates. However this comparison is not done on the entire set of subimages of the image pyramid, but on a small subset, which is defined by the 'local maxima method'. False positives are found by using a set of non-face templates. The system is tested on two databases, each include over 1000 images.

This paper describes information theoretic methods for the determination of the optimal subset of pixels for the problem of face detection in complex backgrounds. A view-based method is described which has limitations due to misalignments. This motivates the modular feature based method which minimizes the misalignment problem. Empirical comparisons between the view-based, modular, and sum of squared difference methods are made using four databases from three universities.

Digital storage of large photo collections opens the way to computer-aided queries based on visual rather than thematic search patterns. The objective of our queries in this research was the 19th-century mass-produced studio portrait or carte-de-visite, whose front and back sides provide a testbed for gray level and binary images classes. We established a ground truth for detecting highly similar images (former copies) in different classes of B/W images. The results will serve as a reference benchmark for yet to be developed visual search methods. The similarity measure used for locating near-copies was the average distance in pixel intensity for shifted image pairs with normalized position, orientation, resolution and lighting. To measure the performance of possible hierarchical comparison and ranking protocols, we scanned in test sets of known copies and near-copies together with over a thousand similar format pictures. The results show that projections are highly effective and efficient in indexing raw image data: reduced dimensionality, low noise, conservation of pattern characteristics, separable x- and y-translation (best shift) and suitable for hierarchical indexing. A query by example WWW demo program with precalculated ranking result files was developed for visual inspection and evaluation of similar image location methods.

Face detection in complex environments is an unsolved problem which has fundamental importance to face recognition, model based video coding, content based image retrieval, and human computer interaction. In this paper we model the face detection problem using information theory, and formulate information based measures for detecting faces by maximizing the feature class separation. The underlying principle is that search through an image can be viewed as a reduction of uncertainty in the classification of the image. The face detection algorithm is empirically compared using multiple test sets, which include four face databases from three universities.

This paper addresses the novel idea of whether it is possible to perform spatial matching of compressed images. What are the fundamental problems and limitations of limiting the knowledge to that of the compressed image? We present an algorithm which matches transform block encoded images, without performing the decoding stage. The theoretical 2D image matching model is developed and applied to the discrete Karhunen-Loeve transform and the discrete cosine transform. In addition, we discuss the tradeoffs in using our algorithm for matching compressed images. As a validation step, we apply our algorithm to a video sequence and discuss the results for 2D matching and 3D reconstruction.>

We present a novel stereo matching algorithm which integrates learning, feature selection, and surface reconstruction. First, a new instance based learning (IBL) algorithm is used to generate an approximation to the optimal feature set for matching. In addition, the importance of two separate kinds of knowledge, image dependent knowledge and image independent knowledge, is discussed. Second, we develop an adaptive method for refining the feature set. This adaptive method analyzes the feature error to locate areas of the image that would lead to false matches. Then these areas are used to guide the search through feature space towards maximizing the class separation distance between the correct match and the false matches. Third, we introduce a self-diagnostic method for determining when apriori knowledge is necessary for finding the correct match. If the a priori knowledge is necessary then we use a surface reconstruction model to discriminate between match possibilities. Our algorithm is comprehensively tested against fixed feature set algorithms and against a traditional pyramid algorithm. Finally, we present and discuss extensive empirical results of our algorithm based on a large set of real images.>

Most work in pyramidal or hierarchical stereo matching has primarily used the direct coarse-to-fine or hill climbing search method. However, there are two significant problems in using the hill climbing method. First, the match at the initial scale level can be wrong. Second, the binary decision at any of the finer scale levels may be wrong. The authors present a method which is essentially a best-first or cost minimization search method and which alleviates both of these problems. Results of application to real world image pairs are presented as well as a direct comparison in terms of percentage of correct matches to the traditional hill climbing method.>