Deploying a pre-trained model on the target data can be challenging due to the domain shift between the target and the source data. Test Time Adaptation (TTA) has recently gained increasing attention, aiming to adapt the domain shift during inference without needing annotated data. However, most previous TTA methods still require the classifier trained on the source data for adaptation, entropy minimization is applied based on the classifier’s output. Therefore, previous TTA methods are not applicable for metric learning tasks, e.g., identify verification systems. In this paper, we propose IDentity-based TTA (ID-TTA), which can adapt pre-trained models for metric learning tasks without the need for the classifier. We implement two types of adaptations: (1) self-supervised, which generates different variations of any given sample and minimizes the distance of the embeddings of the variations from the embedding of the original sample. (2) pair-wise, that compares any two given samples and aims to minimize the distance between the embeddings of the two samples if the pre-trained model predicts them to be from the same identity. We evaluate ID-TTA against multiple face recognition datasets by adding varying degrees of domain shift in the image. We show that ID-TTA is notably effective in adapting the baseline models to the target domain.

A visual Scene Graph (VSG) is a visually-grounded graph over objects in an image, where the edges represent the relations between the objects. Visual and semantic information is extracted from image objects and processed by the relation inference module. One main challenge in VSG generation is that the training data is highly imbalanced, only a few relations dominate the categories of predicates. Existing solutions mainly rely on alternative loss functions or data-level approaches like sampling. This paper addresses the long-tail problem in VSG from a new perspective, we enrich the semantic information using a tailored embedding based on Common Sense Knowledge Graphs (CSKG). We first study the relatedness and explore the gap between the visual domain graphs and CSKG, highlighting their differences. To bridge the gap, we investigate the effect of different knowledge graph embedding (KGE) techniques and sources of CSKGs. Our study shows understanding the gap between the two tasks, i.e., KGE and VSG, and the nature of the data is crucial to designed a specific embedding for the long-tail problem. Our proposed solution can be efficiently created in an off-line manner and used as a replacement to other existing embeddings. And our method can be combined with other de-biasinz techniques to further improve the efficacy.

In this paper, we challenge the common assumption that collapsing the spatial dimensions of a 3D (spatial-channel) tensor in a convolutional neural network (CNN) into a vector via global pooling removes all spatial information. Specifically, we demonstrate that positional information is encoded based on the ordering of the channel dimensions, while semantic information is largely not. Following this demonstration, we show the real world impact of these findings by applying them to two applications. First, we propose a simple yet effective data augmentation strategy and loss function which improves the translation invariance of a CNN’s output. Second, we propose a method to efficiently determine which channels in the latent representation are responsible for (i) encoding overall position information or (ii) region-specific positions. We first show that semantic segmentation has a significant reliance on the overall position channels to make predictions. We then show for the first time that it is possible to perform a ‘region-specific’ attack, and degrade a network’s performance in a particular part of the input. We believe our findings and demonstrated applications will benefit research areas concerned with understanding the characteristics of CNNs. Code is available at: https://github.com/islamamirul/PermuteNet.

In this paper, we propose a new metric to address the long-standing problem of center bias in saliency evaluation. We first show that distribution-based metrics cannot measure saliency performance across datasets due to ambiguity in the choice of standard deviation, especially for Convolutional Neural Networks. Therefore, our proposed metric is AUC-based because ROC curves are relatively robust to the standard deviation problem. However, this requires sufficient unique values in the saliency prediction to compute AUC scores. Secondly, we propose a global smoothing function for the problem of few value degrees in predicted saliency output. Compared with random noise, our smoothing function can create unique values without losing the existing relative saliency relationship. Finally, we show our proposed AUC-based metric can generate a more directional negative set for evaluation, denoted as Farthest-Neighbor AUC (FN-AUC). Our experiments show FN-AUC can measure spatial biases, central and peripheral, more effectively than S-AUC without penalizing the fixation locations.

In this paper, we proposed a novel method for No-Reference Image Quality Assessment (NR-IQA) by combining deep Convolutional Neural Network (CNN) with saliency map. We first investigate the effect of depth of CNNs for NR-IQA by comparing our proposed ten-layer Deep CNN (DCNN) for NR-IQA with the state-of-the-art CNN architecture proposed by Kang et al. ( 2014 ). Our results show that the DCNN architecture can deliver a higher accuracy on the LIVE dataset. To mimic human vision, we introduce saliency maps combining with CNN to propose a Saliency-based DCNN (SDCNN) framework for NR-IQA. We compute a saliency map for each image and both the map and the image are split into small patches. Each image patch is assigned with a patch importance value based on its saliency patch. A set of Salient Image Patches (SIPs) are selected according to their saliency and we only apply the model on those SIPs to predict the quality score for the whole image. Our experimental results show that the SDCNN framework is superior to other state-of-the-art approaches on the widely used LIVE dataset. The TID2008 and the CISQ image quality datasets are utilised to report cross-dataset results. The results indicate that our proposed SDCNN can generalise well on other datasets.

In this paper we propose a No-Reference Image Quality Assessment (NR-IQA) method on High Dynamic Range (HDR) images by combining deep Convolutional Neural Networks (CNNs) with saliency maps. The proposed method utilises the power of deep CNN architectures to extract quality features which can be applied cross HDR and Standard Dynamic Range (SDR) domains. To introduce human visual system to CNNs, a saliency map algorithm is used to select a subset of salient image patches to evaluate on. Our CNN-based method delivers a state-of-the-art performance in HDR NR-IQA experiment, competitive with full reference IQA methods.