Vision Transformers have proven their mettle across a variety of computer vision problems, however, their reliance on pretraining with very large-scale datasets such as JFT-300M is also no secret, as large amounts of data is very conductive to effective feature learning. In this paper, however, we propose a novel Vision Transformer architecture PAEViT that aims to effectively learn generalized features from small-sized datasets, usually having only a few hundred images per class at most. By refining the attention scores based on patch-level interactions and modulating it to enhance the trained model’s ability to focus more on task relevant patches, PAEViT is able to significantly improve upon the performance of the regular ViT model, as well as other variants of the same, when the model is being trained solely on small-sized datasets. In data-constrained situations and visual recognition tasks that do not conform well with the existing large-scale datasets, PAEViT can be used to create effective and scalable solutions with all the features and attention scores being based only on relevant data. Our code is publicly available at https://github.com/AkashVermaIN/PAEViT.

Unsupervised image retrieval aims to learn the important visual characteristics without any given level to retrieve the similar images for a given query image. The Convolutional Neural Network (CNN)-based approaches have been extensively exploited with self-supervised contrastive learning for image hashing. However, the existing approaches suffer due to lack of effective utilization of global features by CNNs and biased-ness created by false negative pairs in the contrastive learning. In this paper, we propose a TransClippedCLR model by encoding the global context of an image using Transformer having local context through patch based processing, by generating the hash codes through product quantization and by avoiding the potential false negative pairs through clipped contrastive learning. The proposed model is tested with superior performance for unsupervised image retrieval on benchmark datasets, including CIFAR10, NUS-Wide and Flickr25K, as compared to the recent state-of-the-art deep models. The results using the proposed clipped contrastive learning are greatly improved on all datasets as compared to same backbone network with vanilla contrastive learning.

The stochastic gradient descent (SGD) optimizers are generally used to train the convolutional neural networks (CNNs). In recent years, several adaptive momentum based SGD optimizers have been introduced, such as Adam, diffGrad, Radam and AdaBelief. However, the existing SGD optimizers do not exploit the gradient norm of past iterations and lead to poor convergence and performance. In this paper, we propose a novel AdaNorm based SGD optimizers by correcting the norm of gradient in each iteration based on the adaptive training history of gradient norm. By doing so, the proposed optimizers are able to maintain high and representive gradient throughout the training and solves the low and atypical gradient problems. The proposed concept is generic and can be used with any existing SGD optimizer. We show the efficacy of the proposed AdaNorm with four state-of-the-art optimizers, including Adam, diffGrad, Radam and AdaBelief. We depict the performance improvement due to the proposed optimizers using three CNN models, including VGG16, ResNet18 and ResNet50, on three benchmark object recognition datasets, including CIFAR10, CIFAR100 and TinyImageNet.

Recently, Transformer has emerged as a new architecture in deep learning by utilizing self-attention without convolution. Transformer is also extended to Vision Transformer (ViT) for the visual recognition with a promising performance on ImageNet. In this paper, we propose a Vision Transformer Hashing (VTS) for image retrieval. We utilize the pre-trained ViT on ImageNet as the backbone network and add the hashing head. The proposed VTS model is fine tuned for hashing under six different image retrieval frameworks with their objective functions. We perform the extensive experiments on CIFAR10, ImageNet, NUS-Wide, and COCO datasets. The proposed VTS based image retrieval outperforms the recent state-of-the-art hashing techniques with a significant margin. We also find the proposed VTS model as the backbone network is better than the existing networks, such as AlexNet and ResNet. The code is released at https://github.com/shivram1987/VisionTransformerHashing.

The content based image retrieval aims to find the similar images from a large scale dataset against a query image. Generally, the similarity between the representative features of the query image and dataset images is used to rank the images for retrieval. In early days, various hand designed feature descriptors have been investigated based on the visual cues such as color, texture, shape, etc. that represent the images. However, the deep learning has emerged as a dominating alternative of hand-designed feature engineering from a decade. It learns the features automatically from the data. This paper presents a comprehensive survey of deep learning based developments in the past decade for content based image retrieval. The categorization of existing state-of-the-art methods from different perspectives is also performed for greater understanding of the progress. The taxonomy used in this survey covers different supervision, different networks, different descriptor type and different retrieval type. A performance analysis is also performed using the state-of-the-art methods. The insights are also presented for the benefit of the researchers to observe the progress and to make the best choices. The survey presented in this paper will help in further research progress in image retrieval using deep learning.

Recently, convolutional neural networks (CNNs) have exhibited commendable performance for hyperspectral image (HSI) classification. Generally, an important number of samples are needed for each class to properly train CNNs. However, existing HSI data sets suffer from a significant class imbalance problem, where many classes do not have enough samples to characterize the spectral information. The performance of existing CNN models is biased toward the majority classes, which possess more samples for the training. This article addresses this issue of imbalanced data in HSI classification. In particular, a new3D-HyperGAMOmodel is proposed, which uses generative adversarial minority oversampling. The proposed3D-HyperGAMOautomatically generates more samples for minority classes at training time, using the existing samples of that class. The samples are generated in the form of a 3-D hyperspectral patch. A different classifier from the generator and the discriminator is used in the3D-HyperGAMOmodel, which is trained using both original and generated samples to determine the classes of newly generated samples to which they actually belong. The generated data are combined classwise with the original training data set to learn the network parameters of the class. Finally, the trained 3-D classifier network validates the performance of the model using the test set. Four benchmark HSI data sets, namely, Indian Pines (IP), Kennedy Space Center (KSC), University of Pavia (UP), and Botswana (BW), have been considered in our experiments. The proposed model shows outstanding data generation ability during the training, which significantly improves the classification performance over the considered data sets. The source code is available publicly athttps://github.com/mhaut/3D-HyperGAMO.

Deep learning‐based approaches have become very prominent in recent years due to its outstanding performance as compared to the hand‐extracted feature‐based methods. Convolutional neural network (CNN) is a type of deep learning architecture to deal with the image/video data. Residual network and squeeze and excitation network (SENet) are among recent developments in CNN for image classification. However, the performance of SENet depends on the squeeze operation done by global pooling, which sometimes may lead to poor performance. In this study, the authors propose a bilinear fusion mechanism over different types of squeeze operation such as global pooling and max pooling. The excitation operation is performed using the fused output of squeeze operation. They used to model the proposed fused SENet with the residual unit and name it as FuSENet . Here the classification experiments are performed over benchmark hyperspectral image datasets. The experimental results confirm the superiority of the proposed FuSENet method with respect to the state‐of‐the‐art methods. The source code of the complete system is made publicly available at https://github.com/swalpa/FuSENet .

Hyperspectral image (HSI) classification is widely used for the analysis of remotely sensed images. Hyperspectral imagery includes varying bands of images. Convolutional neural network (CNN) is one of the most frequently used deep learning-based methods for visual data processing. The use of CNN for HSI classification is also visible in recent works. These approaches are mostly based on 2-D CNN. On the other hand, the HSI classification performance is highly dependent on both spatial and spectral information. Very few methods have used the 3-D-CNN because of increased computational complexity. This letter proposes a hybrid spectral CNN (HybridSN) for HSI classification. In general, the HybridSN is a spectral-spatial 3-D-CNN followed by spatial 2-D-CNN. The 3-D-CNN facilitates the joint spatial-spectral feature representation from a stack of spectral bands. The 2-D-CNN on top of the 3-D-CNN further learns more abstract-level spatial representation. Moreover, the use of hybrid CNNs reduces the complexity of the model compared to the use of 3-D-CNN alone. To test the performance of this hybrid approach, very rigorous HSI classification experiments are performed over Indian Pines, University of Pavia, and Salinas Scene remote sensing data sets. The results are compared with the state-of-the-art hand-crafted as well as end-to-end deep learning-based methods. A very satisfactory performance is obtained using the proposed HybridSN for HSI classification. The source code can be found at https://github.com/gokriznastic/HybridSN.

Stochastic gradient descent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic SGD is to change by equal-sized steps for all parameters, irrespective of the gradient behavior. Hence, an efficient way of deep network optimization is to have adaptive step sizes for each parameter. Recently, several attempts have been made to improve gradient descent methods such as AdaGrad, AdaDelta, RMSProp, and adaptive moment estimation (Adam). These methods rely on the square roots of exponential moving averages of squared past gradients. Thus, these methods do not take advantage of local change in gradients. In this article, a novel optimizer is proposed based on the difference between the present and the immediate past gradient (i.e., diffGrad). In the proposed diffGrad optimization technique, the step size is adjusted for each parameter in such a way that it should have a larger step size for faster gradient changing parameters and a lower step size for lower gradient changing parameters. The convergence analysis is done using the regret bound approach of the online learning framework. In this article, thorough analysis is made over three synthetic complex nonconvex functions. The image categorization experiments are also conducted over the CIFAR10 and CIFAR100 data sets to observe the performance of diffGrad with respect to the state-of-the-art optimizers such as SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam. The residual unit (ResNet)-based convolutional neural network (CNN) architecture is used in the experiments. The experiments show that diffGrad outperforms other optimizers. Also, we show that diffGrad performs uniformly well for training CNN using different activation functions. The source code is made publicly available at https://github.com/shivram1987/diffGrad.

In the current era, biometric based access control is becoming more popular due to its simplicity and ease to use by the users. It reduces the manual work of identity recognition and facilitates the automatic processing. The face is one of the most important biometric visual information that can be easily captured without user cooperation in an uncontrolled environment. Precise detection of spoofed faces should be on the high priority to make face based identity recognition and access control robust against possible attacks. The recently evolved Con-volutional Neural Network (CNN) based deep learning technique has proven as one of the excellent method to deal with the visual information very effectively. The CNN learns the hierarchical features at intermediate layers automatically from the data. Several CNN based methods such as Inception and ResNet have shown outstanding performance for image classification problem. This paper does a performance evaluation of CNNs for face anti-spoofing. The Inception and ResNet CNN architectures are used in this study. The results are computed over benchmark MSU Mobile Face Spoofing Database. The experiments are done by considering the different aspects such as the depth of the model, random weight initialization vs weight transfer, fine tuning vs training from scratch and different learning rate. The favorable results are obtained using these CNN architectures for face anti-spoofing in different settings.

Facial expression recognition in videos is an active area of research in computer vision. However, fake facial expressions are difficult to be recognized even by humans. On the other hand, facial micro-expressions generally represent the actual emotion of a person, as it is a spontaneous reaction expressed through human face. Despite of a few attempts made for recognizing micro-expressions, still the problem is far from being a solved problem, which is depicted by the poor rate of accuracy shown by the state-of-the-art methods. A few CNN based approaches are found in the literature to recognize micro-facial expressions from still images. Whereas, a spontaneous microexpression video contains multiple frames that have to be processed together to encode both spatial and temporal information. This paper proposes two 3D-CNN methods: MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatiotemporal information in CNN framework. The MicroExpSTCNN considers the full spatial information, whereas the MicroExpFuseNet is based on the 3D-CNN feature fusion of the eyes and mouth regions. The experiments are performed over CAS(ME)2and SMIC microb expression databases. The proposed MicroExpSTCNN model outperforms the state-of-the-art methods.

Efficient and precise classification of histological cell nuclei is of utmost importance due to its potential applications in the field of medical image analysis. It would facilitate the medical practitioners to better understand and explore various factors for cancer treatment. The classification of histological cell nuclei is a challenging task due to the cellular heterogeneity. This paper proposes an efficient Convolutional Neural Network (CNN) based architecture for classification of histological routine colon cancer nuclei named as RCCNet. The main objective of this network is to keep the CNN model as simple as possible. The proposed RCCNet model consists of 1, 512, 868 learnable parameters which are significantly less compared to the popular CNN models such as AlexNet, CIFAR-VGG, GoogLeNet, and WRN. The experiments are conducted over publicly available routine colon cancer histological dataset “CRCHistoPhenotypes”. The results of the proposed RCCNet model are compared with five state-of-the-art CNN models in terms of the accuracy, weighted average F1 score and training time. The proposed method has achieved a classification accuracy of 80.61% and 0.7887 weighted average F1 score. The proposed RCCNet is more efficient and generalized in terms of the training time and data over-fitting, respectively.

Face recognition in images is an active area of interest among the computer vision researchers. However, recognizing human face in an unconstrained environment, is a relatively less-explored area of research. Multiple face recognition in unconstrained environment is a challenging task, due to the variation of view-point, scale, pose, illumination and expression of the face images. Partial occlusion of faces makes the recognition task even more challenging. The contribution of this paper is two-folds: introducing a challenging multi-face dataset (i.e., IIITS_MFace Dataset) for face recognition in unconstrained environment and evaluating the performance of state-of-the-art hand-designed and deep learning based face descriptors on the dataset. The proposed IIITS_MFace dataset contains faces with challenges like pose variation, occlusion, mask, spectacle, expressions, change of illumination, etc. We experiment with several state-of-the-art face descriptors, including recent deep learning based face descriptors like VGGFace, and compare with the existing benchmark face datasets. Results of the experiments clearly show that the difficulty level of the proposed dataset is much higher compared to the benchmark datasets.

Local binary pattern (LBP) is widely adopted for efficient image feature description and simplicity. To describe the color images, it is required to combine the LBPs from each channel of the image. The traditional way of binary combination is to simply concatenate the LBPs from each channel, but it increases the dimensionality of the pattern. In order to cope with this problem, this paper proposes a novel method for image description with multichannel decoded LBPs. We introduce adder- and decoder-based two schemas for the combination of the LBPs from more than one channel. Image retrieval experiments are performed to observe the effectiveness of the proposed approaches and compared with the existing ways of multichannel techniques. The experiments are performed over 12 benchmark natural scene and color texture image databases, such as Corel-1k, MIT-VisTex, USPTex, Colored Brodatz, and so on. It is observed that the introduced multichannel adder- and decoder-based LBPs significantly improve the retrieval performance over each database and outperform the other multichannel-based approaches in terms of the average retrieval precision and average retrieval rate.

A novel image feature descriptor based on the local bit-plane decoded pattern (LBDP) is introduced for indexing and retrieval of biomedical images in this paper. A local bit-plane transformation scheme is proposed to compute the local bit-plane transformed values for each image pixel from the bit-plane binary contents of its each neighboring pixels. The introduced LBDP is generated by finding a binary pattern using the difference of center pixel's intensity value with the local bit-plane transformed values. The efficacy of the LBDP is tested under biomedical image retrieval using average retrieval precision and average retrieval rate. Three benchmark databases Emphysema-CT, NEMA-CT, and Open Access Series of Imaging Studies magnetic resonance imaging are used for the evaluation and comparison of the proposed approach with recent state-of-art methods. The experimental results confirm the discriminative ability and the efficiency of the proposed LBDP for biomedical image indexing and retrieval and prove the outperformance of existing biomedical image retrieval approaches.

Content‐based image retrieval (CBIR) is demanding accurate with efficient retrieval approaches to index and retrieve the most similar images from the huge image databases. This study introduces a novel local neighbourhood‐based robust colour occurrence descriptor (LCOD) to encode the colour information present in the local structure of the image. The colour information is processed in two steps: first, the number of colours is reduced into a less number of shades by quantising the red–green–blue colour space; second, the reduced colour shade information of the local neighbourhood is used to compute the descriptor. A local colour occurrence binary pattern is generated for each pixel of the image by representing each reduced colour shade occurrence in its local neighbourhood using a binary pattern. The descriptor is constructed by summing the local colour occurrence binary patterns of all the pixels in the image. LCOD is tested over the natural and colour texture databases for CBIR and experimental results suggest that LCOD outperforms other state‐of‐the‐art descriptors. The performance of the proposed descriptor is promising in the case of illumination difference, rotation and scaling also and it can be effectively used for accurate image retrieval under various image transformations.

The medical image retrieval plays an important role in medical diagnosis where a physician can retrieve most similar images from template images against a query image of a particular patient. In this letter, a new and efficient image features descriptor based on the local diagonal extrema pattern (LDEP) is proposed for CT image retrieval. The proposed approach finds the values and indexes of the local diagonal extremas to exploit the relationship among the diagonal neighbors of any center pixel of the image using first-order local diagonal derivatives. The intensity values of the local diagonal extremas are compared with the intensity value of the center pixel to utilize the relationship of central pixel with its neighbors. Finally, the descriptor is formed on the basis of the indexes and comparison of center pixel and local diagonal extremas. The consideration of only diagonal neighbors greatly reduces the dimension of the feature vector which speeds up the image retrieval task and solves the “Curse of dimensionality” problem also. The LDEP is tested for CT image retrieval over Emphysema-CT and NEMA-CT databases and compared with the existing approaches. The superiority in terms of performance and efficiency in terms of speedup of the proposed method are confirmed by the experiments.

A new image feature description based on the local wavelet pattern (LWP) is proposed in this paper to characterize the medical computer tomography (CT) images for content-based CT image retrieval. In the proposed work, the LWP is derived for each pixel of the CT image by utilizing the relationship of center pixel with the local neighboring information. In contrast to the local binary pattern that only considers the relationship between a center pixel and its neighboring pixels, the presented approach first utilizes the relationship among the neighboring pixels using local wavelet decomposition, and finally considers its relationship with the center pixel. A center pixel transformation scheme is introduced to match the range of center value with the range of local wavelet decomposed values. Moreover, the introduced local wavelet decomposition scheme is centrally symmetric and suitable for CT images. The novelty of this paper lies in the following two ways: 1) encoding local neighboring information with local wavelet decomposition and 2) computing LWP using local wavelet decomposed values and transformed center pixel values. We tested the performance of our method over three CT image databases in terms of the precision and recall. We also compared the proposed LWP descriptor with the other state-of-the-art local image descriptors, and the experimental results suggest that the proposed method outperforms other methods for CT image retrieval.

The region descriptors using local intensity ordering patterns have become more popular recent years for image matching due to its enhanced discriminative ability. However, the dimension of these descriptors increases rapidly with the slight increase in the number of local neighbors under consideration and becomes unreasonable for image matching due to time constraint. In this paper, we reduce the dimension of the descriptor and matching time significantly while keeping up the comparable performance by considering the number of neighboring sample points in an interleaved manner. The proposed interleaved order based local descriptor (IOLD) considers the local neighbors of a pixel as a set of interleaved neighbors and constructs the descriptor over each set separately and finally combines them to produce a single pattern. We extract the local ordering pattern to cope up with the illumination effect in an inherent rotation invariant manner. The novelty lies with using multiple neighboring sets in an interleaved fashion. We also explored the local intensity order pattern in a multisupport-region scenario. Results are compared over three challenging and widely adopted image matching data sets with other prominent descriptors under various image transformations. Results based on experiments suggest that the proposed IOLD descriptor outperforms in terms of both improved matching performance and reduced matching time. We also found that the amount of improvement is significant under complex illumination difference while showing more robustness toward noise.