Real-world datasets often exhibit temporal dynamics characterized by evolving data distributions. Disregarding this phenomenon, commonly referred to as concept drift, can significantly diminish a model's predictive accuracy. Furthermore, the presence of hyperparameters in online models exacerbates this issue. These parameters are typically fixed and cannot be dynamically adjusted by the user in response to the evolving data distribution. Gaussian Process (GP) models offer powerful non-parametric regression capabilities with uncertainty quantification, making them ideal for modeling complex data relationships in an online setting. However, conventional online GP methods face several critical limitations, including a lack of drift-awareness, reliance on fixed hyperparameters, vulnerability to data snooping, absence of a principled decay mechanism, and memory inefficiencies. In response, we propose DAO-GP (Drift-Aware Online Gaussian Process), a novel, fully adaptive, hyperparameter-free, decayed, and sparse non-linear regression model. DAO-GP features a built-in drift detection and adaptation mechanism that dynamically adjusts model behavior based on the severity of drift. Extensive empirical evaluations confirm DAO-GP's robustness across stationary conditions, diverse drift types (abrupt, incremental, gradual), and varied data characteristics. Analyses demonstrate its dynamic adaptation, efficient in-memory and decay-based management, and evolving inducing points. Compared with state-of-the-art parametric and non-parametric models, DAO-GP consistently achieves superior or competitive performance, establishing it as a drift-resilient solution for online non-linear regression.

Ocular biometrics in the visible spectrum have emerged as a prominent modality due to their high accuracy, resistance to spoofing, and non-invasive nature. However, morphing attacks, synthetic biometric traits created by blending features from multiple individuals, threaten biometric system integrity. While extensively studied for nearinfrared iris and face biometrics, morphing in visiblespectrum ocular data remains underexplored. Simulating such attacks demands advanced generation models that handle uncontrolled conditions while preserving detailed ocular features like iris boundaries and periocular textures. To address this gap, we introduce DOOMGAN, that encompasses landmark-driven encoding of visible ocular anatomy, attention-guided generation for realistic morph synthesis, and dynamic weighting of multi-faceted losses for optimized convergence. DOOMGAN achieves over 20% higher attack success rates than baseline methods under stringent thresholds, along with 20% better elliptical iris structure generation and 30% improved gaze consistency. We also release the first comprehensive ocular morphing dataset to support further research in this domain. The code is available at Vcbsl/DOOMGAN.

With the rise of voice synthesis technology threatening digital media integrity, audio watermarking has become a crucial defense for content authentication. However, current solutions often lack robustness to effects like high-pass filtering and temporal modifications and suffer from poor watermark localization. We introduce WaveVerify, a watermarking system that addresses these challenges using a Feature-wise Linear Modulation (FiLM)-based generator for resilient multiband watermark embedding and a Mixture-of-Experts detector for accurate extraction and localization. Our unified training framework enhances robustness by applying multiple distortions per backpropagation step via a dynamic effect scheduler. Evaluations across multiple datasets show WaveVerify outperforms SOTA models like AudioSeal and WavMark, achieving zero Bit Error Rate (BER) under common distortions and MIoU scores of 0.98+ under severe temporal modifications. The parallel FiLM-based generator also reduces training time by ∼80% compared to sequential embedding approaches. Code and pretrained models are available at: https://github.com/vcbsl/WaveVerify.

In unconstrained environments, it encounters a number of challenges when considering handcrafted features for face recognition, including changes in pose, illumination and facial expression and plastic surgery variations, look-alike faces and selfie images. As majority of the published works on local descriptors are based on the relationship between the centre pixel and neighbourhood pixels at different radial widths, by including the relationship between different neighbourhood pixels of a target pixel with it can facilitate the decoupling of the discriminatory features further. To exploit such relationships and to mitigate the challenges, the proposed study reports a novel local descriptor called Local Directional Octa Pattern (LDOP) that makes use of derivative operation and three well-known metrics, namely, histogram intersection, K-nearest neighbour and Euclidean distance for face identification. Rigorous experiments have been conducted on six benchmark face databases, namely, Extended Yale Face B, Labeled Faces in the Wild, Plastic Surgery, Look-alike, UMDAA-02-FD and ORL, confirming the superiority of the proposed local descriptor over state-of-the-art descriptors with accuracies of 97.37%, 49.09%, 75.47%, 31.2% and 93.31% at Rank–10 and 100% at Rank-5 determined on the databases, respectively.

EMG-based hand gesture recognition uses electromyographic (EMG) signals to interpret and classify hand movements by analyzing electrical activity generated by muscle contractions. It has wide applications in prosthesis control, rehabilitation training, and human-computer interaction. Using electrodes placed on the skin, the EMG sensor captures muscle signals, which are processed and filtered to reduce noise. Numerous feature extraction and machine learning algorithms have been proposed to extract and classify muscle signals to distinguish between various hand gestures.This paper aims to benchmark the performance of EMG-based hand gesture recognition using novel feature extraction methods, namely, fused time-domain descriptors, temporal-spatial descriptors, and wavelet transform-based features, combined with the state-of-the-art machine and deep learning models. Experimental investigations on the Grabmyo dataset demonstrate that the 1D Dilated CNN performed the best with an accuracy of 97% using fused time-domain descriptors such as power spectral moments, sparsity, irregularity factor and waveform length ratio. Similarly, on the FORS-EMG dataset, random forest performed the best with an accuracy of 94.95% using temporal-spatial descriptors (which include time domain features along with additional features such as coefficient of variation (COV), and Teager-Kaiser energy operator (TKEO)).

In the digital age, the emergence of deepfakes and synthetic media presents a significant threat to societal and political integrity. Deepfakes based on multi-modal manipulation, such as audio-visual, are more realistic and pose a greater threat. Current multi-modal deepfake detectors are often based on the attention-based fusion of heterogeneous data streams from multiple modalities. However, the heterogeneous nature of the data (such as audio and visual signals) creates a distributional modality gap and poses a significant challenge in effective fusion and hence multi-modal deepfake detection. In this paper, we propose a novel multi-modal attention framework based on recurrent neural networks (RNNs) that leverages contextual information for audio-visual deepfake detection. The proposed approach applies attention to multi-modal multi-sequence representations and learns the contributing features among them for deepfake detection and localization. Thorough experimental validations on audio-visual deepfake datasets, namely FakeAVCeleb, AV-Deepfake1M, TVIL, and LAV-DF datasets, demonstrate the efficacy of our approach. Cross-comparison with the published studies demonstrates superior performance of our approach with an improved accuracy and precision by 3.47% and 2.05% in deepfake detection and localization, respectively. Thus, obtaining state-of-the-art performance. To facilitate reproducibility, the code and the datasets information is available at https://github.com/vcbsl/audiovisual-deepfake/.

Published research highlights the presence of demographic bias in automated facial attribute classification. The proposed bias mitigation techniques are mostly based on supervised learning, which requires a large amount of labeled training data for generalizability and scalability. However, labeled data is limited, requires laborious annotation, poses privacy risks, and can perpetuate human bias. In contrast, self-supervised learning (SSL) capitalizes on freely available unlabeled data, rendering trained models more scalable and generalizable. However, these label-free SSL models may also introduce biases by sampling false negative pairs, especially at low-data regimes (< 200K images) under low compute settings. Further, SSL-based models may suffer from performance degradation due to a lack of quality assurance of the unlabeled data sourced from the web. This paper proposes a fully self-supervised pipeline for demographically fair facial attribute classifiers. Leveraging completely unlabeled data pseudolabeled via pre-trained encoders, diverse data curation techniques, and meta-learning-based weighted contrastive learning, our method significantly outperforms existing SSL approaches proposed for downstream image classification tasks. Extensive evaluations on the FairFace and CelebA datasets demonstrate the efficacy of our pipeline in obtaining fair performance over existing baselines. Thus, setting a new benchmark for SSL in the fairness of facial attribute classification. To facilitate reproducibility, the code, and the curated dataset information is available at https://github.com/nsf-ocular-bias/ssl-ijcb.

With the advances in generative adversarial networks (GAN), facial manipulations called DeepFakes have caused major security risks and raised severe societal concerns. However, the popular DeepFake passive detection is an ex-post forensics countermeasure and fails in blocking the disinformation spread in advance. Alternatively, precautions such as adding perturbations to the real data for unnatural distorted DeepFake output easily spotted by the human eyes are introduced as proactive defenses. Recent studies suggest that these existing proactive defenses can be easily bypassed by employing simple image transformation and reconstruction techniques when applied to the perturbed real data and the distorted output, respectively. The aim of this article is to propose a novel proactive DeepFake detection technique using GAN-based visible watermarking. To this front, we propose a reconstructive regularization added to the GAN’s loss function that embeds a unique watermark to the assigned location of the generated fake image. Thorough experiments on multiple datasets confirm the viability of the proposed approach as a proactive defense mechanism against DeepFakes from the perspective of detection by human eyes. Thus, our proposed watermark-based GANs prevent the abuse of the pretrained GANs and smartphone apps, available via online repositories, for DeepFake creation for malicious purposes. Further, the watermarked DeepFakes can also be detected by the SOTA DeepFake detectors. This is critical for applications where automatic DeepFake detectors are used for mass audits due to the huge cost associated with human observers examining a large amount of data manually.

Due to an alarming trend related to obesity affecting 93.3 million adults in the United States alone, body mass index (BMI) and body weight have drawn significant interest in various health monitoring applications. Consequently, several studies have proposed self-diagnostic facial image-based BMI prediction methods for healthy weight monitoring. These methods have mostly used convolutional neural network (CNN) based regression baselines, such as VGG19, ResNet50, and EfficientNetB0, for BMI prediction from facial images. However, the high computational requirement of these heavy-weight CNN models limits their deployment to resource-constrained mobile devices, thus deterring weight monitoring using smartphones. This paper aims to develop a lightweight facial patch-based ensemble (PatchBMI-Net) for BMI prediction to facilitate the deployment and weight monitoring using smartphones. Extensive experiments on BMI-annotated facial image datasets suggest that our proposed PatchBMI-Net model can obtain Mean Absolute Error (MAE) in the range [3.58, 6.51] with a size of about 3.3 million parameters. On cross-comparison with heavyweight models, such as ResNet-50 and Xception, trained for BMI prediction from facial images, our proposed PatchBMI-Net obtains equivalent MAE along with the model size reduction of about 5.4× and the average inference time reduction of about 3× when deployed on Apple-14 smartphone. Thus, demonstrating performance efficiency as well as low latency for on-device deployment and weight monitoring using smartphone applications.

Deepfakes are synthetic media generated using deep generative algorithms and have posed a severe societal and political threat. Apart from facial manipulation and synthetic voice, recently, a novel kind of deepfakes has emerged with either audio or visual modalities manipulated. In this regard, a new generation of multimodal audio-visual deepfake detectors is being investigated to collectively focus on audio and visual data for multimodal manipulation detection. Existing multimodal (audio-visual) deepfake detectors are often based on the fusion of the audio and visual streams from the video. Existing studies suggest that these multimodal detectors often obtain equivalent performances with unimodal audio and visual deepfake detectors. We conjecture that the heterogeneous nature of the audio and visual signals creates distributional modality gaps and poses a sig-nificant challenge to effective fusion and efficient performance. In this paper, we tackle the problem at the representation level to aid the fusion of audio and visual streams for multimodal deepfake detection. Specifically, we propose the joint use of modality (audio and visual) invariant and specific representations. This ensures that the common patterns and patterns specific to each modality representing pristine or fake content are preserved and fused for multimodal deepfake manipulation detection. Our experimental results on FakeAVCeleb and KoDF audio-visual deepfake datasets suggest the enhanced accuracy of our proposed method over SOTA unimodal and multimodal audio-visual deepfake detectors by 17.8% and 18.4%, respectively. Thus, obtaining state-of-the-art performance.

Facial forgery by deepfakes has caused major se-curity risks and raised severe societal concerns. As a counter-measure, a number of deepfake detection methods have been proposed. Most of them model deepfake detection as a binary classification problem using a backbone convolutional neural network (CNN) architecture pretrained for the task. These CNN-based methods have demonstrated very high efficacy in deepfake detection with the Area under the Curve (AUC) as high as 0.99. However, the performance of these methods degrades signifi-cantly when evaluated across datasets and deepfake manipulation techniques. This draws our attention towards learning more subtle, local, and discriminative features for deepfake detection. In this paper, we formulate deepfake detection as a fine-grained classification problem and propose a new fine-grained solution to it. Specifically, our method is based on learning subtle and generalizable features by effectively suppressing background noise and learning discriminative features at various scales for deepfake detection. Through extensive experimental validation, we demonstrate the superiority of our method over the published research in cross-dataset and cross-manipulation generalization of deepfake detectors for the majority of the experimental scenarios.

With computer vision reaching an inflection point in the past decade, face recognition technology has become pervasive in policing, intelligence gathering, and consumer applications. Recently, face recognition technology has been deployed on body-worn cameras to keep officers safe, enabling situational awareness and providing evidence for trial. However, limited academic research has been conducted on this topic using traditional techniques on datasets with small sample size. This paper aims to bridge the gap in the state-of-the-art face recognition using body-worn cameras (BWC). To this aim, the contribution of this work is two-fold: (1) collection of a dataset called BWCFace consisting of a total of 178K facial images of 132 subjects captured using the body-worn camera in in-door and daylight conditions, and (2) open-set evaluation of the latest deep-learning-based Convolutional Neural Network (CNN) architectures combined with five different loss functions for face identification, on the collected dataset. Experimental results on our BWCFace dataset suggest a maximum of 33.89% Rank-1 accuracy obtained when facial features are extracted using SENet-50 trained on a large scale VGGFace2 facial image dataset. However, performance improved up to a maximum of 99.00% Rank-1 accuracy when pretrained CNN models are fine-tuned on a subset of identities in our BWCFace dataset. Equivalent performances were obtained across body-worn camera sensor models used in existing face datasets. The collected BWCFace dataset and the pretrained/ fine-tuned algorithms are publicly available to promote further research and development in this area. A downloadable link of this dataset and the algorithms is available by contacting the authors.

Automated gender classification has important applications in many domains, such as demographic research, law enforcement, online advertising, as well as human-computer interaction. Recent research has questioned the fairness of this technology across gender and race. Specifically, the majority of the studies raised the concern of higher error rates of the face-based gender classification system for darker-skinned people like African-American and for women. However, to date, the majority of existing studies were limited to African-American and Caucasian only. The aim of this paper is to investigate the differential performance of the gender classification algorithms across gender-race groups. To this aim, we investigate the impact of (a) architectural differences in the deep learning algorithms and (b) training set imbalance, as a potential source of bias causing differential performance across gender and race. Experimental investigations are conducted on two latest large-scale publicly available facial attribute datasets, namely, UTKFace and FairFace. The experimental results suggested that the algorithms with architectural differences varied in performance with consistency towards specific gender-race groups. For instance, for all the algorithms used, Black females (Black race in general) always obtained the least accuracy rates. Middle Eastern males and Latino females obtained higher accuracy rates most of the time. Training set imbalance further widens the gap in the unequal accuracy rates across all gender-race groups. Further investigations using facial landmarks suggested that facial morphological differences due to the bone structure influenced by genetic and environmental factors could be the cause of the least performance of Black females and Black race, in general.

Self-diagnostic image-based methods for healthy weight monitoring is gaining increased interest following the alarming trend of obesity. Only a handful of academic studies exist that investigate AI-based methods for Body Mass Index (BMI) inference from facial images as a solution to healthy weight monitoring and management. To promote further research and development in this area, we evaluate and compare the performance of five different deep-learning based Convolutional Neural Network (CNN) architectures i.e., VGG19, ResNet50, DenseNet, MobileNet, and lightCNN for BMI inference from facial images. Experimental results on the three publicly available BMI annotated facial image datasets assembled from social media, namely, VisualBMI, VIP-Attributes, and Bollywood datasets, suggest the efficacy of the deep learning methods in BMI inference from face images with minimum Mean Absolute Error (MAE) of 1.04 obtained using ResNet50.

The skills of focusing and paying attention are critical to student learning. According to Piontkowski et al. [40], "Educators often talk about attention as a general mental state in which the mind focuses on some special feature of the environment. As such, attention is considered essential for learning. It is hard to believe that the student who disregards instruction will benefit from it. Thus, the teacher needs reliable signs of the student's state of attention."It is challenging, however, for instructors to ascertain signs of student attention in large classrooms with many students. Additional challenges arise in online classrooms, which often limit instructors to watching students' body language in video feeds, where they cannot see, for example, distractions in the students' environment. Biometrics and machine learning approaches can help instructors evaluate their students' level of attentiveness in both physical and online classrooms and introduce appropriate interventions to improve learning outcomes. Although the field of automated attention tracking research is steadily amassing new publications, no survey works have charted the progress of research or encouraged new research. We have filled this gap with this survey of salient works that use biometrics and machine learning to track attention. Specifically, we focus on discussions and analyses of methods that use eye gazing, facial movements and expressions, behavioral biometrics such as body movements, analyses of brainwave signals and psychological states, and multimodal biometrics. We conclude with a discussion of promising future research directions with focus on multimodal biometric techniques.

Recent reported advances in smartphone based ocular biometric recognition in visible spectrum demonstrated the efficacy of deep-learning schemes. In this paper, we evaluate convolutional neural networks (CNNs) pretrained for large scale object recognition, namely VGG-16, VGG-19, InceptionNet and ResNet, and fine-tuned for ocular recognition using RGB images captured by smartphones. Fine-tuning pretrained CNN models is advantageous in case of insufficient training data, and the partial training is faster compared to custom CNN trained from scratch. Experiments on VISOB dataset yielded TPR of up to 100% at FPR of 10-4using VGG-16 model fine-tuned for ocular recognition.

Automated age classification has drawn significant interest in numerous applications such as marketing, forensics, human-computer interaction, and age simulation. A number of studies have demonstrated that age can be automatically deduced from face images. However, few studies have explored the possibility of computational estimation of age information from other modalities such as fingerprint or ocular region. The main challenge in age classification is that age progression is person-specific which depends on many factors such as genetics, health conditions, life style, and stress level. In this paper, we investigate age classification from ocular images acquired using smart-phones. Age information, though not unique to the individual, can be combined along with ocular recognition system to improve authentication accuracy or invariance to the ageing effect. To this end, we propose a convolutional neural network (CNN) architecture for the task. We evaluate our proposed CNN model on the ocular crops of the recent large-scale Adience benchmark for gender and age classification captured using smart-phones. The obtained results establish a baseline for deep learning approaches for age classification from ocular images captured by smart-phones.

With the unprecedented mobile technology revolution, a number of ocular biometric based personal recognition schemes have been proposed for mobile use cases. The aim of this competition is to evaluate and compare the performance of mobile ocular biometric recognition schemes in visible light on a large scale database (VISOB Dataset ICIP2016 Challenge Version) using standard evaluation methods. Four different teams from universities across the world participated in this competition, submitting five algorithms altogether. The submitted algorithms applied different texture analysis in a learning or a non-learning based framework for ocular recognition. The best results were obtained by a team from Norwegian Biometrics Laboratory (NTNU, Norway), achieving an Equal Error Rate of 0.06% over a quarantined test set.

A fingerprint spoof detector is a pattern classifier that is used to distinguish a live finger from a fake (spoof) one in the context of an automated fingerprint recognition system. Most spoof detectors are learning-based and rely on a set of training images. Consequently, the performance of any such spoof detector significantly degrades when encountering spoofs fabricated using novel materials not found in the training set. In real-world applications, the problem of fingerprint spoof detection must be treated as an open set recognition problem where incomplete knowledge of the fabrication materials used to generate spoofs is present at training time, and novel materials may be encountered during system deployment. To mitigate the security risk posed by novel spoofs, this paper introduces: 1) the use of the Weibull-calibrated SVM (W-SVM), which is relatively robust for open set recognition, as a novel-material detector and a spoof detector and 2) a scheme for the automatic adaptation of the W-SVM-based spoof detector to new spoof materials that leverages interoperability across classifiers. Experiments conducted on new partitions of the LivDet 2011 database designed for open set evaluation suggest: 1) a 97% increase in the error rate of the existing spoof detectors when tested using new spoof materials and 2) up to 44% improvement in spoof detection performance across spoof materials when the proposed adaptive approach is used.

A fingerprint liveness detector is a pattern classifier that is used to distinguish a live finger from a fake (spoof) one in the context of an automated fingerprint recognition system. Most liveness detectors are learning-based and rely on a set of training images. Consequently, the performance of a liveness detector significantly degrades upon encountering spoofs fabricated using new materials not used during the training stage. To mitigate the security risk posed by new spoofs, it is necessary to automatically adapt the liveness detector to new spoofing materials. The aim of this work is to design a scheme for automatic adaptation of a liveness detector to novel spoof materials encountered during the operational phase. To facilitate this, a novel-material detector is used to flag input images that are deemed to be made of a new spoofing material. Such flagged images are then used to retrain the liveness detector. Experiments conducted on the LivDet 2011 database suggest (i) a 62% increase in the error rate of existing liveness detectors when tested using new spoof materials, and (ii) upto 46% improvement in liveness detection performance across spoof materials when the proposed adaptive approach is used.

Fingerprint liveness detection algorithms have been used to disambiguate live fingerprint samples from spoof (fake) fingerprints fabricated using materials such as latex, gelatine, etc. Most liveness detection algorithms are learning-based and dependent on the fabrication materials used to generate spoofs during the training stage. Consequently, the performance of a liveness detector is significantly degraded upon encountering fabrication materials that were not used during the training stage. The aim of this work is to design a simple pre-processing scheme that can improve the interoperability of liveness detectors across different fabrication materials - including those not observed during the training stage. Such a generalization ability is desirable in liveness detectors. Experiments on the LivDet 2011 fake fingerprint dataset suggest that (a) different fabrication materials when used in the training stage impart different degrees of generalization ability to the liveness detector and (b) the proposed pre-processing scheme improves generalization performance by upto 44%.

This paper presents a novel face recognition technique with graph topology drawn on scale invariant feature transform (SIFT) features and is compared with all the available well known techniques on SIFT features, and elastic bunch graph matching (EBGM) technique drawn on gabor wavelet feature. IITK face database is used for evaluation purpose. Test results show that the proposed graph matching technique will be an appropriate one for face recognition. Finally, the test results have been compared with the results found on BANCA face database following MC protocol only.