Relation Extraction (RE) is a multi-task process that is a crucial part of all information extraction pipelines. With the introduction of the generative language models, Large Language Models (LLMs) have showcased significant performance boosts for complex natural language processing and understanding tasks. Recent research in RE has also started incorporating these advanced machines in their pipelines. However, the full extent of the LLM’s potential for extracting relations remains unknown. Consequently, this study aims to conduct the first feasibility analysis to explore the viability of LLMs for RE by investigating their robustness to various complex RE scenarios stemming from data-specific characteristics. By conducting an exhaustive analysis of five state-of-the-art LLMs backed by more than 2100 experiments, this study posits that LLMs are not robust enough to tackle complex data characteristics for RE, and additional research efforts focusing on investigating their behaviors at extracting relationships are needed. The source code for the evaluation pipeline can be found at https://aaig.ece.ufl.edu/projects/relation-extraction .

Multimodal learning systems often struggle in non-stationary environments due to concept drift, where changing data distributions can degrade performance. Modality-specific drifts and the lack of mechanisms for continuous, stable adaptation compound this challenge. This paper introduces LS-OGD, a novel adaptive control framework for robust multimodal learning in the presence of concept drift. LS-OGD uses an online controller that dynamically adjusts the model's learning rate and the fusion weights between different data modalities in response to detected drift and evolving prediction errors. We prove that under bounded drift conditions, the LS-OGD system's prediction error is uniformly ultimately bounded and converges to zero if the drift ceases. Additionally, we demonstrate that the adaptive fusion strategy effectively isolates and mitigates the impact of severe modality-specific drift, thereby ensuring system resilience and fault tolerance. These theoretical guarantees establish a principled foundation for developing reliable and continuously adapting multimodal learning systems.

Two common attacks against biometric systems are direct (or physical) access and indirect (or logical) access. While most detection techniques focus on the former, often called presentation attacks, that occur at pre-sensor level, the attack surface for indirect access, that takes place post-sensor, is larger. In this paper, an indirect attack in the realm of faces is explored that utilizes a unique soft-biometric feature called ‘Kinship Cues’. Unlike gender and ethnicity, kinship is less explored but powerful; we find that its knowledge can significantly increase the chances of an attacker getting access to a system. Due to lack of kin data in other domains, our attack is only performed against facial biometric systems. Nevertheless, the results underscore the impact of kinship cues and their need to be investigated in other domains such as fingerprint and iris. This kinship artifact boosts the convergence speed of state-of-the-art iterative adaptive Bayesian hill climbing attacks. Further, it is exploited to generate a dictionary of input images, commonly called wolf images, in a novel kinship-based non-iterative indirect attack that we call Kin-Wolf. A classical image fusion technique (morphing) and a deep learning based kinship framework utilizing pre-trained StyleGAN2 are investigated to generate the wolf images. The trade-off between kinship cues and randomization is also studied and a 6× average improvement in attack accuracy is achieved for Kin-Wolf over random probes.

Advances in vision and deep learning have revolutionized feature extraction for face recognition and verification systems, yet, performing kinship verification from such features is still challenging. Ongoing research attempts to imitate a human by identifying features for kinship verification. In this paper, we propose KinfaceNet, a deep learning based kinship feature extractor, capable of extracting kinship features from a single input image independently without requiring its kin pair image. The base model of the method is adopted from face recognition domain which is then transfer learned in the domain of kinship by learning a distance mapping from face images to a compact Euclidean space where distances directly correspond to a measure of kinship similarity. Thus, unlike most of the works in deep learning based kinship domain, the extracted features can be used in many other applications such as image generation and family based clustering, etc. Training is performed by rearranging the data into classes of kin pairs and using a state-of-the-art triplet mining algorithm to address the unbalanced kinship data problem which causes overfitting. Also, one of the major advantages of our framework is that training can be performed on any face feature extractor model pre-trained on large face recognition data, thereby reducing training time by a considerable amount. Comparable verification accuracy is obtained from simple MLP network at only 20th epoch with KinfaceNet features extracted from the Family-In-the-Wild dataset, the largest in the wild kinship dataset available, as well as KinfaceW-I and II datasets.

Outsourced PCB fabrication necessitates increased hardware assurance capabilities. Several assurance techniques based on AOI have been proposed that leverage PCB images acquired using digital cameras. We review state-of-the-art AOI techniques and observe a strong, rapid trend toward ML solutions. These require significant amounts of labeled ground truth data, which is lacking in the publicly available PCB data space. We contribute the FPIC dataset to address this need. Additionally, we outline new hardware security methodologies enabled by our dataset.

For successful printed circuit board (PCB) reverse engineering (RE), the resulting device must retain the physical characteristics and functionality of the original. Although the applications of RE are within the discretion of the executing party, establishing a viable, non-destructive framework for analysis is vital for any stakeholder in the PCB industry. A widely regarded approach in PCB RE uses non-destructive x-ray computed tomography (CT) to produce three-dimensional volumes with several slices of data corresponding to multi-layered PCBs. However, the noise sources specific to x-ray CT and variability from designers hampers the thorough acquisition of features necessary for successful RE. This article investigates a deep learning approach as a successor to the current state-of-the-art for detecting vias on PCB x-ray CT images; vias are a key building block of PCB designs. During RE, vias offer an understanding of the PCB’s electrical connections across multiple layers. Our method is an improvement on an earlier iteration which demonstrates significantly faster runtime with quality of results comparable to or better than the current state-of-the-art, unsupervised iterative Hough-based method. Compared with the Hough-based method, the current framework is 4.5 times faster for the discrete image scenario and 24.1 times faster for the volumetric image scenario. The upgrades to the prior deep learning version include faster feature-based detection for real-world usability and adaptive post-processing methods to improve the quality of detections.

Artificial intelligence (AI) and machine learning (ML) techniques have been increasingly used in several fields to improve performance and the level of automation. In recent years, this use has exponentially increased due to the advancement of high-performance computing and the ever increasing size of data. One of such fields is that of hardware design—specifically the design of digital and analog integrated circuits, where AI/ ML techniques have been extensively used to address ever-increasing design complexity, aggressive time to market, and the growing number of ubiquitous interconnected devices. However, the security concerns and issues related to integrated circuit design have been highly overlooked. In this article, we summarize the state-of-the-art in AI/ML for circuit design/optimization, security and engineering challenges, research in security-aware computer-aided design/electronic design automation, and future research directions and needs for using AI/ML for security-aware circuit design.

Hardware assurance is a key process in ensuring the integrity, security and functionality of a hardware device. Its heavy reliance on images, especially on Scanning Electron Microscopy images, makes it an excellent candidate for the vision community. The goal of this paper is to provide a pathway for inter-community collaboration by introducing the existing challenges for hardware assurance on integrated circuits in the context of computer vision and support further development using a large-scale dataset with 800,000 images. A detailed benchmark of existing vision approaches in hardware assurance on the dataset is also included for quantitative insights into the problem.

Hardware Trojans are malicious modifications in integrated circuits (ICs) with an intent to breach security and compromise the reliability of an electronic system. This article proposes a framework using self-testing, advanced imaging, and image processing with machine learning to detect hardware Trojans inserted by untrusted foundries. It includes on-chip test structures with negligible power, delay, and silicon area overheads. The core step of the framework is on-chip golden circuit design, which can provide authentic samples for image-based Trojan detection through self-testing. This core step enables a golden-chip-free Trojan detection that does not rely on an existing image data set from Trojan-free chip or image synthesizing. We have conducted an in-depth analysis of detection steps and discussed possible attacks with countermeasures to strengthen this framework. The performance evaluation on a 28-nm FPGA and a 90-nm IC validates its high accuracy and reliability for practical applications.

In the context of hardware trust and assurance, reverse engineering has been often considered as an illegal action. Generally speaking, reverse engineering aims to retrieve information from a product, i.e., integrated circuits (ICs) and printed circuit boards (PCBs) in hardware security-related scenarios, in the hope of understanding the functionality of the device and determining its constituent components. Hence, it can raise serious issues concerning Intellectual Property (IP) infringement, the (in)effectiveness of security-related measures, and even new opportunities for injecting hardware Trojans. Ironically, reverse engineering can enable IP owners to verify and validate the design. Nevertheless, this cannot be achieved without overcoming numerous obstacles that limit successful outcomes of the reverse engineering process. This article surveys these challenges from two complementary perspectives: image processing and machine learning. These two fields of study form a firm basis for the enhancement of efficiency and accuracy of reverse engineering processes for both PCBs and ICs. In summary, therefore, this article presents a roadmap indicating clearly the actions to be taken to fulfill hardware trust and assurance objectives.

A Hierarchical Bloom Filter (HBF) -based biometric framework was recently proposed to provide compact storage, noise tolerance, and fast query processing for resource-constrained environments, e.g., Internet of things (IoT). While security and privacy were also touted as features of the HBF, it was not thoroughly evaluated. Compared to the classical BFs, the HBF uses a threshold parameter to make robust authentication decisions when the HBF encounters noise in the biometric input which one would think might lead to security issues. In this paper, the attack vectors that could compromise the HBF security by increasing the false positive authentication of non-members and by leaking soft information about enrolled members are explored. With quantitative analyses, HBF-based biometric system security under these well-defined attack vectors is evaluated and it is concluded that the framework is more difficult to attack than the classical Bloom Filter. Further, experimental results show that soft biometric information is also kept private.

The future of smart environments is likely to involve both passive and active interactions on the part of users. Depending on what sensors are available in the space, users may make use of multimodal interaction modalities such as hand gestures or voice commands. There is a shortage of robust yet controlled multimodal interaction datasets for smart environment applications. One application domain of interest based on current state-of-the-art is authentication for sensitive or private tasks, such as banking and email. We present a novel, large multimodal dataset for authentication interactions in both gesture and voice, collected from 106 volunteers who each performed 10 examples of each of a set of hand gesture and spoken voice commands chosen from prior literature (10,600 gesture samples and 13,780 voice samples). We present the data collection method, raw data and common features extracted, and a case study illustrating how this dataset could be useful to researchers. Our goal is to provide a benchmark dataset for testing future multimodal authentication solutions, enabling comparison across approaches.

Over the past two decades, globalized outsourcing in the semiconductor supply chain has lowered manufacturing costs and shortened the time-to-market for original equipment manufacturers (OEMs). However, such outsourcing has rendered the printed circuit boards (PCBs) vulnerable to malicious activities and alterations on a global scale. In this article, we take an in-depth look into one such attack, called the “Big Hack,” that was recently reported by Bloomberg Buisnessweek. The article provides background on the Big Hack from three perspectives: an attacker, a security investigator, and the societal impacts. This study provides details on vulnerabilities in the modern PCB supply chain, the possible attacks, and the existing and emerging countermeasures. The necessity for novel visual inspection techniques for PCB assurance is emphasized throughout the article. Further, a review of various imaging modalities, image analysis algorithms, and open research challenges are provided for automated visual inspection.

Authorship attribution typically uses all information representing both content and style whereas attribution based only on stylistic aspects may be robust in cross-domain settings. This paper analyzes different linguistic aspects that may help represent style. Specifically, we study the role of syntax and lexical words (nouns, verbs, adjectives and adverbs) in representing style. We use a purely syntactic language model to study the significance of sentence structures in both single-domain and cross-domain attribution, i.e. cross-topic and cross-genre attribution. We show that syntax may be helpful for cross-genre attribution while cross-topic attribution and single-domain may benefit from additional lexical information. Further, pure syntactic models may not be effective by themselves and need to be used in combination with other robust models. To study the role of word choice, we perform attribution by masking all words or specific topic words corresponding to nouns, verbs, adjectives and adverbs. Using a single-domain dataset, IMDB1M reviews, we demonstrate the heavy influence of common nouns and proper nouns in attribution, thereby highlighting topic interference. Using cross-domain Guardian10 dataset, we show that some common nouns, verbs, adjectives and adverbs may help with stylometric attribution as demonstrated by masking topic words corresponding to these parts-of-speech. As expected, it was observed that proper nouns are heavily influenced by content and cross-domain attribution will benefit from completely masking them.

Chipless Radio Frequency Identification (RFID) tags that do not include an integrated circuit (IC) in the transponder are more appropriate for supply-chain management of low-cost commodities and have been gaining extensive attention due to their relatively lower price. However, existing chipless RFID tags consume considerable tag area and manufacturing time/cost because of complex fabrication process (e.g., requiring removing or shorting some resonators on the tag substrate to encode data). Worse still, their identifiers (IDs) are deterministic, clonable, and small in terms of bitwidth. To address these shortcomings and help preserve the cold chain for commodities (e.g., vaccines, pharmaceuticals, etc.) sensitive to temperature, we develop a novel unclonable environmentally sensitive chipless RFID (UCR) tag that intrinsically generates a unique ID from both manufacturing variations and ambient temperature variation. A UCR tag consists of two parts: (i) a certain number of concentric ring slot resonators integrated on a certain laminate (e.g., TACONIC TLX-0), whose resonance frequencies rely on geometric parameters of slot resonators and dielectric constant of substrate material that are sensitive to manufacturing variations, and (ii) a stand-alone circular ring slot resonator integrated on a particular substrate (e.g., grease) that will be melted at a high temperature, whose resonance frequency relies on geometric parameters of slot resonator, dielectric constant of substrate material, and ambient temperature. UCR tags have the capability to track commodities and their temperatures in the supply chain. The area of UCR tag is comparable to regular quick response (QR) code. Experimental results based on UCR tag prototypes have verified their uniqueness and reliability.

Electrocardiogram (ECG) has long been regarded as a biometric modality which is impractical to copy, clone, or spoof. However, it was recently shown that an ECG signal can be replayed from arbitrary waveform generators, computer sound cards, or off-the-shelf audio players. In this paper, we develop a novel presentation attack where a short template of the victim's ECG is captured by an attacker and used to map the attacker's ECG into the victim's, which can then be provided to the sensor using one of the above sources. Our approach involves exploiting ECG models, characterizing the differences between ECG signals, and developing mapping functions that transform any ECG into one that closely matches an authentic user's ECG. Our proposed approach, which can operate online or on-the-fly, is compared with a more ideal offline scenario where the attacker has more time and resources. In our experiments, the offline approach achieves average success rates of 97.43% and 94.17% for non-fiducial and fiducial based ECG authentication. In the online scenario, the performance is de-graded by 5.65% for non-fiducial based authentication, but is nearly unaffected for fiducial authentication.

While mobile devices are no longer a new technology, using the data generated from the use of these devices for security purposes has just recently been explored. Current methods, such as passwords, are quickly becoming antiquated, lacking the robustness, accuracy, and convenience desired to serve as reliable security measures. Since, researchers have resorted to alternative techniques, such as measurements obtained from keyboard interactions and movement, and behavioral interactions, such as application usage. However, practical implementations require further evaluation of circumvention. Thus, this work thoroughly analyzes various threats against mobile devices which use mobile device usage data as behavioral biometrics for authentication. Experimental results indicate that an outsider with a certain level of knowledge regarding the behavior of the device's owner poses a great security threat. Possible countermeasures to prevent such attacks are also provided.

Because passwords and personal identification numbers are easily forgotten, stolen, or reused on multiple accounts, the current norm for mobile device security is quickly becoming inefficient and inconvenient. Thus, manufacturers have worked to make physiological biometrics accessible to mobile device owners as improved security measures. While behavioral biometrics has yet to receive commercial attention, researchers have continued to consider these approaches as well. However, studies of interactive data are limited, and efforts which are aimed at improving the performance of such techniques remain relevant. Thus, this paper provides a performance analysis of application, Bluetooth, and Wi-Fi data collected from 189 subjects on a mobile device for user verification. Results indicate that user authentication can be achieved with up to 91% accuracy, demonstrating the effectiveness of associative classification as a feature extraction technique.

Periocular biometrics is the recognition of individuals based on the appearance of the region around the eye. Periocular recognition may be useful in applications where it is difficult to obtain a clear picture of an iris for iris biometrics, or a complete picture of a face for face biometrics. Previous periocular research has used either visible-light (VL) or near-infrared (NIR) light images, but no prior research has directly compared the two illuminations using images with similar resolution. We conducted an experiment in which volunteers were asked to compare pairs of periocular images. Some pairs showed images taken in VL, and some showed images taken in NIR light. Participants labeled each pair as belonging to the same person or to different people. Untrained participants with limited viewing times correctly classified VL image pairs with 88% accuracy, and NIR image pairs with 79% accuracy. For comparison, we presented pairs of iris images from the same subjects. In addition, we investigated differences between performance on light and dark eyes and relative helpfulness of various features in the periocular region under different illuminations. We calculated performance of three computer algorithms on the periocular images. Performance for humans and computers was similar.

In this paper, we compare genetic and evolutionary feature selection (GEFeS) and weighting (GEFeW) using a number of biometric datasets. GEFeS and GEFeW have been implemented as instances of Steady-State Genetic and Estimation of Distribution Algorithms. Our results show that GEFeS and GEFeW dramatically improve recognition accuracy as well as reduce the number of features needed for facial recognition. Our results also show that the Estimation of Distribution Algorithm implementation of GEFeW has the best overall performance.

In this paper, we use Genetic and Evolutionary Computation (GEC) to optimize the weights assigned to the biometric modalities of a multi-biometric system for score-level fusion. Our results show that GEC-based multi-biometric fusion provides a significant improvement in the recognition accuracy over evenly fused biometric modalities, increasing the accuracy from 90.77% to 95.24%.

In this paper, we compare the performance of a Steady-State Genetic Algorithm (SSGA) and an Estimation of Distribution Algorithm (EDA) for multi-biometric feature selection and weighting. Our results show that when fusing face and periocular modalities, SSGA-based feature weighting (GEFeWSSGA) produces higher average recognition accuracies, while EDA-based feature selection (GEFeSEDA) performs better at reducing the number of features needed for recognition.

A wide variety of applications in forensic, government, and commercial fields require reliable personal identification. However, the recognition performance is severely affected when encountering non-ideal images caused by motion blur, poor contrast, various expressions, or illumination artifacts. In this paper, we investigated the use of shape-based eyebrow features under non-ideal imaging conditions for biometric recognition and gender classification. We extracted various shape-based features from the eyebrow images and compared three different classification methods: Minimum Distance Classifier (MD), Linear Discriminant Analysis Classifier (LDA) and Support Vector Machine Classifier (SVM). The methods were tested on images from two publicly available facial image databases: The Multiple Biometric Grand Challenge (MBGC) database and the Face Recognition Grand Challenge (FRGC) database. Obtained recognition rates of 90% using the MBGC database and 75% using the FRGC database as well as gender classification recognition rates of 96% and 97% for each database respectively, suggests the shape-based eyebrow features maybe be used for biometric recognition and soft biometric classification.

One of the most important modules of any bio-metric system is the feature extraction module. Given a sample it is important for the feature extraction method to extract a rich set of features that can be used for identity recognition. This form of feature extraction has been referred to as Type I feature extraction and for some biometric systems it is used exclusively. However, a second form of feature extraction does exist and is concerned with optimizing/minimizing the original feature set given by a Type I feature extraction method. This second form of feature extraction has been referred to as Type II feature extraction (also known as feature selection). In this paper, we compare two GEC-based Type II feature extraction methods as applied to periocular-based recognition, an exciting new area of research within the Biometric research community that to date has used Type I feature extraction exclusively. Our results show that GEC-based Type II feature extraction is effective in optimizing recognition accuracy as well as minimizing the overall feature set size.

Given an image from a biometric sensor, it is important for the feature extraction module to extract an original set of features that can be used for identity recognition. This form of feature extraction has been referred to as Type I feature extraction. For some biometric systems, Type I feature extraction is used exclusively. However, a second form of feature extraction does exist and is concerned with optimizing/minimizing the original feature set given by a Type I feature extraction method. This second form of feature extraction has been referred to as Type II feature extraction (feature selection). In this paper, we present a genetic-based Type II feature extraction system, referred to as GEFE (Genetic & Evolutionary Feature Extraction), for optimizing the feature sets returned by Loocal Binary Pattern Type I feature extraction for periocular biometric recognition. Our results show that not only does GEFE dramatically reduce the number of features needed but the evolved features sets also have higher recognition rates.

Human recognition based on the iris biometric is severely impacted when encountering non-ideal images of the eye characterized by occluded irises, motion and spatial blur, poor contrast, and illumination artifacts. This paper discusses the use of the periocular region surrounding the iris, along with the iris texture patterns, in order to improve the overall recognition performance in such images. Periocular texture is extracted from a small, fixed region of the skin surrounding the eye. Experiments on the images extracted from the Near Infra-Red (NIR) face videos of the Multi Biometric Grand Challenge (MBGC) dataset demonstrate that valuable information is contained in the periocular region and it can be fused with the iris texture to improve the overall identification accuracy in non-ideal situations.

In this paper we present a novel approach for personal identification, which utilizes finger surface features as a biometric identifier. Using dense range data images of the hand, we calculate the curvature-based surface representation, shape index, for the index, middle, and ring fingers. This representation is used for comparisons to determine subject similarity. Our experiments involve the use of a large data set of range images collected over time. We examine the performance of individual finger surfaces as a biometric identifier as well as the performance when using the three finger surfaces in conjunction. The results of our experiments are presented, which indicate that this approach performs well for a first-of-its-kind biometric technique.