Event Extraction (EE) involves automatically identifying and extracting structured information about events from unstructured text, including triggers, event types, and arguments. Traditional discriminative models demonstrate high precision but often exhibit limited recall, particularly for nuanced or infrequent events. Conversely, generative approaches leveraging Large Language Models (LLMs) provide higher semantic flexibility and recall but suffer from hallucinations and inconsistent predictions. To address these challenges, we propose Agreement-based Reflective Inference System (ARIS), a hybrid approach combining a Self Mixture of Agents with a discriminative sequence tagger. ARIS explicitly leverages structured model consensus, confidence-based filtering, and an LLM reflective inference module to reliably resolve ambiguities and enhance overall event prediction quality. We further investigate decomposed instruction fine-tuning for enhanced LLM event extraction understanding. Experiments demonstrate our approach outperforms existing state-of-the-art event extraction methods across three benchmark datasets.

Social media platforms are being increasingly used by malicious actors to share unsafe content, such as images depicting sexual activity, cyberbullying, and self-harm. Consequently, major platforms use artificial intelligence (AI) and human moderation to obfuscate such images to make them safer. Two critical needs for obfuscating unsafe images is that an accurate rationale for obfuscating image regions must be provided, and the sensitive regions should be obfuscated (e.g. blurring) for users' safety. This process involves addressing two key problems: (1) the reason for obfuscating unsafe images demands the platform to provide an accurate rationale that must be grounded in unsafe image-specific attributes, and (2) the unsafe regions in the image must be minimally obfuscated while still depicting the safe regions. In this work, we address these key issues by first performing visual reasoning by designing a visual reasoning model (VLM) conditioned on pre-trained unsafe image classifiers to provide an accurate rationale grounded in unsafe image attributes, and then proposing a counterfactual explanation algorithm that minimally identifies and obfuscates unsafe regions for safe viewing, by first utilizing an unsafe image classifier attribution matrix to guide segmentation for a more optimal subregion segmentation followed by an informed greedy search to determine the minimum number of subregions required to modify the classifier's output based on attribution score. Extensive experiments on uncurated data from social networks emphasize the efficacy of our proposed method. We make our code available at: https://github.com/SecureAIAutonomyLab/ConditionalVLM

Lateral Movement (LM) is one of the core stages of advanced persistent threats which continues to compromise the security posture of enterprise networks at large. Recent research work have employed Graph Neural Network (GNN) techniques to detect LM in intricate networks. Such approaches employ transductive graph learning, where fixed graphs with full nodes' visibility are employed in the training phase, along with ingesting benign data. These two assumptions in real-world setups (i) do not take into consideration the evolving nature of enterprise networks where dynamic features and connectivity prevail among hosts, users, virtualized environments, and applications, and (ii) hinder the effectiveness of detecting LM by solely training on normal data, especially given the evasive, stealthy, and benign-like behaviors of contemporary malicious maneuvers. Additionally, (iii) complex networks typically do not have the entire visibility of their run-time network processes, and if they do, they often fall short in dynamically tracking LM due to latency issues with passive data analysis.To this end, this paper proposes Jbeil, a data-driven framework for self-supervised deep learning on evolving networks represented as sequences of authentication timed events. The premise of the work lies in applying an encoder on a continuous-time evolving graph to produce the embedding of the visible graph nodes for each time epoch, and a decoder that leverages these embeddings to perform LM link prediction on unseen nodes. Additionally, we enclose a threat sample augmentation mechanism within Jbeil to ensure a well-informed notion on advanced LM attacks. We evaluate Jbeil using authentication timed events from the Los Alamos network which achieves an AUC score of 99.73% and a recall score of 99.25% in predicting LM paths, even when 30% of the nodes/edges are not present in the training phase. Additionally, we assess different realistic attack scenarios and demonstrate the potential of Jbeil in predicting LM paths with an AUC score of 99% in its inductive and transductive settings, out performing the state-of-the-art by a significant margin.

A crucial part of any home is the roof over our heads to protect us from the elements. In this paper we present the Zeitview Rooftop Geometry (ZRG) dataset for residential rooftop understanding. ZRG is a large-scale residential rooftop dataset of over 20k properties collected through roof inspections from across the U.S. and contains multiple modalities including high resolution aerial orthomosaics, digital surface models (DSM), colored point clouds, and 3D roof wireframe annotations. We provide an in-depth analysis and perform several experimental baselines including roof outline extraction, monocular height estimation, and planar roof structure extraction, to illustrate a few of the numerous potential applications unlocked by this dataset.1

Over the years, open-source software systems have become prey to threat actors. Even highly-adopted software has been crippled by unforeseeable attacks, leaving millions of devices exposed. Even as open-source communities act quickly to patch the breach, code vulnerability screening should be an integral part of agile software development from the beginning. Unfortunately, current vulnerability screening techniques are ineffective at identifying novel vulnerabilities or providing developers with code vulnerability and classification. Furthermore, the datasets used for vulnerability learning often exhibit distribution shifts from the real-world testing distribution due to novel attack strategies deployed by adversaries and as a result, the machine learning model’s performance may be hindered or biased. To address these issues, we propose a joint interpolated multitasked unbiased vulnerability classifier comprising a transformer "RoBERTa" and graph convolution neural network (GCN). We present a training process utilizing a semantic vulnerability graph (SVG) representation from source code, created by integrating edges from a sequential flow, control flow, and data flow, as well as a novel flow dubbed Poacher Flow (PF). Poacher flow edges reduce the gap between dynamic and static program analysis and handle complex long-range dependencies. Moreover, our approach reduces biases of classifiers regarding unbalanced datasets by integrating Focal Loss objective function along with SVG. Remarkably, experimental results show that our classifier outperforms state-of-the-art results on vulnerability detection with fewer false negatives and false positives. After testing our model across multiple datasets, it shows an improvement of at least 2.41% and 18.75% in the best-case scenario. Evaluations using N-day program samples demonstrate that our proposed approach achieves a 93% accuracy and was able to detect 4, zero-day vulnerabilities from popular GitHub repositories. Our code and data are available at https://github.com/pial08/SemVulDet

Text classifiers are applied at scale in the form of one-size-fits-all solutions. Nevertheless, many studies show that classifiers are biased regarding different languages and dialects. When measuring and discovering these biases, some gaps present themselves and should be addressed. First, “Does language, dialect, and topical content vary across geographical regions?” and secondly “If there are differences across the regions, do they impact model performance?”. We introduce a novel dataset called GeoOLID with more than 14 thousand examples across 15 geographically and demographically diverse cities to address these questions. We perform a comprehensive analysis of geographical-related content and their impact on performance disparities of offensive language detection models. Overall, we find that current models do not generalize across locations. Likewise, we show that while offensive language models produce false positives on African American English, model performance is not correlated with each city’s minority population proportions. Warning: This paper contains offensive language.

Remote sensing change detection, identifying changes between scenes of the same location, is an active area of research with a broad range of applications. Recent advances in multimodal self-supervised pretraining have resulted in state-of-the-art methods which surpass vision models trained solely on optical imagery. In the remote sensing field, there is a wealth of overlapping 2D and 3D modalities which can be exploited to supervise representation learning in vision models. In this paper we propose Contrastive Surface-Image Pretraining (CSIP) for joint learning using optical RGB and above ground level (AGL) map pairs. We then evaluate these pretrained models on several building segmentation and change detection datasets to show that our method does, in fact, extract features relevant to downstream applications where natural and artificial surface information is relevant.1

The speed and volume at which misinformation spreads on social media have motivated efforts to automate fact-checking which begins with stance detection. For fake news stance detection, for example, many classification-based models have been proposed often with high complexity and hand-crafted features. Although these models can achieve high accuracy scores on a targeted small corpus of fake news, few are evaluated on a larger corpus of fake and conspiracy sites due to efficiency limitations and the lack of compatibility with the actual fact-checking process. In this article, we propose a practical two-stage stance detection model that is tailored to the real-life problem. Specifically, we integrate an information retrieval system with an end to end memory network model to sort articles based on their relevance to the claim and then identify the fine-grained stance of each relevant article towards its given claim. We evaluate our model on the Fake News Challenge dataset (FNC-1). The results show that the performance of our model is comparable to those of the state-of-the-art models, average weighted accuracy of 82.1, while it closely follows the real-life process of fact-checking. We also validate our model with a large dataset from a real-life fact-checking website (i.e.,Snopes.com), and the findings demonstrate the capability of the model in distinguishing false from true news headlines.

Human behavior is the confluence of output from voluntary and involuntary motor systems. The neural activities that mediate behavior, from individual cells to distributed networks, are in a state of constant flux. Artificial intelligence (AI) research over the past decade shows that behavior, in the form of facial muscle activity, can reveal information about fleeting voluntary and involuntary motor system activity related to emotion, pain, and deception. However, the AI algorithms often lack an explanation for their decisions, and learning meaningful representations requires large datasets labeled by a subject-matter expert. Motivated by the success of using facial muscle movements to classify brain states and the importance of learning from small amounts of data, we propose an explainable self-supervised representation-learning paradigm that learns meaningful temporal facial muscle movement patterns from limited samples. We validate our methodology by carrying out comprehensive empirical study to predict future speech behavior in a real-world dataset of adults who stutter (AWS). Our explainability study found facial muscle movements around the eyes (p

With the ever-increasing amount of data, the central challenge in multimodal learning involves limitations of labelled samples For the task of classification, techniques such as meta-learning, zero-shot learning, and few-shot learning showcase the ability to learn information about novel classes based on prior knowledge. Recent techniques try to learn a cross-modal mapping between the semantic space and the image space. However, they tend to ignore the local and global semantic knowledge. To overcome this problem, we propose a Multimodal Variational Auto-Encoder (M-VAE) which can learn the shared latent space of image features and the semantic space. In our approach we concatenate multimodal data to a single embedding before passing it to the VAE for learning the latent space. We propose the use of a multi-modal loss during the reconstruction of the feature embedding through the decoder. Our approach is capable to correlating modalities and exploit the local and global semantic knowledge for novel sample predictions. Our experimental results using a MLP classifier on four benchmark datasets show that our proposed model outperforms the current state-of-the-art approaches for generalized zero-shot learning.

Visual Question Answering (VQA) models have achieved significant success in recent times. Despite the success of VQA models, they are mostly black-box models providing no reasoning about the predicted answer, thus raising questions for their applicability in safety-critical such as autonomous systems and cyber-security. Current state of the art fail to better complex questions and thus are unable to exploit compositionality. To minimize the black-box effect of these models and also to make them better exploit compositionality, we propose a Dynamic Neural Network (DMN), which can understand a particular question and then dynamically assemble various relatively shallow deep learning modules from a pool of modules to form a network. We incorporate compositional temporal attention to these deep learning based modules to increase compositionality exploitation. This results in achieving better understanding of complex questions and also provides reasoning as to why the module predicts a particular answer. Experimental analysis on the two benchmark datasets, VQA2.0 and CLEVR, depicts that our model outperforms the previous approaches for Visual Question Answering task as well as provides better reasoning, thus making it reliable for mission critical applications like safety and security.

High-fidelity visual simulation-based environments and advanced learning algorithms can be used to train robots to carry out specific tasks. Behavior cloning is a fast and easy way to train robots to learn from experience by modeling their actions according to human actions. As we make use of these agents in our day-to-day life, the robustness of such system-of-systems trained on simulation environments are of great concern. In this paper, we explore adversarial attacks in simulation environments, specifically for behavioral cloning models that cause the adversary to be able to take control of the steering mechanism of an autonomous agent. We focus our attention on improving latency and noticeability, two fundamental issues with adversarial attacks, by reducing the number of iterations to a single step during a white-box adversarial attack within a noticeability threshold. More specifically, the gradients at the image input layer and the output layer of the neural network are utilized in the adversarial attack. We implement a hybridized version of the fast gradient sign and basic iterative methods to attack the input image and fool the agent. We've shown that our method reduces the attack time per frame to within 3 milliseconds.

Cybersickness is one of the primary challenges to the usability and acceptability of virtual reality (VR). Cybersickness can cause motion sickness-like discomforts, including disorientation, headache, nausea, and fatigue, both during and after the VR immersion. Prior research suggested a significant correlation between physiological signals and cybersickness severity, as measured by the simulator sickness questionnaire (SSQ). However, SSQ may not be suitable for automatic detection of cybersickness severity during immersion, as it is usually reported before and after the immersion. In this study, we introduced an automated approach for the detection and prediction of cybersickness severity from the user's physiological signals. We collected heart rate, breathing rate, heart rate variability, and galvanic skin response data from 31 healthy participants while immersed in a VR roller coaster simulation. We found a significant difference in the participants' physiological signals during their cybersickness state compared to their resting baseline. We compared a support vector machine classifier and three deep neural classifiers for cybersickness severity detection and prediction in two minutes' future, given the previous two minutes of physiological signals. Our proposed simplified convolutional long short-term memory classifier achieved an accuracy of 97.44% for detecting current cybersickness severity and 87.38% for predicting future cybersickness severity from the physiological signals.

Deep learning (DL) is applied to predict signal-to-noise ratio (SNR) in de facto LTE and 5G systems in a non-data-aided (NDA) manner. Various channel conditions and impairments are considered, including modulation types, path delays, and Doppler shifts. Both time-domain and frequency-domain signal grids are evaluated as inputs for SNR prediction. A combination of convolutional neural network (CNN) and long short term memory (LSTM) - CNN-LSTM - is used as the SNR predictor. Learning both spatial and temporal features is known to improve DL prediction accuracy. Techniques employed to enhance performance are SNR range/resolution manipulation, binary prediction, and multiple input prediction. Computer simulation is conducted using MATLAB LTE, 5G, and DL toolboxes to generate OFDM signals, model fading channels with AWGN noise, and construct CNN-LSTM. Simulation results show, with off-line training, DL based prediction of SNR in LTE and 5G systems has better accuracy and latency than traditional estimation techniques. Specifically, SNR prediction for SNR range of [-4, 32] dB and resolution of 2 dB utilizing time-domain signals has an accuracy of 100%, hence normalized mean square error (NMSE) of zero, and a latency of 1 millisecond or less.

Twitter has a significant user base, with reportedly over 300 million active user accounts. Twitter, a micro blog service, limits the length of each tweet, keeping them short and concise. The contents of tweets include news, trending topics, emotions, and opinions. This makes Twitter a (popular) source of data for social science, marketing, psychology and news. Twitter users tend to use emojis, slang, and acronyms in order to fit more content within the character limit. The use of emojis in tweets complicates efforts in text mining and emotion analysis, as such emojis can also be used to express sarcasm when used in different contexts. In this paper, we use Twitter API to mine tweets that were geotagged by users and apply text analytics to the tweets. We also develop a system to detect events using the geospatial emotion vector in the area we are monitoring. Combining graph theory, machine learning semantics, and statistics with the geospatial emotion vectors, we track trending topics during times of extreme emotion. Our system correctly detected ten out of 11 events that we used in our study. Our findings suggest that Robert Parks theory on Expressive Groups and Gustave Le Bons Theory on Social Contagion hold true in the Twittersphere.

The aptitude to identify the emotional states of others and response to exposed emotions is an important aspect of human social intelligence. Robots are expected to be prevalent in society to assist humans in various tasks. Human-robot interaction (HRI) is of critical importance in the assistive robotics sector. Smart digital assistants and assistive robots fail quite often when a request is not well defined verbally. When the assistant fails to provide services as desired, the person may exhibit an emotional response such as anger or frustration through expressions in their face and voice. It is critical that robots understand not only the language, but also human psychology. A novel affection-based perception architecture for cooperative HRIs is studied in this paper, where the agent is expected to recognize human emotional states, thus encourages a natural bonding between the human and the robotic artifact. We propose a method to close the loop using measured emotions to grade HRIs. This metric will be used as a reward mechanism to adjust the assistant's behavior adaptively. Emotion levels from users are detected through vision and speech inputs processed by deep neural networks (NNs). Negative emotions exhibit a change in performance until the user is satisfied.

During humanitarian crises, there is a need for a large amount of information in a short period of time. Such need creates the base for misinformation such as rumors, fake news or hoaxes to spread within and outside the affected community. This results in (mis)information harms that can generate serious short term or long-term consequences. In such situations, there is a need for a joint human-machine effort to mitigate such harms. Computational scientists have created misinformation detection systems and algorithms, while social scientists have examined the roles of involved parties, examined the way misinformation spreads and convinces people. However, there has been no work, to our knowledge, in examining situations when the machine and human interact with each other in the context of misinformation. In order to systematically examine the harms from misinformation, we draw on Activity Theory to suggest a suitable framework. Such a framework enables interactions among the human and machines and their respective loops for the purpose of mitigation of misinformation harms

Deep neural networks are widely used in many walks of life. Techniques such as transfer learning enable neural networks pre-trained on certain tasks to be retrained for a new duty, often with much less data. Users have access to both pre-trained model parameters and model definitions along with testing data but have either limited access to training data or just a subset of it. This is risky for system-critical applications, where adversarial information can be maliciously included during the training phase to attack the system. Determining the existence and level of attack in a model is challenging. In this paper, we present evidence on how adversarially attacking training data increases the boundary of model parameters using as an example of a CNN model and the MNIST data set as a test. This expansion is due to new characteristics of the poisonous data that are added to the training data. Approaching the problem from the feature space learned by the network provides a relation between them and the possible parameters taken by the model on the training phase. An algorithm is proposed to determine if a given network was attacked in the training by comparing the boundaries of parameters distribution on intermediate layers of the model estimated by using the Maximum Entropy Principle and the Variational inference approach.

This paper focuses on convolution neural network quantization problem. The quantization has a distinct stage of data conversion from floating-point into integer-point numbers. In general, the process of quantization is associated with the reduction of the matrix dimension via limited precision of the numbers. However, the training and inference stages of deep learning neural network are limited by the space of the memory and a variety of factors including programming complexity and even reliability of the system. On the whole the process of quantization becomes more and more popular due to significant impact on performance and minimal accuracy loss. Various techniques for networks quantization have been already proposed, including quantization aware training and integer arithmetic-only inference. Yet, a detailed comparison of various quantization configurations, combining all proposed methods haven't been presented yet. This comparison is important to understand selection of quantization hyperparameters during training to optimize networks for inference while preserving their robustness. In this work, we perform in-depth analysis of parameters in the quantization aware training, the process of simulating precision loss in the forward pass by quantizing and dequantizing tensors. Specifically, we modify rounding modes, input preprocessing, output data signedness, bitwidth of the quantization and locations of precision loss simulation to evaluate how they affect accuracy of deep neural network aimed at performing efficient calculations on resource-constrained devices.

Predicting driver's cognitive states using deep learning from electroencephalography (EEG) signals is considered this paper. To address the challenge posed by limited labeled training samples, a semi-supervised Wasserstein Generative Adversarial Network with gradient penalty (sWGAN-GP) is proposed. The proposed sWGAN-GP includes a classifier with the shared architecture with the discriminator in GAN and its loss function enables the augmentation of limited training samples with generated EEG samples during training, thus resulting in improved classification performance. The several modeling challenges including frequency artifacts and training instability, are also considered. The test results on predicting the alert and drowsy states from a simulated driving experiment demonstrate improved prediction performance and training stability over the baseline semi-supervised GAN and a convolutional neural network model.

Electroencephalography (EEG) data is difficult to obtain due to complex experimental setups and reduced comfort due to prolonged wearing. This poses challenges to train powerful deep learning model due to the limited EEG data. Hence, being able to generate EEG data computationally is highly desirable. We propose a novel Conditional Wasserstein Generative Adversarial Network with gradient penalty (cWGAN-GP) that can be trained to synthesize EEG data for different cognitive events. This network addresses several modeling challenges, including frequency artifacts and training instability. The proposed GAN model is tested to generate one channel EEG data for the rapid serial visual presentation. We demonstrated the validity of the generated samples using several evaluation metrics and show that the synthesized EEG data can augment the real EEG data to achieve improved event classification performance.

The capability to perform identity reidentification in a crowd (e.g., from video feeds from a network of cameras, and social media platforms, such as Facebook and Instagram) efficiently and effectively is increasingly important, as evident in recent real-world events (e.g., terrorist attacks on places of mass gatherings in different countries). However, real-time reidentification in a network of cameras, such as those deployed in a smart city, and from other sources, such as social media platforms, remains a challenging task. In this paper, a new embedding algorithm pipeline is presented to extract and administrate the crowd-sourced facial image features (e.g., social media platforms and multicameras in a dense crowd, such as a stadium or airport). The proposed facial embedding is a privacy-aware parameterized function, which maps facial images to high-dimensional vectors in order to facilitate the identification and tracking of individuals. In other words, we are able to uniquely identify person(s) of interest, without the need to determine their true identity. To extract the facial embedding information in crowds, concurrent residual neural network (ResNet) embedding pipeline for each camera is proposed. Specifically, facial embedding feature vectors are generated in real-time by each camera using the proposed enhanced ResNet architecture, which is trained with vectorized-l2-loss function for face recognition. The multivariate kernel density estimation matching algorithm is then applied to facial embedding pipelines generated by cameras at the fog cloud for identity reidentification and security verification. This allows us to ensure the privacy of individuals captured by the camera without compromising on the capability for identity reidentification. Evaluations using mixed datasets in real-time demonstrate that our proposed approach achieves a 2.6% accuracy over other state-of-the-art approaches.

Hypervisor-based virtualization rapidly becomes a commodity, and it turns valuable in many scenarios such as resource optimization, uptime maximization, and consolidation. Container-based application virtualization is an appropriate solution to develop a light weighted partitioning by providing application isolation with less overhead. Undoubtedly, container based virtualization delivers a lightweight and efficient environment, however raises some security concerns as it allows isolated processes to utilize an underlying host kernel. A new security layer with the Most Privileged Container (MPC) is proposed in this article. The proposed MPC layer exhibits three main functional blocks: Access policies, Black list database, and Runtime monitoring. The introduced MPC layer implements privilege based access control and assigns resource access permissions based on policies and the security profiles of containerized application user processes. Furthermore, the monitoring block examines the runtime behavior of containers and black list database is updated if the container violets its policies. The proposed MPC layer provides higher level of application container security against potential threats.

Recently, capabilities of many computer vision tasks have significantly improved due to advances in Convolutional Neural Networks. In our research, we demonstrate that it can be also used for face detection from low resolution thermal images, acquired with a portable camera. The physical size of the camera used in our research allows for embedding it in a wearable device or indoor remote monitoring solution for elderly and disabled people. The benefits of the proposed architecture were experimentally verified on the thermal video sequences, acquired in various scenarios to address possible limitations of remote diagnostics: movements of the person performing a diagnose and movements of the examined person. The achieved short processing time (42.05±0.21ms) along with high model accuracy (false positives -0.43%; true positives for the patient focused on a certain task -89.2%) clearly indicates that the current state of the art in the area of image classification and face tracking in thermography was significantly outperformed.

License Plate Recognition System (LPRS) plays a vital role in smart city initiatives such as traffic control, smart parking, toll management and security. In this article, a cloud-based LPRS is addressed in the context of efficiency where accuracy and speed of processing plays a critical role towards its success. Signature-based features technique as a deep convolutional neural network in a cloud platform is proposed for plate localization, character detection and segmentation. Extracting significant features makes the LPRS to adequately recognize the license plate in a challenging situation such as i) congested traffic with multiple plates in the image ii) plate orientation towards brightness, iii) extra information on the plate, iv) distortion due to wear and tear and v) distortion about captured images in bad weather like as hazy images. Furthermore, the deep learning algorithm computed using bare-metal cloud servers with kernels optimized for NVIDIA GPUs, which speed up the training phase of the CNN LPDS algorithm. The experiments and results show the superiority of the performance in both recall and precision and accuracy in comparison with traditional LP detecting systems.