Recently, Knowledge Graphs (KGs) have been successfully coupled with Large Language Models (LLMs) to mitigate their hallucinations and enhance their reasoning capability, e.g., KG-based retrieval-augmented framework. However, current KG-LLM frameworks lack rigorous uncertainty estimation, limiting their reliable deployment in applications where the cost of errors is significant. Directly incorporating uncertainty quantification into KG-LLM frameworks presents a challenge due to their more complex architectures and the intricate interactions between the knowledge graph and language model components. To address this crucial gap, we propose a new trustworthy KG-LLM framework, UAG (Uncertainty Aware Knowledge-Graph Reasoning), which incorporates uncertainty quantification into the KG-LLM framework. We design an uncertainty-aware multi-step reasoning framework that leverages conformal prediction to provide a theoretical guarantee on the prediction set. To manage the error rate of the multi-step process, we additionally introduce an error rate control module to adjust the error rate within the individual components. Extensive experiments show that UAG can achieve any pre-defined coverage rate while reducing the prediction set/interval size by 40% on average over the baselines.

We consider the problem of late multimodal fusion for discriminative learning. Motivated by noisy, multi-source domains that require understanding the reliability of each data source, we explore the notion of credibility in the context of multimodal fusion. We propose a combination function that uses probabilistic circuits (PCs) to combine predictive distributions over individual modalities. We also define a probabilistic measure to evaluate the credibility of each modality via inference queries over the PC. Our experimental evaluation demonstrates that our fusion method can reliably infer credibility while being competitive with the state-of-the-art.

Information Fusion seeks to combine observations from sensor data and/or human opinions to make a decision with reduced uncertainty. One common method for physics-based and human-derived information fusion (PHIF) that is well established is that of the analytical hierarchy processing (AHP). When decisions are not crisp, the Fuzzy AHP is a method towards establishing the boundary between decisions and the consistency among the decisions. In this paper, the Fuzzy AHP is utilized to assess information fusion designs developed from the Multisource AI Scorecard Table (MAST). MAST has devised a set of scores that can be ordinal (crisp), discrete (probabilities), and semantic (labels); and this paper explores a measure of uncertainty as from static fuzzy logic. The main finding is that assessment from different users with different ratings (i.e., Level 5 Fusion) can be enhanced by fuzzy AHP to check for consistency. If the combined PHIF results are not consistent, users would then need to subjectively identify weaknesses and limitations to continue the information fusion systems engineering design process.

We propose a neuro-symbolic approach for learning causal complex event models from multi-source data, integrating causal discovery and temporal logic. Given resource constraints, we employ signal-level fusion by averaging the data from different antennas of the same WiFi receiver, followed by downsampling to reduce computational overhead. We consider a dataset of WiFi Channel State Information capturing human activities alongside video data from which we extract atomic symbolic activities such as “moving the upper arm.” The extracted symbolic information is processed through LPCMCI (Latent PCMCI). This causal discovery method extends PCMCI (Peter and Clark Momentary Conditional Independence) to handle latent dependencies across multiple time steps while mitigating false discoveries due to auto-correlations. The resulting causal structure is then translated into a temporal logic formula, which serves as a symbolic constraint in a neuro-symbolic learning pipeline. To efficiently process and learn from these structured constraints under resource limitations, we leverage Spiking Neural Networks, which offer energy-efficient computation while preserving temporal dynamics.

This paper focuses on the multi-target tracking using the Stone Soup framework. In particular, we aim at evaluation of two multi-target tracking scenarios based on the simulated class-B dataset and ADS-B class-A dataset provided by OpenSky Network. The scenarios are evaluated w.r.t. selection of a local state estimator using a range of the Stone Soup metrics. Source code with scenario definitions and Stone Soup set-up are provided along with the paper.

Evaluating uncertainty in large language model (LLM) reasoning is challenging due to their vast parameter space, abstract knowledge representation, and limited transparency regarding training data. While normative formalisms, such as deductive logic, clearly define sound reasoning in the absence of uncertainty, reasoning under uncertainty admits multiple approaches, including probabilistic reasoning (e.g. Bayesian), belief function reasoning (e.g. Dempster-Shafer), or fuzzy logic, to name a few. This paper examines how LLMs handle uncertainty by analyzing outcomes based on an atomic fusion and reasoning problem. We establish a point of reference using the simplest of fusion topologies to facilitate transparency and understanding of how LLMs align with established theories. The reasoning approaches of different LLMs with varying complexities are compared to established normative frameworks, providing insights into which formalism best aligns with LLM reasoning and assessing its soundness and consistency. A deviation function for assessment is developed, and the results indicate that the tested LLMs' reasoning under uncertainty does not consistently align with established theories, even for the simplest information fusion topologies. These preliminary results form the basis for further investigations and LLM refinements.

Wi-Fi devices, akin to passive radars, can discern human activities within indoor settings due to the human body’s interaction with electromagnetic signals. Current Wi-Fi sensing applications predominantly employ data-driven learning techniques to associate the fluctuations in the physical properties of the communication channel with the human activity causing them. However, these techniques often lack the desired flexibility and transparency. This paper introduces DeepProbHAR, a neuro-symbolic architecture for Wi-Fi sensing, providing initial evidence that Wi-Fi signals can differentiate between simple movements, such as leg or arm movements, which are integral to human activities like running or walking. The neuro-symbolic approach affords gathering such evidence without needing additional specialised data collection or labelling. The training of DeepProbHAR is facilitated by declarative domain knowledge obtained from a camera feed and by fusing signals from various antennas of the Wi-Fi receivers. DeepProbHAR achieves results comparable to the state-of-the-art in human activity recognition. Moreover, as a by-product of the learning process, DeepProbHAR generates specialised classifiers for simple movements that match the accuracy of models trained on finely labelled datasets, which would be particularly costly.

This paper deals with state estimation of nonlinear stochastic dynamic models. In particular, the stochastic integration rule, which provides asymptotically unbiased estimates of the moments of nonlinearly transformed Gaussian random variables, is reviewed together with the recently introduced stochastic integration filter (SIF). Using SIF, the respective multi-step prediction and smoothing algorithms are developed in full and efficient square-root form. The stochastic-integration-rule-based algorithms are implemented in Python (within the Stone Soup framework) and in MATLAB® and are numerically evaluated and compared with the well-known unscented and extended Kalman filters using the Stone Soup defined tracking scenario.

The Unscented Kalman Filter (UKF) stands out as a versatile and dynamic algorithm, celebrated for its prowess in estimating the states of nonlinear dynamical systems within uncertain environments. However, the accuracy of UKF state estimations hinges significantly on the thoughtful selection of pivotal hyperparameters, $\alpha, \beta$, and $\kappa$, which are integral in shaping the distribution of sigma points around the current state estimate. Prevailing methods for tuning these parameters encompass heuristic approaches such as arbitrarily fix $\alpha$ at 0.001, $\kappa$ at 0, and $\beta$ at 2 for Gaussian noise, though the efficacy of such rules heavily hinges on the intricacies of the specific problem. Alternatively, the grid search technique seeks to optimize these hyperparameters, but it can become computationally burdensome, particularly when the search space is extensive and intricate. To navigate these hurdles, this paper introduces the RNN-UKF algorithm-a pioneering strategy that leverages recurrent neural networks (RNNs) to autonomously fine-tune UKF hyperparameters. The inherent adaptability of RNNs is harnessed to dynamically adjust the $\alpha, \beta$, and $\kappa$ parameters of the unscented transformation during each state estimation step, all aimed at minimizing the root mean squared error (RMSE). Demonstrated through numerical simulations, we provide compelling evidence that the RNN-UKF approach outperforms both heuristic rule of thumb and grid search techniques in terms of RMSE performance. Moreover, the RNN-UKF methodology showcases its superiority over the conventional extended Kalman filter (EKF) approach, particularly in scenarios characterized by substantial system noise.

Multimodal fusion is important for building intelligent systems that exploit patterns across diverse data sources for improved decision-making. However, the reliability and robustness of these systems in safety-critical domains are often compromised by the inherent noise and incompleteness of data. Probabilistic Circuits (PCs) have recently emerged as a promising approach for late (or decision) fusion. Their strength lies in being both expressive and capable of inferring source credibility due to their ability to tractably perform exact probabilistic inference. However, their ability to handle missing data and their reliability in practical scenarios remains underexplored. This work investigates the robustness of PCs as fusion functions in scenarios with missing and noisy data; particularly by examining their impact on the calibration and reliability of the resulting classifiers. Our findings show that PCs not only enable the modeling of complex correlations across modalities but also lead to calibrated and reliable classifiers, highlighting their potential as a robust fusion mechanism in multimodal systems.

Consistent bi-directional inferences are the key for many machine learning applications. Without consistency, inverse learning-based inferences can cause fuzzy images, erroneous control signals, and cascading failure in SCADA systems. Since standard deep neural networks (DNNs) are not inherently invertible to offer consistency, some past methods reconstruct DNN architecture analytically for one-to-one correspondence but compromise key features such as universal approximation. Other work maintains the capability of universal approximation in DNNs via iterative numerical approximation. However, these methods limit their applications significantly due to Lipschitz conditions and issues of numerical convergence. The dilemma of the analytical and numerical methods is the incompatibility between nonlinear layer compositions and bijective function construction for inverse modeling. Based on the observation, we propose decomposed-invertible-pathway DNNs (DipDNN). It relaxes the redundant reconstruction of nested DNN in the former methods and eases the Lipschitz constraint. As a result, we strictly guarantee the consistency of global inverse modeling without harming DNN's capability for universal approximation. As numerical stability and generalizability are keys for controlling critical infrastructures, we integrate contractive property with a parallel structure for inductive biases, leading to stable performance. Numerical results show that DipDNN performs significantly better than past methods, thanks to its enforcement of inverse consistency, numerical stability, and physical regularization.

Although the performance of Temporal Action Segmentation (TAS) has been improved in recent years, achieving promising results often comes with a high computational cost due to dense inputs, complex model structures, and resource-intensive post-processing requirements. To improve the efficiency while keeping the high performance, we present a novel perspective centered on per-segment classification. By harnessing the capabilities of Transformers, we tokenize each video segment as an instance token, endowed with intrinsic instance segmentation. To realize efficient action segmentation, we introduce BaFormer, a boundary-aware Transformer network. It employs instance queries for instance segmentation and a global query for class-agnostic boundary prediction, yielding continuous segment proposals. During inference, BaFormer employs a simple yet effective voting strategy to classify boundary-wise segments based on instance segmentation. Remarkably, as a single-stage approach, BaFormer significantly reduces the computational costs, utilizing only 6% of the running time compared to the state-of-the-art method DiffAct, while producing better or comparable accuracy over several popular benchmarks. The code for this project is publicly available at https://github.com/peiyao-w/BaFormer.

The Multisource AI Scorecard Table (MAST) is a checklist tool to inform the design and evaluation of trustworthy AI systems based on the U.S. Intelligence Community’s analytic tradecraft standards. In this study, we investigate whether MAST can be used to differentiate between high and low trustworthy AI-enabled decision support systems (AI-DSSs). Evaluating trust in AI-DSSs poses challenges to researchers and practitioners. These challenges include identifying the components, capabilities, and potential of these systems, many of which are based on the complex deep learning algorithms that drive DSS performance and preclude complete manual inspection. Using MAST, we developed two interactive AI-DSS testbeds. One emulated an identity-verification task in security screening, and another emulated a text-summarization system to aid in an investigative task. Each testbed had one version designed to reach low MAST ratings, and another designed to reach high MAST ratings. We hypothesized that MAST ratings would be positively related to the trust ratings of these systems. A total of 177 subject-matter experts were recruited to interact with and evaluate these systems. Results generally show higher MAST ratings for the high-MAST compared to the low-MAST groups, and that measures of trust perception are highly correlated with the MAST ratings. We conclude that MAST can be a useful tool for designing and evaluating systems that will engender trust perceptions, including for AI-DSS that may be used to support visual screening or text summarization tasks. However, higher MAST ratings may not translate to higher joint performance, and the connection between MAST and appropriate trust or trustworthiness remains an open question.

Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

This paper presents an eXplainable Deep Reinforcement Learning (XDRL) based strategy for solving the proposed problem of fleet-level aircraft maintenance scheduling (AMS) optimization. The XDRL-AMS considers various factors such as the aircraft’s initial status, mission requirements, maintenance resource capacity, and operational constraints to create a maintenance schedule for a specified period. The schedule aims to balance both mission readiness and cost reduction. We developed an RL environment, called AMS-Gym, using the OpenAI Gym toolkit specifically designed for this problem. AMS-Gym is highly flexible, allowing for easy extension to more complex scenarios and incorporating additional explanatory capabilities. The explainable RL capability was achieved by utilizing a decomposed reward Deep Q-Network (drDQN) algorithm. In the context of the AMS scenario, the drDQN consists of two parts: (i) a DQN that aims to maximize the mission accomplishment objective, and (ii) a DQN that aims to minimize the maintenance cost objective. As a result, the proposed drDQN strategy can generate real-time aircraft maintenance decisions, explain why those decisions were selected, and present the tradeoffs between the chosen action and non-selected alternatives. Experiment results show that the proposed drDQN performs well, providing an approximate solution to the vanilla DQN with a simpler structure while offering the ability to explain its decisions. In addition, a web-based prototype with an intuitive textual and visual user interface was developed to demonstrate the feasibility of the drDQN approach.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

AI-based decision support solutions require life cycles that adequately address critical steps, such as (i) finding suitable machine learning (ML) methods for the problem at hand, (ii) preparing and executing adequate data acquisition processes and (iii) tractable evaluation of the overall solution. Understanding the data generating processes is key in achieving this. Training and test data can be seen as a result of a causal data generation process, a sampling process in which the data is collected from different sources that are influenced by multiple interdependent phenomena. This is represented by a Qualitative Model of Data Generation Processes (QM-DGP), a causal graphical model. QM-DGP facilitates analysis of the complexity of the underlying data generating processes that can inform the development of trustable ML-based solutions in multiple ways. Firstly, this analysis is the basis for the determination of the required complexity of the ML models. Secondly, it facilitates the determination of the quantities of training data supporting good learning results. Thirdly, it can provide guidance for a systematic simplification of the models, supporting tractable solutions without significantly reduced performance. The construction of QM-DGP and the analysis benefit from sound theoretical concepts, such as d-separation and I-Maps. Experimental results with simulated data indicate that the approach can be effective in predicting the required quantities of training data and the determination of the modelling complexity using different types of models.

The recent advancements in the Internet of Video Things (IoVT) and Edge-Fog-Cloud Computing paradigm make smart public safety surveillance (SPSS) a realistic solution for an effective public safety service in smart cities. Typically, a fully functional SPSS system requires multiple sensory inputs for situational awareness (SAW). As an essential component in the context of highly complex, dynamic, and heterogeneous smart city operations, SPSS is expected to be environment-resilient. Personal safety is among the top concerns of the residents in smart cities, and correspondingly pedestrian detectors are critical. Contemporary pedestrian detectors use optical cameras, whose accuracy is diminished in low-light environments, and they are rendered ineffective when obstacles block the direct line of sight to the camera. Complementary imaging sensors such as infrared have shown promise. This paper presents a full-spectrum, environment-resilient surveillance platform as an ultimate solution, which consists of multiple imaging units to cover a wide sensing spectrum. The initial hybrid pedestrian detection (HYPE) scheme is based on the fusion of data obtained from an IoVT network equipped with optical and thermal cameras. We demonstrate that training the YOLOv5 object detection model on a dataset of infrared images improves its accuracy in the detection of humans present in thermal surveillance images. A 41% decrease in objectness loss is achieved after transfer learning is performed.

In high-rate structural health monitoring, it is crucial to quickly and accurately assess the current state of a component under dynamic loads. State information is needed to make informed decisions about timely interventions to prevent damage and extend the structure’s life. In previous studies, a dynamic reproduction of projectiles in ballistic environments (DROPBEAR) testbed was used to evaluate the accuracy of state estimation techniques through dynamic analysis. This paper extends previous research by incorporating the local eigenvalue modification procedure (LEMP) and data fusion techniques to create a more robust state estimate using optimal sampling methodologies. The process of estimating the state involves taking a measured frequency response of the structure, proposing frequency response profiles, and accepting the most similar profile as the new mean for the position estimate distribution. Utilizing LEMP allows for a faster approximation of the proposed model with linear time complexity, making it suitable for 2D or sequential damage cases. The current study focuses on two proposed sampling methodology refinements: distilling the selection of candidate test models from the position distribution and applying a Kalman filter after the distribution update to find the mean. Both refinements were effective in improving the position estimate and the structural state accuracy, as shown by the time response assurance criterion and the signal-to-noise ratio with up to 17% improvement. These two metrics demonstrate the benefits of incorporating data fusion techniques into the high-rate state identification process.

Accurate tracking of transparent objects, such as glasses, plays a critical role in many robotic tasks such as robot-assisted living. Due to the adaptive and often reflective texture of such objects, traditional tracking algorithms that rely on general-purpose learned features suffer from reduced performance. Recent research has proposed to instill trans-parency awareness into existing general object trackers by fusing purpose-built features. However, with the existing fusion techniques, the addition of new features causes a change in the latent space making it impossible to incorporate transparency awareness on trackers with fixed latent spaces. For example, many of the current days' transformer-based trackers are fully pre-trained and are sensitive to any latent space perturbations. In this paper, we present a new feature fusion technique that integrates transparency information into a fixed feature space, enabling its use in a broader range of trackers. Our proposed fusion module, composed of a transformer encoder and an MLP module, leverages key query-based transformations to embed the transparency information into the tracking pipeline. We also present a new two-step training strategy for our fusion module to effectively merge transparency features. We propose a new tracker architecture that uses our fusion techniques to achieve superior results for transparent object tracking. Our proposed method achieves competitive results with state-of-the-art trackers on TOTB, which is the largest transparent object tracking benchmark recently released. Our results and the implementation of code will be made publicly available at https://github.com/kalyan0510/TOTEM.

In recent years, indoor positioning systems (IPSs) have received attention from many research fields, such as robotics, navigation, human–computer interaction, etc. However, IPS based on passive radio frequency (PRF) technology is still rare. This article proposes a 3-D IPS based on received signal strength (RSS) distribution and Gaussian process regression (GPR). Traditional RSS-based positioning systems have a transmitter with known frequencies, while in the proposed PRf signal of Opportunity—3D IPS (PRO-3DIPS), the system neither deploys new transmitters nor uses any a priori knowledge of transmitters. Furthermore, PRO-3DIPS integrates multiple Signal of Opportunity (SoOP) sources, shadowing, fading, and also captures scenario signatures. Data collection and analysis of PRF-based RSS distribution in 3-D space enables the capability of 3-D positioning. Three methods are applied and compared to find the frequency band most impacted by the scenario to achieve the best positioning performance as well as used in the estimation of RSS distribution. The RSS distribution is estimated by measuring the PRF spectrum on a fixed grid in the scenario. Using the RSS distribution, the GPR can accurately locate the receiver position. RSS at 90-gridded positions were collected in the experiment scenario, with one hundred samples at each position. The experimental result shows that a root-mean-square error (RMSE) of the proposed PRO-3DIPS is 0.292 m when the sampling distance is 1 m. The result demonstrates that the PRF spectrum is a new modality for the positioning task, which demonstrates better performance than most existing RF-based technologies.

The fast integration of 5G communication, Artificial Intelligence (AI), and Internet-of-Things (IoT) technologies is envisioned to enable Next Generation Networks (NGNs) for diverse smart services and user-defined applications for Smart Cities. However, it is still challenging to build a scalable and efficient infrastructure that satisfies the various performance, security, and management demands by heterogeneous IoT applications across multiple administrative domains. This paper presents a dynamic edge resource federation architecture, which integrates the concept of network slicing (NS) and blockchain to improve scalability, dynamicity, and security for multi-domain IoT applications. A NS-enabled dynamic edge resource federation framework adopts intelligent mechanisms to support efficient multi-domain service coordination that satisfies diverse Quality of Service (QoS) and security requirements. We propose a Hierarchical Integrated Federated Ledger (HIFL), which aims to guarantee decentralized security and privacy-preserving properties in multi-domain resource orchestration and service re-adjustment. As a secure-by-design solution, HIFL is promising to support efficient, trust and secured end-to-end IoT services. A preliminary proof-of-concept prototype has been implemented for comparing intra- and inter-domain performance expectations.

The large amounts of movement data collected from various sources such as wide area motion imagery (WAMI) and mobile phone apps call for innovative technologies to infer valuable information from these data. In this paper, we present two such tools to extract pattern of life (PoL) information from vehicle trajectories. The first tool, intersection traffic analysis (ITA) detects abnormal traffic patterns in major street intersections; while the second one, frequent trajectory patterns analysis (FTPA) discerns the most frequent trajectory patterns in a given time-interval in the region of concern. Both tools support comprehensive trajectory-based pattern of life situation awareness. Using both simulated trajectories and the measurements extracted from WAMI imagery demonstrate ITA and FTPA utility.

In modern complex physical systems, advanced sensing technologies extend the sensor coverage but also increase the difficulties of improving system monitoring capabilities based on real-time data availability. Traditional model-based methods of sensor management are limited to specific systems/settings, which can be challenged when system knowledge is intractable. Fortunately, the large amount of data collected in real-time allows machine learning methods to be a complement. Especially, reinforcement learning-based control is recognized for its capability to dynamically interact with systems. However, the direct implementation of learning methods easily overfits and results in inaccurate physics modeling for sensor management. Although physical regularization is a popular direction to bridge the gap, learning-based sensor control still suffers from convergence failure under highly complex and uncertain scenarios. This paper develops physics-embedded and self-supervised reinforcement learning for sensor management using an intrinsic reward. Specifically, the intrinsic-motivated sensor management (IMSM) constructs the local surprise information from the physical latent features, which captures hidden states in observations, and thus intrinsically motivates the agent to speed-up exploration. We show that the designs can not only relieve the lack of consistency with underlying physics/physical dynamics, but also adapt the global objective of maximizing monitoring capabilities to local environment changes. We demonstrate its effectiveness by experiments on physical system sensor control. The proposed model is implemented for the sensor management of unmanned vehicles and sensor rescheduling in complex/settled power systems, with or without observability constraints. Numerical results show that our model provides consistently higher threat detection accuracy and better observability recovery, as compared to existing methods.

Leveraging modern Artificial Intelligence (AI) technology, Deepfake attacks manipulate audio/video streams (AVS) to mimic any targeted person or scenario. Deepfake attacks are highly disturbing, and the misinformation can mislead the public, raising further challenges in policy, technical, social, and legal aspects. Electrical Network Frequency (ENF) signals embedded in AVS data are promising to be utilized as fingerprints to authenticate digital media and timely detect deepfaked audio or video. Meanwhile, the success of ENF-based deepfake detection approaches will be forfeited if attackers can create false ENF fingerprints to fool the detector. In this paper, a thorough experimental study validates the robustness of ENF signals as a fingerprint for digital media authentication. Taking statistical, supervised learning, and deep learning approaches, this work shows that it is infeasible to forecast the future ENF signals based on historical records. While strict theoretical proof is yet to be done, this work experimentally verifies ENF signals as a reliable fingerprint to authenticate digital media.

While facial recognition (FR) has been widely used by businesses and governments for various purposes, it gives rise to privacy concerns once the consent of users is not handled properly. Hence, researchers have proposed methods to evade FR technology by attaching adversarial perturbations to user profile images. Nonetheless, image denoising-based methods have been proposed to increase the model robustness over adversarial examples. This paper investigates the impact of transformations on adversarial facial images. In particular, a simple but effective framework, TaD (Transformations as Denoising), is proposed to remove possible adversarial perturbations from user images generated by popular FR privacy protection frameworks. Extensive evaluations show the reliability of Fawkes and LowKey with various simple transformations. Experimental results indicate that simple transformations can impact the protection performance, and the choice of DNN-based facial feature extractors can enhance the robustness of facial images with adversarial perturbations. The experimental results also demonstrate strengths and weaknesses of FR methods and give suggestions for further improvements of privacy safeguard tools.

The Internet of Things (IoT) encompasses a near-incalculable collection of dispersed and embedded computing devices acting as sensors and actuators, generating data at an incredible scale. However, a lack of coherency and cross-compatibility in IoT deployments has lead to increasing redundancy and waste of resources. To combat this, various concepts have been proposed for an open IoT search engine (IoT-SE) that serves human and machine users. Invariably, the IoT-SE envisions distributed query retrieval to handle massive volumes of devices and data. Incorporating the massively heterogeneous protocols and properties of devices deployed, the search of such a system for timely and pertinent data is massively challenging, to provide useful knowledge and service for IoT systems. Moreover, enabling and maintaining security and privacy in an IoT-SE is likewise a prodigious task, as end users, IoT devices, and the search system itself, have different protocols and requirements. To this end, a study of security issues in IoT search is conducted to outline the challenges ahead, and a case study to resolve practical security vulnerabilities in an IoT-SE system is carried out. The pertinent issues of security in an IoT-SE system are reviewed. Particularly: 1) a taxonomy is detailed for IoT-SE security issues; 2) the vulnerabilities of machine learning (ML) models in the IoT-SE are considered; and 3) defensive mechanisms are presented for securing IoT Search. A case study is carried out to implement basic security features in the IoT search, addressing the risk of false queries through the design of ML-based solutions. Finally, a roadmap for future research is provided, including the security and privacy for IoT systems connected to the IoT-SE, distributed edge computing in IoT-SE, privacy-preserving data markets in IoT-SE, and distributed ML in IoT-SE.

The fast development of the Internet of Things (IoT) and deep learning enables learning useful patterns from the massive amount of collected data with sporadic nodes in IoT systems. Federated learning has received increasing attention in distributed machine learning where only intermediate parameters are exchanged with training samples that resided at local nodes. Nonetheless, most of the existing federated learning schemes assume a homogeneous distribution of data. The assumption, however, does not apply to IoT systems because of the heterogeneity of the IoT architecture. The nonindependent and identical distribution (non-IID) property in data volume and statistical distribution of IoT nodes can impact the performance of an aggregated global model that fits all nodes. Existing federated learning solutions for non-IID data sets either have to train additional models or require extra data exchange to check the node distribution. However, due to resource constraints in IoT systems, these approaches will increase the burden on limited computation capacity and cause network overhead. To address the issue, in this article, a novel weight-similarity-based client clustering (WSCC) approach is proposed, in which clients are split into different groups based on their data set distributions. An affinity-propagation-based method with the cosine distance of the client’s weight parameters is designed to iteratively and automatically determine dynamic clusters. The proposed approach is ideal for IoT systems since there are no auxiliary models and extra data transmissions are needed. Through the theoretical convergence analysis and empirical results, we show that our proposed WSCC scheme outperforms the representative federated learning schemes under different non-IID settings, achieving up to 20% improvements in accuracy.

Modern video conferencing technologies provide state-of-the-art end-to-end encryption models but do not verify the authenticity of the media broadcast, where the verification of the media is left to the end-users. A perpetrator can forge the video or audio streams using replay attacks or deepfake attacks to manipulate the real-time perception of transcribed events. Leveraging Electrical Network Frequency (ENF) signals as an environmental fingerprint, this paper proposes a distributed consensus network-based audio authentication scheme named DeFake - Decentralized ENF-consensus based deepFake detection, which detects multimedia manipulations in real-time. Since the fluctuations in an ENF signal are of a distributed and random nature, a novel Proof-of-ENF (PoENF) algorithm can guarantee byzantine resistant deepfakes detection on audio streams with minimal computational resources. By utilizing audio conferencing or audio editing applications as the frontend software service, the DeFake solution can effectively and efficiently verify the authenticity of the recorded video clip.

Rapid advances in the Internet of Video Things (IoVT) deployment in modern smart cities has enabled secure infrastructures with minimal human intervention. However, attacks on audio-video inputs affect the reliability of large-scale multimedia surveillance systems as attackers are able to manipulate the perception of live events. For example, Deepfake audio/video attacks and frame duplication attacks can cause significant security breaches. This paper proposes a Lightweight Environmental Fingerprint Consensus based detection of compromised smart cameras in edge surveillance systems (LEFC). LEFC is a partial decentralized authentication mechanism that leverages Electrical Network Frequency (ENF) as an environmental fingerprint and distributed ledger technology (DLT). An ENF signal carries randomly fluctuating spatio-temporal signatures, which enable digital media authentication. With the proposed DLT consensus mechanism named Proof-of-ENF (PoENF) as a backbone, LEFC can estimate and authenticate the media recording and detect byzantine nodes controlled by the perpetrator. The experimental evaluation shows feasibility and effectiveness of proposed LEFC scheme under a distributed byzantine network environment.

Wildfires are one of the costliest and deadliest natural disasters in the US, causing damage to millions of hectares of forest resources and threatening the lives of people and animals. Of particular importance are risks to firefighters and operational forces, which highlights the need for leveraging technology to minimize danger to people and property. FLAME (Fire Luminosity Airborne-based Machine learning Evaluation) offers a dataset of aerial images of fires along with methods for fire detection and segmentation which can help firefighters and researchers to develop optimal fire management strategies. This paper provides a fire image dataset collected by drones during a prescribed burning piled detritus in an Arizona pine forest. The dataset includes video recordings and thermal heatmaps captured by infrared cameras. The captured videos and images are annotated, and labeled frame-wise to help researchers easily apply their fire detection and modeling algorithms. The paper also highlights solutions to two machine learning problems: (1) Binary classification of video frames based on the presence [and absence] of fire flames. An Artificial Neural Network (ANN) method is developed that achieved a 76% classification accuracy. (2) Fire detection using segmentation methods to precisely determine fire borders. A deep learning method is designed based on the U-Net up-sampling and down-sampling approach to extract a fire mask from the video frames. Our FLAME method approached a precision of 92%, and recall of 84%. Future research will expand the technique for free burning broadcast fire using thermal images.

Situation awareness (SAW) is essential for many mission-critical applications. However, SAW is challenging when trying to immediately identify objects of interest or focus on suspicious activities from thousands of video frames. This article develops a queryable system to instantly select interesting content. While face recognition technology is mature, in many scenarios, such as public safety monitoring, the features of objects of interest may be much more complicated than face features. In addition, human operators may not be always able to provide a descriptive, simple, and accurate query. Actually, it is more often that there are only rough, general descriptions of certain suspicious objects or accidents. This article proposes interactive video surveillance as an edge service (I-ViSE) based on unsupervised feature queries. Adopting unsupervised methods that do not reveal any private information, the I-ViSE scheme utilizes general features of a human body and color of clothes. An I-ViSE prototype is built following the edge-fog computing paradigm and the experimental results verified the I-ViSE scheme meets the design goal of scene recognition and target analysis in less than 2 s.

Context enhancement is critical for the environmental perception in night vision applications, especially for the dark night situation without sufficient illumination. In this article, we propose a thermal image translation method, which can translate thermal/infrared (IR) images into color visible (VI) images, called IR2VI. The IR2VI consists of two cascaded steps: translation from nighttime thermal IR images to gray-scale visible images (GVI), which is called IR-GVI; and the translation from GVI to color visible images (CVI), which is known as GVI-CVI in this article. For the first step, we develop the Texture-Net, a novel unsupervised image translation neural network based on generative adversarial networks. Texture-Net can learn the intrinsic characteristics from the GVI and integrate them into the IR image. In comparison with the state-of-the-art unsupervised image translation methods, the proposed Texture-Net is able to address some common challenges, e.g., incorrect mapping and lack of fine details, with a structure connection module and a region-of-interest focal loss. For the second step, we investigated the state-of-the-art gray-scale image colorization methods and integrate the deep convolutional neural network into the IR2VI framework. The results of the comprehensive evaluation experiments demonstrate the effectiveness of the proposed IR2VI image translation method. This solution will contribute to the environmental perception and understanding in varied night vision applications.

The International Society of Information Fusion (ISIF) Stone Soup project seeks to bring together advances in target tracking through an open-source repository of software libraries. Additionally, the ISIF uncertainty reasoning working group provides an open-source ontology. This paper seeks to demonstrate the correspondence between the open source tracking repository and the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. For example, many target tracking challenge problems propose a scenario for data fusion techniques to solve, from which various performance metrics are considered for evaluation. The Stone Soup framework has the MetricGenerator class and the URREF has the accuracy class. The example presented in the paper utilizes the cubature Kalman filter to determine the impact of corrupted measurements on the track accuracy as an instance of the Stone Soup and URREF metrics.

There have been many advances in image fusion to support multi-modal, multi-perspective, and multi-focal day-night robust surveillance. Contextual analysis of multi-sensor exploitation supports many information fusion systems for target tracking, situation awareness, and scene understanding. This paper highlights an example for electro-optical and infrared image fusion analysis and colorization in support of simultaneous tracking and identification for physics-based and human-derived information fusion (PHIF). The results provide an ongoing discussion and analysis of the importance of context as both a source to guide information fusion systems (e.g., sensor mode selection), but also the need to be able to provide users with robust systems over context changes (e.g., variations in illumination).

This paper considers a novel formulation of inverse reinforcement learning with behavioral economics constraints to address inverse sequential hypothesis testing (SHT) in Bayesian agents. The aim is to estimate the detection costs by observing the actions of the sequential hypothesis detector. Our methodology involves Bayesian revealed preferences from microeconomics and rational inattention from behavioral economics. First, we show that Bayesian agents optimally performing SHT are rationally inattentive utility maximizers. Using established results in Bayesian revealed preferences, we outline a feasibility test for a data analyst observing the Bayesian agents to estimate their detection costs. Numerical examples illustrate the performance of the inverse sequential hypothesis testing algorithm.

Detecting objects in aerial images is challenging for at least two reasons: (1) target objects like pedestrians are very small in pixels, making them hardly distinguished from surrounding background; and (2) targets are in general sparsely and non-uniformly distributed, making the detection very inefficient. In this paper, we address both issues inspired by observing that these targets are often clustered. In particular, we propose a Clustered Detection (ClusDet) network that unifies object clustering and detection in an end-to-end framework. The key components in ClusDet include a cluster proposal sub-network (CPNet), a scale estimation sub-network (ScaleNet), and a dedicated detection network (DetecNet). Given an input image, CPNet produces object cluster regions and ScaleNet estimates object scales for these regions. Then, each scale-normalized cluster region is fed into DetecNet for object detection. ClusDet has several advantages over previous solutions: (1) it greatly reduces the number of chips for final object detection and hence achieves high running time efficiency, (2) the cluster-based scale estimation is more accurate than previously used single-object based ones, hence effectively improves the detection for small objects, and (3) the final DetecNet is dedicated for clustered regions and implicitly models the prior context information so as to boost detection accuracy. The proposed method is tested on three popular aerial image datasets including VisDrone, UAVDT and DOTA. In all experiments, ClusDet achieves promising performance in comparison with state-of-the-art detectors.

With ubiquitous data acquired from sensors, there is an ever increasing ability to abstract content from the combination of physics-based and human-derived information fusion (PHIF). The advancement of PHIF tools include graphical information fusion methods, target tracking techniques, and natural language understanding. Current discussions revolve around dynamic data-driven applications systems (DDDAS) which seeks to leverage high-dimensional modeling with real-time physical systems. An example of a model includes a learned dictionary that can be leveraged as information queries. In this paper, we discuss the DDDAS paradigm of sensor measurements, information processing, environmental modeling, and software implementation to deliver content for PHIF systems. Experimental results demonstrate the DDDAS-based Live Video Computing DataBase Modeling Systems (LVC-DBMS) approach affording data discovery and query-based flexibility for awareness to provide narratives of unknown situations.

Information fusion includes the integration of data for situational understanding. As a situation unfolds, maintaining awareness depends on diverse collections of data. In complex and dynamic scenarios, human operators face the difficult task of choosing which data to collect next. Hence, there is a need for multilayered fusion processes that exploit multiple models and levels of abstraction for understanding and sense-making Data collection has its roots in sensor management; however, there is an analogous need for data management - such as the incorporation of public domain data. Mature sensor management includes methods to utilize platform, sensor, and scene modeling so as to guide the user for future data collection. Additionally, these physics-based models could be a method to guide human-derived information models. Using the Data Fusion Information Group Model (DFIG), we develop an equivalent method for diffusion control. This paper focuses on recent techniques in statistical relational learning (SRL), Markov logic networks (MLN), and ontologies to support the control diffusion of data sensing to answer user queries.

Trajectory association and clustering are two key problems in motion analysis. While association links the points of interest to form trajectories, clustering discovers motion patterns of these trajectories and group them into clusters. Despite mutually related, the two problems have been typically studied separately in the literature. In this letter, we formulate them as a unified optimization problem and take the advantage of high-order information to capture the interrelations for Motion Segmentation by Trajectory Association and Clustering (MSTAC). To solve this unified problem, we propose an alternating optimization strategy to improve the association and clustering in each iteration. Specifically, a tensor-based multidimensional assignment method with high-order motion context information is proposed for trajectory association; and a minimum cost multicut-based trajectory clustering method is introduced for trajectory clustering. While the association process provides incomplete trajectories to clustering, the clustering method presents high-order context information to improve the performance of association; and thus they benefit from each other. Experiments on the Hopkins 155 dataset and a realistic airport sequence demonstrate that the proposed MSTAC framework obtains high accuracy on both trajectory association and clustering.

This paper presents a comparative analysis of performances of two types of multi-target tracking algorithms: 1) the Joint Probabilistic Data Association Filter (JPDAF), and 2) classical Kalman Filter based algorithms for multi-target tracking improved with Quality Assessment of Data Association (QADA) method using optimal data association. The evaluation is based on Monte Carlo simulations for difficult maneuvering multiple-target tracking (MTT) problems in clutter.

The paper introduces a novel approach to an estimator design, the cooperative filter design, for state estimation of nonlinear systems. The approach is based on the idea of combining estimates of several different approximate (and thus sub-optimal) nonlinear filters, which are configured to perform the same task. Within the concept, two strategies are proposed, namely the cooperative estimation and cooperative monitoring. The strategies have the potential of an improvement of the estimation performance in terms of accuracy and consistency, which was confirmed by a numerical illustration.

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.

Skin detection from images, typically used as a preprocessing step, has a wide range of applications such as dermatology diagnostics, human computer interaction designs, and etc. It is a challenging problem due to many factors such as variation in pigment melanin, uneven illumination, and differences in ethnicity geographics. Besides, age and gender introduce additional difficulties to the detection process. It is hard to determine whether a single pixel is skin or nonskin without considering the context. An efficient traditional hand-engineered skin color detection algorithm requires extensive work by domain experts. Recently, deep learning algorithms, especially convolutional neural networks (CNNs), have achieved great success in pixel-wise labeling tasks. However, CNN-based architectures are not sufficient for modeling the relationship between pixels and their neighbors. In this letter, we integrate recurrent neural networks (RNNs) layers into the fully convolutional neural networks (FCNs), and develop an end-to-end network for human skin detection. In particular, FCN layers capture generic local features, while RNN layers model the semantic contextual dependencies in images. Experimental results on the COMPAQ and ECU skin datasets validate the effectiveness of the proposed approach, where RNN layers enhance the discriminative power of skin detection in complex background situations.

Most Wide Area Motion Imagery (WAMI) based trackers use motion based cueing for detecting and tracking moving objects. The results are very high false alarm rates in urban environments with tall structures due to parallax effects. This paper proposes an accurate moving object detection method using a precise orthorectification approach for ground stabilization combined with accurate multiview depth maps to reduce the number of false positives induced by parallax effects by 90 percent. Proposed hybrid moving vehicle detection approach for large scale aerial urban imagery is based on fusion of motion detection mask obtained from median-based background subtraction and tall structures height mask provided by image depth map information. Using buildings mask enables us to improve the object level detection accuracy in terms of F-measure by 57 percent from 22.2% to 79.2%.

Abstract For spectrum sensing, energy detection has the advantages of low complexity, rapid analysis, and requires no knowledge of the transmission signal, which makes it suitable for a wide range of applications. However, under low signal‐to‐noise ratio conditions, the required window length (or the time‐bandwidth product) for energy detection to achieve a desired detection performance is large. In addition, conventional energy detection assumes that the detection tests are independent, that is, there is no overlap between individual detection tests. These properties significantly reduce the detection speed when energy detection is used for the continuous monitoring over a communication channel for the detection of signal transmission activities. In this paper, we propose a sliding window detection analysis with overlap among multiple tests. Algorithms for effective performance analysis of the proposed sliding window energy detection are proposed. The impact of window length on distribution of detection time is investigated. Simulation results on the proposed sliding window energy detection are also compared with the theoretically predicted and conventional energy detection performance estimates. Copyright © 2015 John Wiley & Sons, Ltd.

In this paper we present probabilistic graph fusion algorithms to support information fusion and reasoning over multi-source text media. Our methods resolve misinformation by combining knowledge similarity analysis and conflict identification with source characterization. For experimental purposes, we used the dataset of the articles about current military conflict in Eastern Ukraine. We show that automated knowledge fusion and conflict detection is feasible and high accuracy of detection can be obtained. However, to correctly classify mismatched knowledge fragments as misinformation versus additionally reported facts, the knowledge reliability and credibility must be assessed. Since the true knowledge must be reported by many reliable sources, we compute knowledge frequency and source reliability by incorporating knowledge provenance and analyzing historical consistency between the knowledge reported by the sources in our dataset.

While color information is known to provide rich discriminative clues for visual inference, most modern visual trackers limit themselves to the grayscale realm. Despite recent efforts to integrate color in tracking, there is a lack of comprehensive understanding of the role color information can play. In this paper, we attack this problem by conducting a systematic study from both the algorithm and benchmark perspectives. On the algorithm side, we comprehensively encode 10 chromatic models into 16 carefully selected state-of-the-art visual trackers. On the benchmark side, we compile a large set of 128 color sequences with ground truth and challenge factor annotations (e.g., occlusion). A thorough evaluation is conducted by running all the color-encoded trackers, together with two recently proposed color trackers. A further validation is conducted on an RGBD tracking benchmark. The results clearly show the benefit of encoding color information for tracking. We also perform detailed analysis on several issues, including the behavior of various combinations between color model and visual tracker, the degree of difficulty of each sequence for tracking, and how different challenge factors affect the tracking performance. We expect the study to provide the guidance, motivation, and benchmark for future work on encoding color in visual tracking.

In recent years, quantum-based techniques have attracted significant research attention because of its unique advantages on satellite communications, especially for security problem. Security guarantee is one of the most important requirements in QoS-driven 3D satellite networks. Quantum key distribution (QKD) is a methodology for generating and distributing random encryption keys using the principles of quantum physics, which enables two distant communications parties to securely communicate in a way that cannot be eavesdropped on without being detected. Although the QKD method can ensure the absolute security transmission over 3D satellite networks, it imposes many new implementation challenges due to the various limitations on quantum communication over long distances via 3D free space, including quantum channel attenuation, photon-state disruption and vulnerability to noise/interference, laser-beam widening, and constrained security-key generation rate. These problems get even more challenging when QoS provisioning is required for the applications over the 3D satellite networks. To overcome the aforementioned difficulties, we propose the framework to efficiently implement the QKD for security guarantees over quantum-repeater-based QoS-driven 3D satellite networks. First, we develop the quantum-repeater-based QKD satellite network architecture. Then, we design the quantum repeater including the purification scheduling algorithm and the optimal QoS-based repeating-router selection scheme in quantum-repeater-based QKD satellite networks. Finally, the obtained simulations evaluation validate and evaluate our proposed algorithms and schemes.

Face de-identification, the process of preventing a person' identity from being connected with personal information, is an important privacy protection tool in multimedia data processing. With the advance of face detection algorithms, a natural solution is to blur or block facial regions in visual data so as to obscure identity information. Such solutions however often destroy privacy-insensitive information and hence limit the data utility, e.g., gender and age information. In this paper we address the de-identification problem by proposing a simple yet effective framework, named GARP-Face, that balances utility preservation in face deidentification. In particular, we use modern facial analysis technologies to determine the Gender, Age, and Race attributes of facial images, and Preserving these attributes by seeking corresponding representatives constructed through a gallery dataset. We evaluate the proposed approach using the MORPH dataset in comparison with several state-of-the-art face de-identification solutions. The results show that our method outperforms previous solutions in preserving data utility while achieving similar degree of privacy protection.

A typical method to obtain valuable information is to extract the sentiment or opinion from a message. Machine learning technologies are widely used in sentiment classification because of their ability to “learn” from the training dataset to predict or support decision making with relatively high accuracy. However, when the dataset is large, some algorithms might not scale up well. In this paper, we aim to evaluate the scalability of Naïve Bayes classifier (NBC) in large datasets. Instead of using a standard library (e.g., Mahout), we implemented NBC to achieve fine-grain control of the analysis procedure. A Big Data analyzing system is also design for this study. The result is encouraging in that the accuracy of NBC is improved and approaches 82% when the dataset size increases. We have demonstrated that NBC is able to scale up to analyze the sentiment of millions movie reviews with increasing throughput.

Tracking multiple vehicles in wide area traffic scenes is challenging due to high target density, severe similar target ambiguity, and low frame rate. In this paper, we propose a novel spatio-temporal context model, named maximum consistency context (MCC), to leverage the discriminative power and robustness in the scenario. For a candidate association, its MCC is defined as the most consistent association in its neighborhood. Such a maximum selection picks the reliable neighborhood context information while filtering out noisy distraction. We tested the proposed context modeling on multi-target tracking using three challenging wide area motion sequences. Both quantitative and qualitative results show clearly the effectiveness of MCC, in comparison with algorithms that use no context and standard spatial context respectively.

Network-based crime has been increasing in both extent and severity and network-based forensics encapsulates an essential part of legal surveillance. A key network forensics tool is trace back, which can be used to identify true sources of suspects. Both accuracy and secrecy are essential attributes of a successful forensic trace back. In this paper, we present a class of hopping based spread-spectrum techniques for forensic trace back, which fully use the benefits of the spread spectrum approach and preserves a greater degree of secrecy. Our proposed techniques, including Code Hopping-Direct Sequence Spread Spectrum (CHDSSS), Frequency Hopping-Direct Sequence Spread Spectrum (FH-DSSS), and Time Hopping-Spread Spectrum (TH-DSSS), operate to randomize the effects of marking traffic through both the time and frequency domains. Our simulation study validates these techniques in terms of accuracy and secrecy.

Recently, sparse representation has been applied to visual tracking to find the target with the minimum reconstruction error from a target template subspace. Though effective, these L1 trackers require high computational costs due to numerous calculations for l1 minimization. In addition, the inherent occlusion insensitivity of the l1 minimization has not been fully characterized. In this paper, we propose an efficient L1 tracker, named bounded particle resampling (BPR)-L1 tracker, with a minimum error bound and occlusion detection. First, the minimum error bound is calculated from a linear least squares equation and serves as a guide for particle resampling in a particle filter (PF) framework. Most of the insignificant samples are removed before solving the computationally expensive l1 minimization in a two-step testing. The first step, named τ testing, compares the sample observation likelihood to an ordered set of thresholds to remove insignificant samples without loss of resampling precision. The second step, named max testing, identifies the largest sample probability relative to the target to further remove insignificant samples without altering the tracking result of the current frame. Though sacrificing minimal precision during resampling, max testing achieves significant speed up on top of τ testing. The BPR-L1 technique can also be beneficial to other trackers that have minimum error bounds in a PF framework, especially for trackers based on sparse representations. After the error-bound calculation, BPR-L1 performs occlusion detection by investigating the trivial coefficients in the l1 minimization. These coefficients, by design, contain rich information about image corruptions, including occlusion. Detected occlusions are then used to enhance the template updating. For evaluation, we conduct experiments on three video applications: biometrics (head movement, hand holding object, singers on stage), pedestrians (urban travel, hallway monitoring), and cars in traffic (wide area motion imagery, ground-mounted perspectives). The proposed BPR-L1 method demonstrates an excellent performance as compared with nine state-of-the-art trackers on eleven challenging benchmark sequences.

Comparison of image processing techniques is critically important in deciding which algorithm, method, or metric to use for enhanced image assessment. Image fusion is a popular choice for various image enhancement applications such as overlay of two image products, refinement of image resolutions for alignment, and image combination for feature extraction and target recognition. Since image fusion is used in many geospatial and night vision applications, it is important to understand these techniques and provide a comparative study of the methods. In this paper, we conduct a comparative study on 12 selected image fusion metrics over six multiresolution image fusion algorithms for two different fusion schemes and input images with distortion. The analysis can be applied to different image combination algorithms, image processing methods, and over a different choice of metrics that are of use to an image processing expert. The paper relates the results to an image quality measurement based on power spectrum and correlation analysis and serves as a summary of many contemporary techniques for objective assessment of image fusion algorithms.

The recently proposed covariance region descriptor has been proven robust and versatile for a modest computational cost. The covariance matrix enables efficient fusion of different types of features, where the spatial and statistical properties, as well as their correlation, are characterized. The similarity between two covariance descriptors is measured on Riemannian manifolds. Based on the same metric but with a probabilistic framework, we propose a novel tracking approach on Riemannian manifolds with a novel incremental covariance tensor learning (ICTL). To address the appearance variations, ICTL incrementally learns a low-dimensional covariance tensor representation and efficiently adapts online to appearance changes of the target with only O(1) computational complexity, resulting in a real-time performance. The covariance-based representation and the ICTL are then combined with the particle filter framework to allow better handling of background clutter, as well as the temporary occlusions. We test the proposed probabilistic ICTL tracker on numerous benchmark sequences involving different types of challenges including occlusions and variations in illumination, scale, and pose. The proposed approach demonstrates excellent real-time performance, both qualitatively and quantitatively, in comparison with several previously proposed trackers.

Recently, sparse representation has been applied to visual tracking to find the target with the minimum reconstruction error from the target template subspace. Though effective, these L1 trackers require high computational costs due to numerous calculations for ℓ1minimization. In addition, the inherent occlusion insensitivity of the ℓ1minimization has not been fully utilized. In this paper, we propose an efficient L1 tracker with minimum error bound and occlusion detection which we call Bounded Particle Resampling (BPR)-L1 tracker. First, the minimum error bound is quickly calculated from a linear least squares equation, and serves as a guide for particle resampling in a particle filter framework. Without loss of precision during resampling, most insignificant samples are removed before solving the computationally expensive ℓ1minimization function. The BPR technique enables us to speed up the L1 tracker without sacrificing accuracy. Second, we perform occlusion detection by investigating the trivial coefficients in the ℓ1minimization. These coefficients, by design, contain rich information about image corruptions including occlusion. Detected occlusions enhance the template updates to effectively reduce the drifting problem. The proposed method shows good performance as compared with several state-of-the-art trackers on challenging benchmark sequences.

Recent data collections in video and signals collection such as Electro-optical/Infrared video, synthetic aperture radar (SAR), and Hyperspectral (HSI) data afford persistent surveillance. As the rate of the imagery (or other products such as 3D terrain modeling) becomes available, there is a greater need for data exploitation to support and augment user needs for situational awareness. The presentation will focus on (a) emerging technologies, (b) tools and processing needs for the user, and (c) developments for persistent surveillance performance analysis. Standard methods of image processing and imagery collection have been accelerated based on the availability of emerging products such as Unmanned Aerial Vehicles (UAV) technology, traffic monitoring systems, and security products. Each of these systems require coordination over varying resolutions, distributed networks, and graphical interfaces. The sensor coordination in the system network enables surveillance over persistent coverage. Persistent surveillance implies that the information processed is to report the results to a user. Surveillance systems can report situational awareness results of target tracks and identifications, anomalies, and target behaviors. Focus should be placed not only on the processing, but include an interactive user to control the collections, determine priorities for the video processing, and support exploration analysis. Image processing, especially video, requires novel techniques to direct collections, process varying targets in complicated terrains, and efforts to evaluate the success. Evaluation includes assessing the performance over varying operation conditions (i.e. sensors, targets, and environments) to include obscurations, compression, and non-continuous data streams. Persistent surveillance coverage requires new techniques in image exploitation to include: (1) fusion with signals and database intelligence, (2) contextual modeling, and (3) tools for user interactive analysis.

To address the challenges on non-cooperative long-distance human identification and verification, we propose an innovative cost-efficient system for automatic long-range biometric recognition of non-cooperative individuals in 24/7 operations. The system has three cameras. One is a wide field of view (WFOV) CCD video camera with an Infrared (IR) filter and powerful IR illuminators for human scan in a wide area at a long distance. The other two cameras are high resolution video cameras with narrow field of view (NFOV) and an IR filter & illuminators, mounted on a pan-tilt-unit (PTU) to capture the frontal view of human face and iris respectively. Once the frontal views of moving individuals are captured by the NFOV cameras, the face/iris models will be extracted and classified by the state-of-the-art face/iris recognizers. The hardware of the biometric system also includes one FPGA, three DSP processors, and one Zigbee module for fast bio-data analysis and wireless data transmission.

Many communication systems require a two-way, or three-way handshaking process to improve their dependability & authenticity in order to achieve a more successful operation. In this paper, we present a new two-way handshaking reliability model based upon threshold-based cryptography systems. Such systems require a two-way handshaking process to i) establish a group of participated servers in the first handshaking process, and ii) calculate a cipher with successfully connected servers collaboratively in the second handshaking process. When the servers are attempted, each server has three known connection probabilities in the following three states: i) successful, ii) breakdown, and iii) congested. These connection probabilities are unchanged in both handshaking processes. During the first handshaking process, we establish connections that more than servers are willing to participate. For the second handshaking process, the system becomes successful as soon as we can connect these servers successfully again. Because we need to connect servers successfully in the second handshaking process, we would rather connect additional servers besides the servers required to be connected successfully in the first handshaking process. This preference will minimize the chance that the system breaks down when fewer than servers can be reconnected successfully in the second handshaking process. We refer to this system as a Two-Way Handshaking Circular Sequential-out-of-Congestion (TWHCSknC) system. In this paper, we derived analytical formulas for the system's successful probability & average stop length, and we showed that the TWHCSknC system is a communication system with an efficient two-way handshaking process.

Distributed image steganography (DIS) [8] is a new method of concealing secret information in several host images, leaving smaller traces than conventional steganographic techniques, and requiring a collection of affected images for secret information retrieval. Fusion system designs of the future will require enhanced security measures for distributed data communication. DIS, compared to other conventional steganographic techniques, can improve security and information hiding capacity because DIS leaves reduced signatures of hidden information in host images. The open literature does not offer effective detection methods and countermeasures for DIS, indicating that it can be potentially usable to criminals for unchallenged covert communication over the Internet and fusion architectures. In this paper, we explore a new information extraction method for both detecting and reversing DIS method by considering images as pseudo-random processes. The key idea is to estimate secret image as a random process, which is corrupted by a noise source (i.e. host image). The secret images may be nonlinear, non-Gaussian and nonstationary in nature, and can be disclosed by using some estimation techniques such as Kalman filtering. Our proposed method demonstrates great promise to reveal a secret image. Consequently, it is useful for intelligence gathering and information extraction in steganographic - images produced by DIS.

This paper presents a new probabilistic approach to determine survivability of reconfigurable systems as a system-level performance metric in an operational environment. In contrast to known methods of estimating survivability in terms of susceptibility and vulnerability, the proposed method (1) includes directional threats on various subsystems into the analysis and (2) provides a framework for operational information fusion processes to better sustain unpredictable or hostile environmental disturbances. In this paper, we distinguish the survivability and reliability metrics, where we demonstrate the importance of survivability metric in a dynamic information fusion process for an operational environment. We present our main result using a piping system of fluid flow; however, the concept easily extends to other flow systems, such as power networks, computer communication networks, and military reconfigurable information systems, etc. Survivability of these large scale reconfigurable networks depend on their capability of assessing directional threats, situation awareness, and their ability to dynamically adapt to new configurations. The proposed survivability method embedded in an information fusion environment can be used for real time dynamic reconfiguration of large scale systems, optimization and routing of data and information, and detect and mitigate hardware and software threats.

Sensor Resource Management (or process refinement) is a element of any information fusion system. Common Level 4 sensor management (SM) inter-relations to Level 1 target tracking and identification have been developed in the literature. During Fusion06, a panel discussion was held to explore the challenges and issues pertaining to the interaction between SM and situation and threat assessment. This paper summarizes the key tenants of the discussion to filter vast experiences of the invited panel experts. The common themes were: (1) Addressing the user in system management / control, (2) Determining a standard set of metrics for optimization, (3) Optimizing / evaluating fusion systems to deliver timely information needs, (4) Dynamic updating for planning mission time-horizons, (5) Joint optimization of objective functions at all levels (6) L2/3 situation entity definitions for knowledge discovery, modeling, and information projection (7) Addressing constraints for resource planning and scheduling

Summary form only given. Despite significant recent progress in automatic target exploitation (ATE) and recognition (ATR), current ATE systems do not meet the requirements of modern battlefield environments. Next generation ATE systems must actively manage sensor resources, aggregate sensed information across multiple platforms and diverse signaling modalities, and adapt to increasingly agile adversaries and operating conditions. Thus, the fundamental research challenge is to develop an integrated systems theory that jointly treats information fusion, control, and adaptation using multiple, dynamic multimodal sensor platforms in resource constrained environments.

Recent developments in image fusion have produced a variety of approaches like image overlay, image sharpening, and image cueing through pixel, feature, or region/shape combinations. The applicability of these approaches and techniques differ on the image content, contextual information, and generalized metrics of image fusion gain. An image fusion gain can be assessed relative to information gain or entropy reduction. In this paper, we are interested in exploring the techniques and data available with the image fusion toolbox (from www.imagefusion.org) to assess the use of relative entropy analysis for metric evaluation, image fusion gain calculations, and assessment of fused images as templates for automatic target recognition. Examples are demonstrated for medical (PET/MRI), (CT/MRI) and environment (visible/infrared) examples. A mutual information measure of the image fusion quality can be an effective tool to characterize image terrain, content, and contextual information to cue higher-level fusion algorithms.

This paper compares different defense strategies against various attacks utilizing a dynamic game theoretic data fusion framework for cyber network defense. In our game theoretic framework, Alerts generated by Intrusion Detection Sensors (IDSs) or Intrusion Prevention Sensors (IPSs) are fed into the data refinement (Level 0) and object assessment (L1) data fusion components. High-level situation/threat assessment (L2/L3) data fusion based on Markov game model and Hierarchical Entity Aggregation (HEA) are proposed to refine the primitive prediction generated by adaptive feature/pattern recognition and capture new unknown features. A Markov (Stochastic) game method is used to estimate the belief of each possible cyber attack pattern. Game theory captures the nature of cyber conflicts: determination of the attacking-force strategies is tightly coupled to determination of the defense-force strategies and vice versa. A software tool is developed to demonstrate and compare the performance of different defense strategies used in game theoretic high level information fusion for cyber network defense situations and a simulation example shows the enhanced understating of cyber-network defense.

This paper is concerned with tracking of ground targets on roads and investigates possible ways to improve target state estimation via fusing a target’s track with information about a road along which the target is believed to be traveling. A target track is estimated by a surveillance radar whereas a digital map provides the road network of a region under surveillance. When the information about roads is as accurate as (or even better than) radar measurements, it is desired naturally to incorporate such information (fusion) into target state estimation. In this paper, roads are modeled with analytic functions and its fusion with a target track is cast as linear or nonlinear state constraints in an optimization procedure. The constrained optimization is then solved with the Lagrangian multiplier, leading to a closed-form solution for linear constraints and an iterative solution for nonlinear constraints. Geometric interpretations of the solutions are provided for simple cases. Computer simulation results are presented to illustrate the algorithms.

Tracking maneuvering targets presents a great challenge to airborne surveillance radar signal processing and sensor systems management systems. Smears caused by an uncompensated maneuver (either translational or rotational) affect target identification (ID) with distorted target images. An unexpected maneuver introduces large position estimation errors to a tracker and in the worst case loss of track. On the other hand, a sensor manager relies upon an expected performance of a tracker to schedule its resources so as to maintain target ID/tracker performance. To aid a sensor management cost function, we present a simple target maneuver indicator (TMI) specifically for the operational condition of target maneuverability. It relates the slope of a target’s range-Doppler image to the underlying turn rate, if the target undergoes a maneuver. As an intermediate product of the range profile formation process, this approach provides an easy and quick indication of target maneuverability and, under favorable conditions, an estimate of such a maneuver (e.g., the turn rate), which can be incorporated into the tracking algorithm of the tracker..

Subsequent revisions to the Joint-Directors of Lab (JDL) model emphasize the differentiation between fusion (estimation) and sensor management (control). Two diverging groups include one pressing for fusion automation (JDL revisions) and one advocating the role of the user (user-fusion model). The center of debate is real-world delivery of fusion systems which requires presenting fusion results for knowledge representation (fusion estimation) and knowledge reasoning (control management). The purpose of the paper is to highlight the need of Users, with individual differences, facilitated by knowledge representations to reason about user situational awareness (SA). This paper includes: (1) Addressing the user in system management/control. (2) Assessing information quality (metrics) to support SA. (3) Evaluating Fusion systems to deliver user info needs, (4) Planning knowledge delivery for dynamic updating. (5) Designing SA interfaces to support user reasoning

Revisions to the JDL model by the current DFIG team (data fusion information group) include definitions for model usefulness that stressed various control functions of sensor, user, and mission (SUM) management. The purpose of the paper is to highlight issues and challenges to real world separation of control actions. This position paper highlights: 1) addressing the user in system management/control, 2) determining a standard set of metrics for optimization, 3) evaluating fusion systems to deliver timely info needs, 4) dynamic updating for planning mission time-horizons, 5) designing interfaces to support impact assessments

In [Simon and Chia, 2002], an analytic method was developed to incorporate linear state equality constraints into the Kalman filter. When the state constraint is nonlinear, linearization was employed to obtain an approximately linear constraint around the current state estimate. This linearized constrained Kalman filter is subject to approximation errors and may suffer from a lack of convergence. In this paper, we present a method that allows exact use of second-order nonlinear state constraints. It is based on a computational algorithm that iteratively finds the Lagrangian multiplier for the nonlinear constraints. The method therefore provides better approximation when higher order nonlinearities are encountered. Computer simulation results are presented to illustrate the algorithm

Biological information fusion results from the combination of sensed current and learned past information. The paper focuses on modeling visual cortex information fusion where a pulse coupled neural network simulates the linking procedure to segment related spatial information of the primary visual cortex and a belief filter to confirm target identity with movement information of the medial temporal cortex.