The focus of this paper is on the generation of multimodal explanations for information fusion tasks performed on multimodal data. We propose that separating modal components in saliency map explanations provides users with a better understanding of how convolutional neural networks process multimodal data. We adapt established state-of-the-art explainability techniques to mid-level fusion networks in order to better understand (a) which modality of the input contributes most to a model's decision and (b) which parts of the input data are most relevant to that decision. Our method separates temporal from non-temporal information to allow a user to focus their attention on salient elements of the scene that are changing in multiple modalities. The work is experimentally tested on an activity recognition task using video and audio data. In view of the fact that explanations need to be tailored to the type of user in a User Fusion context, we focus on meeting explanation requirements for system creators and operators respectively.

Learning network representation has a variety of \napplications, such as network classification. Most existing work \nin this area focuses on static undirected networks and does not \naccount for presence of directed edges or temporal changes. \nFurthermore, most work focuses on node representations that \ndo poorly on tasks like network classification. In this paper, \nwe propose a novel network embedding methodology, gl2vec, \nfor network classification in both static and temporal directed \nnetworks. gl2vec constructs vectors for feature representation \nusing static or temporal network graphlet distributions and a \nnull model for comparing them against random graphs. We \ndemonstrate the efficacy and usability of gl2vec over existing \nstate-of-the-art methods on network classification tasks such as \nnetwork type classification and subgraph identification in several \nreal-world static and temporal directed networks. We argue that \ngl2vec provides additional network features that are not captured \nby state-of-the-art methods, which can significantly improve their \nclassification accuracy by up to 10% in real-world applications

In this paper we provide a critical analysis with metrics that will inform guidelines for designing distributed systems for Collective Situational Understanding (CSU). CSU requires both collective insight-i.e., accurate and deep understanding of a situation derived from uncertain and often sparse data and collective foresight-i.e., the ability to predict what will happen in the future. When it comes to complex scenarios, the need for a distributed CSU naturally emerges, as a single monolithic approach not only is unfeasible: it is also undesirable. We therefore propose a principled, critical analysis of AI techniques that can support specific tasks for CSU to derive guidelines for designing distributed systems for CSU.

Situational understanding (SU) requires a combination of insight - the ability to accurately perceive an existing situation - and foresight - the ability to anticipate how an existing situation may develop in the future. SU involves information fusion as well as model representation and inference. Commonly, heterogenous data sources must be exploited in the fusion process: often including both hard and soft data products. In a coalition context, data and processing resources will also be distributed and subjected to restrictions on information sharing. It will often be necessary for a human to be in the loop in SU processes, to provide key input and guidance, and to interpret outputs in a way that necessitates a degree of transparency in the processing: systems cannot be “black boxes”. In this paper, we characterize the Coalition Situational Understanding (CSU) problem in terms of fusion, temporal, distributed, and human requirements. There is currently significant interest in deep learning (DL) approaches for processing both hard and soft data. We analyze the state-of-the-art in DL in relation to these requirements for CSU, and identify areas where there is currently considerable promise, and key gaps.