With their increase in performance, neural network architectures also become more complex, necessitating explainability. Therefore, many new and improved methods are currently emerging, which often generate so-called saliency maps in order to improve interpretability. Those methods are often evaluated by visual expectations, yet this typically leads towards a confirmation bias. Due to a lack of a general metric for explanation quality, non-accessible ground truth data about the model's reasoning and the large amount of involved assumptions, multiple works claim to find flaws in those methods. However, this often leads to unfair comparison metrics. Additionally, the complexity of most datasets (mostly images or text) is often so high, that approximating all possible explanations is not feasible. For those reasons, this paper introduces a test for saliency map evaluation: proposing controlled experiments based on all possible model reasonings over multiple simple logical datasets. Using the contained logical relationships, we aim to understand how different saliency methods handle information in different class discriminative scenarios (e.g., via complementary and redundant information). By introducing multiple new metrics, we analyse propositional logical patterns towards a non-informative attribution score baseline to find deviations of typical expectations. Our results show that saliency methods can encode classification relevant information into the ordering of saliency scores, resulting in unreliable interpretations.

A prominent type of explanation for neural networks are saliency/attribution-based approaches, which highlight the most relevant inputs. Here, recent works suggest sub-optimality of those methods and emphasize the challenge of evaluation. In this paper, we present a new dataset-relative baseline to assess the minimal expectations on saliency scores, leading towards a new any-order interpretation evaluation framework. Using the Global Coherence Representation (GCR), we propose the SimpleGCR as an implementation of this framework acting as a stable minimal performance baseline. It thus enables a reference point for comparing different explainability metrics. We evaluate our proposed approach by applying a set of current saliency methods on the univariate UCR UEA time series datasets, and demonstrate the sub-optimality of those methods in this context.

Transformer models have shown significant advances recently based on the general concept of Attention — to focus on specifically important and relevant parts of the input data. However, methods for enhancing their interpretability and explainability are still lacking. This is the problem which we tackle in this paper, to make Multi-Headed Attention more interpretable and explainable for time series classification. We present a method for constructing global coherence representations from Multi-Headed Attention of Transformer architectures. Accordingly, we present abstraction and interpretation methods, leading to intuitive visualizations of the respective attention patterns. We evaluate our proposed approach and the presented methods on several datasets demonstrating their efficacy.