Abstract Insights into the learned latent representations are imperative for verifying deep neural networks (DNNs) in critical computer vision (CV) tasks. Therefore, state-of-the-art supervised Concept-based eXplainable Artificial Intelligence (C-XAI) methods associate user-defined concepts like “car” each with a single vector in the DNN latent space (concept embedding vector). In the case of concept segmentation, these linearly separate between activation map pixels belonging to a concept and those belonging to background. Existing methods for concept segmentation, however, fall short of capturing implicitly learned sub-concepts (e.g., the DNN might split car into “proximate car” and “distant car”), and overlap of user-defined concepts (e.g., between “bus” and “truck”). In other words, they do not capture the full distribution of concept representatives in latent space. For the first time, this work shows that these simplifications are frequently broken and that distribution information can be particularly useful for understanding DNN-learned notions of sub-concepts, concept confusion, and concept outliers. To allow exploration of learned concept distributions, we propose a novel local concept analysis framework. Instead of optimizing a single global concept vector on the complete dataset, it generates a local concept embedding (LoCE) vector for each individual sample. We use the distribution formed by LoCEs to explore the latent concept distribution by fitting Gaussian mixture models (GMMs), hierarchical clustering, and concept-level information retrieval and outlier detection. Despite its context sensitivity, our method’s concept segmentation performance is competitive to global baselines. Analysis results are obtained on three datasets and six diverse vision DNN architectures, including vision transformers (ViTs). The code is available at https://github.com/continental/local-concept-embeddings .

Although clustering under constraints is a current research topic, a hierarchical setting, in which a hierarchy of clusters is the goal, is usually not considered. This paper tries to fill this gap by analyzing a scenario, where constraints are derived from a hierarchy that is partially known in advance. This scenario can be found, e.g., when structuring a collection of documents according to a user specific hierarchy. Major issues of current approaches to constraint based clustering are discussed, especially towards the hierarchical setting. We introduce the concept of hierarchical constraints and continue by presenting and evaluating two approaches using them. The approaches cover the two major fields of constraint based clustering, i.e. instance and metric based constraint integration. Our objects of interest are text documents. Therefore, the presented algorithms are especially fitted to work for these where necessary. Despite showing the properties and ideas of the algorithms in general, we evaluated the case of constraints that are unevenly scattered over the instance space, which is very common for real-world problems but not satisfyingly covered in other work so far.

Hierarchical classification refers to an extension of the standard classification problem, in which labels must be chosen from a class hierarchy. In this paper, we look at hierarchical classification from an information retrieval point of view. More specifically, we consider a scenario in which a user searches a document in a topic hierarchy. This scenario gives rise to the problem of predicting an optimal entry point, that is, a topic node in which the user starts searching. The usefulness of a corresponding prediction strongly depends on the search behavior of the user, which becomes relevant if the document is not immediately found in the predicted node. Typically, users tend to browse the hierarchy in a top-down manner, i.e., they look at a few more specific subcategories but usually refuse exploring completely different branches of the search tree. From a classification point of view, this means that a prediction should be evaluated, not solely on the basis of its correctness, but rather by judging its usefulness against the background of the user behavior. The idea of this paper is to formalize hierarchical classification within a decision-theoretic framework which allows for modeling this usefulness in terms of a user-specific utility function. The prediction problem thus becomes a problem of expected utility maximization. Apart from its theoretical appeal, we provide first empirical results showing that the approach performs well in practice.

A hierarchical structure can provide efficient access to information contained in a collection of documents. However, such a structure is not always available, e.g. for a set of documents a user has collected over time in a single folder or the results of a Web search. We therefore investigate in this paper how we can obtain a hierarchical structure automatically, taking into account some background knowledge about the way a specific user would structure the collection. More specifically, we adapt a hierarchical agglomerative clustering algorithm to take into account user specific constraints on the clustering process. Such an algorithm could be applied, e.g., for user specific clustering of Web search results, where the user's constraints on the clustering process are given by a hierarchical folder or bookmark structure. Besides the discussion of the algorithm itself, we motivate application scenarios and present an evaluation of the proposed algorithm on benchmark data

We discuss a retrieval model in which the task is to complete a sentence, given an initial fragment, and given an application specific document collection. This model is motivated by administrative and call center environments, in which users have to write documents with a certain repetitiveness. We formulate the problem setting and discuss appropriate performance metrics. We present an index-based retrieval algorithm and a cluster-based approach, and evaluate our algorithms using collections of emails that have been written by two distinct service centers.