In this paper, we propose a novel framework for quantifying metacognitive processes in ensembles of Large Language Models (LLMs) and extending the traditional teacher-student model through the lens of dual-process cognitive theory.We introduce a Metacognitive State Vector (MSV) that operationalizes metacognition across five dimensions: emotional response analysis, correctness evaluation, experiential matching, conflicting information estimation, and problem importance task prioritization.

In the legal field, corporate civil matters often entail tens or hundreds of thousands of documents that need review for relevance. Technology-Assisted Review (TAR) systems utilize machine learning classification algorithms, such as logistic regression, SVM, and transformers, to retrieve all, or nearly all, relevant documents from the corpus under review. However, in these e-discovery scenarios, TAR systems are typically perceived as “black boxes” by practitioners; where the TAR system provides little or no insight into why a document is predicted to be relevant. This lack of explainability makes it difficult for attorneys to trust classifications from TAR systems, hinders litigants from participating fully as they cannot understand why documents are being classified as relevant, and relies on the costly interpretation of experts rather than the model itself for understanding.In contrast to these opaque methods, the Fuzzy ARTMAP algorithm is an explainable neural network architecture that is both geometrically interpretable and allows for the extraction of fuzzy If-Then rules from the model at any point in its training. This enables a practitioner or researcher multiple modes with which to understand what the model has learned up to that point, laying the foundation for Explainable e-Discovery (XeD).In this paper, the explainable Fuzzy ARTMAP neural network is extended to include fuzzy subsethood to rank documents for active learning and is then evaluated for use in the TAR domain with several corpora. In addition to achieving desirable performance for a TAR system, it also enables direct insight into how the algorithm decides relevance. This is in contrast to existing approaches for explainable TAR which rely on extracting document snippets as post hoc explanations of why a document is relevant. Additionally, we demonstrate the model’s interpretability with both textual and graphical representations of the learned model for a range of representations including tf-idf, GloVe, and Word2Vec.

In the legal field, Technology-Assisted Review (TAR) systems for e-discovery are typically perceived as "black boxes" by practitioners, providing little to no insight into how the system makes its classification predictions. The lack of explainability in TAR systems for e-discovery renders their decisions opaque, making it difficult for attorneys to trust their recommendations and thus to discharge ethical obligations to clients. In addition, litigants cannot fully participate in the process if they cannot understand the relevance judgments, and jurists cannot make well-informed judgments on discovery matters. The Fuzzy ARTMAP algorithm is an explainable neural network architecture that permits the extraction of fuzzy If-Then rules from the model at any point in its training, the model is also geometrically interpretable, allowing a researcher or practitioner to understand what the model has learned up to that point.This paper evaluates the explainable Fuzzy ARTMAP algorithm for use in the TAR domain. Not only does it achieve suitable document classification performance for a TAR system, as measured by recall and recall-at-effort, but it also enables direct insight into how the algorithm decides relevance. This is in contrast to existing approaches for explainable TAR which only rely on extracting document snippets as post hoc explanations of why a document is relevant.In addition, the effect of different document features (tf-idf, word2vec, and GloVe) on recall performance is also evaluated. Performance is compared to AutoTAR, the state-of-the-art TAR algorithm which makes relevance predictions but is not able to provide any explanations about them. Experiments on the Reuters-21578 and 20Newsgroups corpora indicate robust recall performance overall and comparable or better metrics than AutoTAR in some circumstances.