Smartphones enable to collect large data streams about phone calls that, once combined with Computational Intelligence techniques, bring great potential for improving the monitoring of patients with mental illnesses. However, the acoustic data streams recorded in uncontrolled environments are dynamically changing due to various sources of uncertainty. In addition, such acoustic data are usually difficult to interpret by psychiatrists. Within this study, we propose an approach based on Linguistic Summaries with Fuzzy Clustering (LS-FC) aiming at the development of human-consistent and easily interpretable summaries about relations between acoustic data and mental state of a patient affected by Bipolar Disorder, e.g., Most calls in the state of hypomania have low loudness compared to the state of euthymia [T = 1]. To capture the dynamics of acoustic data streams, we apply a dynamic incremental semi-supervised fuzzy clustering that synthesizes data into clusters. These clusters are represented by prototypes which are used for the construction of the membership functions describing linguistic terms e.g., low loudness, and then, linguistic summaries. The main contribution of this paper is the incorporation of information about clusters’ prototypes in the generation of linguistic summaries. The primary goal of this research is explainability. The semi-supervised learning algorithm is used mainly for deriving clusters and building improved linguistic summaries. Numerical results indicate that linguistic summaries provide intuitive and clear information about voice features in a patient’s affective state and they are consistent with clinical observation. In particular, during most calls in hypomania/mania both the quality of the patient’s voice and the dynamics of change in the spectrum signal reflected in spectral flux are low compared to euthymia. The proposed approach enables to summarize large data streams into meaningful descriptions that, although relatively simple, offer information granules that are very intuitive for clinicians and are promising to support the smartphone-based monitoring of bipolar disorder patients to inform about the potential change of mental state.

We introduce an approach called PLENARY (exPlaining bLack-box modEls in Natural lAnguage thRough fuzzY linguistic summaries), which is an explainable classifier based on a data-driven predictive model. Neural learning is exploited to derive a predictive model based on two levels of labels associated with the data. Then, model explanations are derived through the popular SHapley Additive exPlanations (SHAP) tool and conveyed in a linguistic form via fuzzy linguistic summaries. The linguistic summarization allows translating the explanations of the model outputs provided by SHAP into statements expressed in natural language. PLENARY accounts for the imprecision related to model outputs by summarizing them into simple linguistic statements and for the imprecision related to the data labeling process by including additional domain knowledge in the form of middle-layer labels. PLENARY is validated on preprocessed speech signals collected from smartphones from patients with bipolar disorder and on publicly available mental health survey data. The experiments confirm that fuzzy linguistic summarization is an effective technique to support meta-analyses of the outputs of AI models. Also, PLENARY improves explainability by aggregating low-level attributes into high-level information granules, and by incorporating vague domain knowledge into a multi-task sequential and compositional multilayer perceptron. SHAP explanations translated into fuzzy linguistic summaries significantly improve understanding of the predictive modelling process and its outputs.