Explainability is crucial for improving the transparency of black-box machine learning models. With the advancement of explanation methods such as LIME and SHAP, various XAI performance metrics have been developed to evaluate the quality of explanations. However, different explainers can provide contrasting explanations for the same prediction, introducing trade-offs across conflicting quality metrics. Although available aggregation approaches improve robustness, reducing explanations’ variability, very limited research employed a multi-criteria decision-making approach. To address this gap, this paper introduces a multi-criteria rank-based weighted aggregation method that balances multiple quality metrics simultaneously to produce an ensemble of explanation models. Furthermore, we propose rank-based versions of existing XAI metrics (complexity, faithfulness and stability) to better evaluate ranked feature importance explanations. Extensive experiments on publicly available datasets demonstrate the robustness of the proposed model across these metrics. Comparative analyses of various multi-criteria decision-making and rank aggregation algorithms showed that TOPSIS and WSUM are the best candidates for this use case.

The crowd-powered systems have been shown to be highly successful in the current decade to manage collective contribution of online workers for solving different complex tasks. It can also be used for soliciting opinions from a large set of people working in a distributed manner. Unfortunately, the online community of crowd workers might involve non-experts as opinion providers. As a result, such approaches may give rise to noise making it hard to predict the appropriate (gold) judgment. Judgment analysis is in general a way of learning about human decision from multiple opinions. A spectrum of algorithms has been proposed in the last few decades to address this problem. They are broadly made up of supervised or unsupervised types. However, they have been readdressed in recent years having focus on different strategies for obtaining the gold judgment from crowdsourced opinions, viz., estimating the accuracy of opinions, difficulties of the problem, spammer identification, handling noise, etc. Besides this, investigation of various types of crowdsourced opinions to solve complex real-life problems provide new insights in this domain. In this survey, we provide a comprehensive overview of the judgment analysis problem and some of its novel variants, addressed with different approaches, where the opinions are crowdsourced.

A vast majority of the existing crowd opinion aggregation models deal with independent opinions of the crowd workers, where the annotators (crowd workers) provide their opinions unanimously and these opinions are not visible to everyone. But in real life, it is seen that there are many applications in which an annotator can see others' opinions. As a result, there is a high chance that the opinion of an annotator gets biased by the other opinions. My proposed research is about finding a better consensus from multiple crowd opinions (dependent) obtained for a particular question. So, this work addresses a novel problem of dependent judgment analysis and proposes methods that derive the final judgment from a given set of ordered opinions. For the said problem, a Markov chain based aggregation method is proposed that can handle the ordered opinions of the crowd workers, and thus aggregate them to find a consensus. I have used a real-life dataset that contains the independent and dependent crowd opinions over some questions. Furthermore, I am studying a recently proposed model that collects crowd opinions twice (in independent and dependent fashion) to understand their relation. In a recent study, I have proposed a biclustering approach to compute aggregated judgment from multiple crowd opinions. In another current work, I have introduced a new problem dealing with constrained crowd opinions and demonstrated that proper aggregation of those type of opinions can be very helpful for smart city planning. These studies altogether focus on the different novel directions of the judgment analysis problem.