The so-called ‘Impossibility Theorem’ for fairness definitions is one of the more striking research results with both theoretical and practical consequences, as it states that satisfying certain combinations of fairness definitions is impossible. To date, this negative result has not yet been complemented with a positive one: a characterization of which combinations of fairness notions are possible. This work aims to fill this gap by identifying maximal sets of commonly used fairness definitions for binary classification that can be simultaneously satisfied. The fairness definitions used are demographic parity, equal opportunity, predictive equality, predictive parity, false omission rate parity, overall accuracy equality and treatment equality. We conclude that in total 12 maximal sets of these fairness definitions are possible, among which are seven combinations of two definitions, and five combinations of three definitions. Our findings also shed light on the practical relevance and utility of each of these 12 maximal fairness definitions in various scenarios, regarding the accuracy of the classifier and ratios of false positives and false negatives, considering the base rates.

Current tools for machine learning fairness only admit a limited range of fairness definitions and have seen little integration with automatic differentiation libraries, despite the central role these libraries play in modern machine learning pipelines. We introduce a framework of fairness regularization terms (fairret) which quantify bias as modular objectives that are easily integrated in automatic differentiation pipelines. By employing a general definition of fairness in terms of linear-fractional statistics, a wide class of fairrets can be computed efficiently. Experiments show the behavior of their gradients and their utility in enforcing fairness with minimal loss of predictive power compared to baselines. Our contribution includes a PyTorch implementation of the fairret framework.

Numerous methods have been implemented that pursue fairness with respect to sensitive features by mitigating biases in machine learning. Yet, the problem settings that each method tackles vary significantly, including the stage of intervention, the composition of sensitive features, the fairness notion, and the distribution of the output. Even in binary classification, the greatest common denominator of problem settings is small, significantly complicating benchmarking.Hence, we introduce ABCFair, a benchmark approach which allows adapting to the desiderata of the real-world problem setting, enabling proper comparability between methods for any use case. We apply this benchmark to a range of pre-, in-, and postprocessing methods on both large-scale, traditional datasets and on a dual label (biased and unbiased) dataset to sidestep the fairness-accuracy trade-off.