Retraining a classifier with new data is inseparable from ML/AI applications, but most of the existing ML methods do not take into account the backward compatibility of predictions. That is, although the overall performance of a new classifier is improved, users will be confused by the wrong predictions of the new classifier, especially when the predictions of the old classifier are correct for the same samples. To this end, several metrics and learning methods for backward compatibility have been actively studied recently. Despite significant interest in backward compatibility, the metrics and methods are not well known from a theoretical perspective. In this paper, we first analyze the existing backward compatibility metrics and reveal that these metrics essentially assess the same quantity between old and new models. In addition, to obtain a unified view of backward compatibility metrics, we propose a generalized backward compatibility (GBC) metric that can represent the existing backward compatibility metrics. We formulate a learning objective based on the GBC metric and derive the estimation error bound, and the result is applied to one of the existing methods. Through further analysis, we reveal that the existing backward compatibility metrics are not suitable for imbalanced classification. We then design a backward compatibility metric for imbalanced classification on the basis of the GBC metric and empirically demonstrate the practicality of the proposed metric.

Prediction in a new domain without any training sample, called zero-shot domain adaptation (ZSDA), is an important task in domain adaptation.While prediction in a new domain has gained much attention in recent years, in this paper, we investigate another potential of ZSDA.Specifically, instead of predicting responses in a new domain, we find a description of a new domain given a prediction.The task is regarded as predictive optimization, but existing predictive optimization methods have not been extended to handling multiple domains.We propose a simple framework for predictive optimization with ZSDA and analyze the condition in which the optimization problem becomes convex optimization.We also discuss how to handle the interaction of characteristics of a domain in predictive optimization.Through numerical experiments, we demonstrate the potential usefulness of our proposed framework.

Predictive optimization is a framework for designing an entire data-analysis pipeline that comprises both prediction and optimization, to be able to maximize overall throughput performance. In practical demand analysis, a knowledge of hierarchies, which might be geographical or categorical, is recognized as useful, though such additional knowledge has not been taken into account in existing predictive optimization. In this paper, we propose a novel hierarchical predictive optimization pipeline that is able to deal with a wide range of applications including inventory management. Based on an existing hierarchical demand prediction model, we present a stochastic matching framework that can manage prediction-uncertainty in decision making. We further provide a greedy approximation algorithm for solving demand matching on hierarchical structures. In experimental evaluations on both artificial and real-world data, we demonstrate the effectiveness of our proposed hierarchical-predictive-optimization pipeline.