For a bucket test with a single criterion for success and a fixed number of samples or testing period, requiring a p-value less than a specified value of α for the success criterion produces statistical confidence at level 1 − α. For multiple criteria, a Bonferroni correction that partitions α among the criteria produces statistical confidence, at the cost of requiring lower p-values for each criterion. The same concept can be applied to decisions about early stopping, but that can lead to strict requirements for p-values. We show how to address that challenge by requiring criteria to be successful at multiple decision points.

Big data is often used to generate reports consisting of computed statistics. When the statistics are displayed, they may be marked for significance based on p-values. For multiple statistics, we propose a system to mark for significance adjusted for the number of statistics, using a Bonferroni correction in general, and a slightly milder correction for independent statistics.