Too often, programmers are under pressure to maximize their confidence in the correctness of their code with a tight testing budget. Should they spend some of that budget on finding “interesting” inputs or spend their entire testing budget on test executions? Work on testing efficiency has explored two competing approaches to answer this question: systematic partition testing (ST), which defines a testing partition and tests its parts, and random testing (RT), which directly samples inputs with replacement. A consensus as to which is better when has yet to emerge. We present Probability Ordered Partition Testing (PopArt), a new systematic partition-based testing strategy that visits the parts of a testing partition in decreasing probability order and in doing so leverages any non-uniformity over that partition. We show how to construct a homogeneous testing partition, a requirement for systematic testing, by using an executable oracle and the path partition. A program's path partition is a naturally occurring testing partition that is usually skewed for the simple reason that some paths execute more frequently than others. To confirm this conventional wisdom, we instrument programs from the Codeflaws repository and find that 80% of them have a skewed path probability distribution.PopArtvisits the parts of a testing partition in decreasing probability order. We then comparePopArtwithRTto characterise the configuration space in which each is more efficient. We show that, when simulating Codeflaws,PopArtoutperformsRTafter$100{,}000$executions. Our results reaffirmRT's power for very small testing budgets but also show that for any application requiring high (above 90%) probability-weighted coveragePopArtshould be preferred. In such cases, despite paying more for each test execution, we prove thatPopArtoutperformsRT: it traverses parts whose cumulative probability bounds that of random testing, showing that sampling without replacement pays for itself, given a nonuniform probability over a testing partition.

There are two major families in cryptanalytic attacks on symmetric ciphers: statistical attacks and algebraic attacks. In this position paper we argue that algebraic cryptanalysis has not yet been developed properly due to the weakness of the theory which has substantial difficulty to prove most basic results on the number of linearly independent equations in algebraic attacks. Consequently most authors present a restricted range of attacks which are shown experimentally to work with their computer but refrain from claiming results which would work on a larger computer but have not yet been tested. For example in recent 2015 work of Raddum we discover that (experimentally) ElimLin attack breaks up to 16 rounds of Simon block cipher however it is hard to know what happens for 17 rounds. In this paper we argue that one CAN predict and model the behavior of such attacks and evaluate complexity of the attacks which we cannot yet execute. To the best of our knowledge this has never been done before.