Abstract Context Program synthesis is the task of automatically finding a program that satisfies the user intention. In previous work, we developed APS‐GA, a program synthesizer based on a genetic algorithm. As genetic algorithms depend on a fitness function, so does APS‐GA. Researchers argue that different distance metrics for a fitness function may reveal behavioral differences in the genetic algorithm. More recently, we presented initial evidence that APS‐GA was not affected by different distance metrics for its fitness function. However, that study was carried out on a medium‐sized scale. Objective In order to investigate our previous study on a larger scale, we extended our experiment to replicate it on a search space that is up to 6500 times larger than our previous work, and we ran it with a synthesis time that was at least 20 times longer. We have chosen the same five distance metrics as fitness functions to check whether they affect the synthesis task of five integer domain imperative toy programs. Method A hypothesis test was proposed and experiments were conducted to observe the number of calls to the fitness function () and to measure the synthesis time (). Results By considering a confidence level of 95% (with ), we found out that there were no significant differences in both and . Conclusion With these results, our extended replication study suggests that the discrete distance metric constitutes the best choice for APS‐GA as it guides the search with the same effectiveness as the other metrics, and is cheaper to compute.

Abstract Context Program synthesis is the task of automatically finding a program that satisfies the user intention. In previous work, we have developed a program synthesizer that integrates genetic algorithm with model finder. A genetic algorithm uses a fitness function to calculate how “distant to a solution” a given candidate program is. Researchers argue that different distance metrics for a fitness function may reveal behavioral differences in the genetic algorithm. Objective We have chosen five distance metrics as fitness functions to check whether they affect the synthesis task of five different integer domain imperative toy‐programs which read/write integer values using fundamental syntactic constructs, such as while, if‐then‐else, and so forth. We have used input/output examples and sketches to constrain the search space of the candidate programs. Method A hypothesis test was proposed and experiments were conducted to observe the number of calls to the fitness function (x) and to measure the synthesis time ( ). Results Regarding x, the synthesizer found a solution for all five subjects after calling the fitness function the same amount of times. For , a one‐way ANOVA was performed with a significance level of 5% ( ). No significant differences were observed in both x and . Conclusion With these preliminary results, this study suggests that the discrete distance metric is the best choice, because it guides the search with the same effectiveness as the others and is not time consuming, and so forth. However, future experimentation with a larger search space will confirm or not this initial impression.