a confident diagnosis can be made. We use an LLM to retrieve an initial pool of evidence, but then refine this set of evidence according to correlations learned by the model. We conduct an in-depth evaluation of the usefulness of our approach by simulating how it might be used by a clinician to decide between a pre-defined list of differential diagnoses.

Multidimensional scaling (MDS) is a technique for visualizing multidimensional data points as a 2D scatter plot. It can be applied to execution profiles of software to reveal how similar executions are to one another. This is useful for certain software engineering applications, which require accurate representation of small dissimilarities and nearest neighbor relationships. However, the high-dimensionality of profiles can cause MDS techniques to represent small dissimilarities poorly. We evaluate several variants of MDS on large sets of profiles, to see which techniques produce the most accurate displays. These include four previously proposed techniques -classical scaling followed by iterative majorization, energy minimization, ordinal AIDS, and cluster differences scaling - and two techniques of our invention - hierarchical MDS and sparse region scaling. The results suggest that each technique except ordinal MDS can significantly improve the representation of small dissimilarities between program executions and that hierarchical MDS and sparse region scaling perform best overall

We experimentally evaluate the effectiveness of using cluster analysis of execution profiles to find failures among the executions induced by a set of potential test cases. We compare several filtering procedures for selecting executions to evaluate for conformance to requirements. Each filtering procedure involves a choice of a sampling strategy and a clustering metric. The results suggest that filtering procedures based on clustering are more effective than simple random sampling for identifying failures in populations of operational executions, with adaptive sampling from clusters being the most effective sampling strategy. The results also suggest that clustering metrics that give extra weight to industrial profile features are most effective. Scatter plots of execution populations, produced by multidimensional scaling, are used to provide intuition for these results.

Observation-based testing calls for analyzing profiles of executions induced by potential test cases, in order to select a subset of executions to be checked for conformance to requirements. A family of techniques for selecting such a subset is evaluated experimentally. These techniques employ automatic cluster analysis to partition executions, and they use various sampling techniques to select executions from clusters. The experimental results support the hypothesis that with appropriate profiling, failures often have unusual profiles that are revealed by cluster analysis. The results also suggest that failures often form small clusters or chains in sparsely-populated areas of the profile space. A form of adaptive sampling called failure-pursuit sampling is proposed for revealing failures in such regions, and this sampling method is evaluated experimentally. The results suggest that failure-pursuit sampling is effective.