CONTEXT: The Heart Disease Program (HDP) is a novel computerized diagnosis program incorporating a computer model of cardiovascular physiology. Physicians can enter standard clinical data and receive a differential diagnosis with explanations. OBJECTIVE: To evaluate the diagnostic performance of the HDP and its usability by physicians in a typical clinical setting. DESIGN: A prospective observational study of the HDP in use by physicians in departments of medicine and cardiology of a teaching hospital. Data came from 114 patients with a broad range of cardiac disorders, entered by six physicians. MEASUREMENTS: Sensitivity, specificity, and positive predictive value (PPV). Comprehensiveness: the proportion of final diagnoses suggested by the HDP or physicians for each case. RELEVANCE: the proportion of HDP or physicians' diagnoses that are correct. Area under the receiver operating characteristic (ROC) curve (AUC) for the HDP and the physicians. Performance was compared with a final diagnosis based on follow-up and further investigations. RESULTS: Compared with the final diagnoses, the HDP had a higher sensitivity (53.0% vs. 34.8%) and significantly higher comprehensiveness (57.2% vs. 39.5%, p < 0.0001) than the physicians. Physicians' PPV and relevance (56.2%, 56.0%) were higher than the HDP (25.4%, 28.1%). Combining the diagnoses of the physicians and the HDPs, sensitivity was 61.3% and comprehensiveness was 65.7%. These findings were significant in the two collection cohorts and for subanalysis of the most serious diagnoses. The AUCs were similar for the HDP and the physicians. CONCLUSIONS: The heart disease program has the potential to improve the differential diagnoses of physicians in a typical clinical setting.

OBJECTIVE: Evaluate the accuracy of the detailed diagnostic reasoning of the Heart Failure Program incorporating a new mechanism to handle temporal relationships and severity constraints. DESIGN: Tools were developed to summarize diagnoses and automatically generate evaluation forms. Five expert cardiologists were asked to review the reasoning of the program, with two analyzing each case. Cases were gathered retrospectively for diversity and difficulty and 26 randomly selected cases were evaluated. The underlying issues were identified and classified. RESULTS: Both reviewers rated the first diagnosis correct in 25% of the cases and at least one rated it wrong in 10%. Analyzing the detailed reasoning, 137 issues were raised, about 5.3 per case. Of these, 53% were possible concerns raised by one reviewer. Of the 5.3 issues per case, 2.5 were attributable to controversies, misunderstandings, or mistakes; 1 was due to the overly simplistic representation of the summaries; and 1.8 were issues related to the program. CONCLUSION: Overall, the program is capable of providing high-quality detailed diagnostic hypotheses for complex cardiovascular cases. The results highlight several issues: 1) the difficulty of effectively summarizing hypotheses, 2) the nature of a physician's causal explanation, and 3) some problems in evaluating detailed diagnostic reasoning. The mistakes the program made imply that some additional refinement is needed but that the reasoning mechanisms developed can support the appropriate reasoning. The appropriate next step is a prospective evaluation addressing the program's usefulness.