Validation of Acute Coronary Syndrome International Classification of Diseases Codes for Artificial Intelligence Model Development

Timely recognition in the emergency department (ED) of ST-segment myocardial infarction (STEMI), a subtype of acute coronary syndrome (ACS), improves outcomes. We developed an artificial intelligence (AI) model to improve ED ACS screening by identifying patients who need early ECG to promptly diagnose STEMI. In building this model, we used International Classification of Diseases (ICD) codes, a practical and widely available labeling source. However, their accuracy in contemporary ED populations is uncertain.

We conducted a retrospective cohort study of adults presenting to the Stanford ED from January 1, 2015, to December 31, 2024. ACS was defined using previously validated ICD codes. The reference standard was the final clinical diagnosis as determined by structured chart review of the treatment team’s documentation. We evaluated the accuracy of ICD codes for ACS by comparing them with chart-reviewed diagnoses documented by the treating team, to assess suitability for training AI models. To capture both correctly coded and potentially missed ACS, we reviewed all encounters with (1) an ACS ICD code; and (2) no ACS code but elevated troponins and an ICD code for coronary artery disease. We estimated raw agreement, Cohen’s kappa (κ), sensitivity, specificity, and positive predictive value (PPV).

Among 647,499 ED encounters, 6,168 (1.0%) received an ACS ICD code. Raw agreement between ICD-coded and true ACS diagnosis was 99.7% with κ = 0.81. Sensitivity was 94.0%, specificity was 99.7%, and PPV was 71.3%. Overall, ICD codes and clinical diagnosis disagreed in 0.31% of encounters (0.27% false positives, 0.04% false negatives). Much of this agreement is driven by the abundance of true negatives in this low-prevalence population. ICD codes captured most true ACS cases, although 3 in 1000 ACS-coded encounters were not ACS by chart review.

ACS ICD codes demonstrated very high agreement and strong κ with chart-reviewed ACS diagnoses. These findings suggest ICD-based labels are suitable for training ACS prediction models. The high sensitivity is particularly important for a screening application in the ED, where the risk of over-calling ACS is outweighed by the danger of missing it.