Artificial Intelligence Screening for Emergency Medicine Residency Applications

Background: Residency application volumes have increased, straining faculty time for holistic medical student application review. Our objective was to develop and compare an artificial intelligence (AI)–assisted scoring tool for emergency medicine (EM) interview screening compared to the current method of faculty screening.

Methods: We developed an AI-assisted tool using the cloud-based platform Airtable to generate scores in 10 predefined domains (examination scores; research, special projects, and community service; leadership and distinction; prior EM experience, Medical Student Performance Evaluation; two Standardized Letters of Evaluation; geographic affinity; exceptional characteristics; and red flag concerns). Seventy-five applicants from one recruitment cycle were independently scored by the AI tool and a faculty reviewer. We calculated percent agreement overall and by domain. All discrepant scores were adjudicated by a trained independent blinded reviewer, who determined which score (AI vs Faculty) was most accurate. Among discrepant ratings, we compared the proportion adjudicated correct for AI versus faculty using McNemar’s test.

Results: Across 750 domain-level ratings, the AI tool and faculty agreed exactly on 551 (73.5%; 95% CI 70.2%–76.5%). Agreement by domain ranged from 44% (leadership and distinction) to 96% (red flag concerns). Among 199 discrepant ratings, the AI was adjudicated correct in 117 (58.8%; 95% CI 51.9%–65.4%) versus faculty in 82 (41.2%; 95% CI 34.3%–48.4%); McNemar p<0.05. Patterns were similar across most individual domains.

Conclusions: An AI-assisted application scoring tool demonstrated substantial agreement with faculty ratings and more often matched adjudicated scores when disagreement occurred. AI-assisted scoring may be a feasible adjunct for initial residency application screening; future work should evaluate time savings and downstream selection outcomes.