A New Tool for Holistic Residency Application Review : Using Natural Language Processing of Applicant Experiences to Predict Interview Invitation
Copyright © 2023 by the Association of American Medical Colleges..
PROBLEM: Reviewing residency application narrative components is time intensive and has contributed to nearly half of applications not receiving holistic review. The authors developed a natural language processing (NLP)-based tool to automate review of applicants' narrative experience entries and predict interview invitation.
APPROACH: Experience entries (n = 188,500) were extracted from 6,403 residency applications across 3 application cycles (2017-2019) at 1 internal medicine program, combined at the applicant level, and paired with the interview invitation decision (n = 1,224 invitations). NLP identified important words (or word pairs) with term frequency-inverse document frequency, which were used to predict interview invitation using logistic regression with L1 regularization. Terms remaining in the model were analyzed thematically. Logistic regression models were also built using structured application data and a combination of NLP and structured data. Model performance was evaluated on never-before-seen data using area under the receiver operating characteristic and precision-recall curves (AUROC, AUPRC).
OUTCOMES: The NLP model had an AUROC of 0.80 (vs chance decision of 0.50) and AUPRC of 0.49 (vs chance decision of 0.19), showing moderate predictive strength. Phrases indicating active leadership, research, or work in social justice and health disparities were associated with interview invitation. The model's detection of these key selection factors demonstrated face validity. Adding structured data to the model significantly improved prediction (AUROC 0.92, AUPRC 0.73), as expected given reliance on such metrics for interview invitation.
NEXT STEPS: This model represents a first step in using NLP-based artificial intelligence tools to promote holistic residency application review. The authors are assessing the practical utility of using this model to identify applicants screened out using traditional metrics. Generalizability must be determined through model retraining and evaluation at other programs. Work is ongoing to thwart model "gaming," improve prediction, and remove unwanted biases introduced during model training.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:98 |
|---|---|
| Enthalten in: |
Academic medicine : journal of the Association of American Medical Colleges - 98(2023), 9 vom: 01. Sept., Seite 1018-1021 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Mahtani, Arun Umesh [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Completed 23.10.2023 Date Revised 24.10.2023 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1097/ACM.0000000000005210 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM354457586 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM354457586 | ||
| 003 | DE-627 | ||
| 005 | 20231226062249.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1097/ACM.0000000000005210 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1181.xml |
| 035 | |a (DE-627)NLM354457586 | ||
| 035 | |a (NLM)36940395 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Mahtani, Arun Umesh |e verfasserin |0 (orcid)https://orcid.org/0000-0002-2101-7157 |4 aut | |
| 245 | 1 | 2 | |a A New Tool for Holistic Residency Application Review |b Using Natural Language Processing of Applicant Experiences to Predict Interview Invitation |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 23.10.2023 | ||
| 500 | |a Date Revised 24.10.2023 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a Copyright © 2023 by the Association of American Medical Colleges. | ||
| 520 | |a PROBLEM: Reviewing residency application narrative components is time intensive and has contributed to nearly half of applications not receiving holistic review. The authors developed a natural language processing (NLP)-based tool to automate review of applicants' narrative experience entries and predict interview invitation | ||
| 520 | |a APPROACH: Experience entries (n = 188,500) were extracted from 6,403 residency applications across 3 application cycles (2017-2019) at 1 internal medicine program, combined at the applicant level, and paired with the interview invitation decision (n = 1,224 invitations). NLP identified important words (or word pairs) with term frequency-inverse document frequency, which were used to predict interview invitation using logistic regression with L1 regularization. Terms remaining in the model were analyzed thematically. Logistic regression models were also built using structured application data and a combination of NLP and structured data. Model performance was evaluated on never-before-seen data using area under the receiver operating characteristic and precision-recall curves (AUROC, AUPRC) | ||
| 520 | |a OUTCOMES: The NLP model had an AUROC of 0.80 (vs chance decision of 0.50) and AUPRC of 0.49 (vs chance decision of 0.19), showing moderate predictive strength. Phrases indicating active leadership, research, or work in social justice and health disparities were associated with interview invitation. The model's detection of these key selection factors demonstrated face validity. Adding structured data to the model significantly improved prediction (AUROC 0.92, AUPRC 0.73), as expected given reliance on such metrics for interview invitation | ||
| 520 | |a NEXT STEPS: This model represents a first step in using NLP-based artificial intelligence tools to promote holistic residency application review. The authors are assessing the practical utility of using this model to identify applicants screened out using traditional metrics. Generalizability must be determined through model retraining and evaluation at other programs. Work is ongoing to thwart model "gaming," improve prediction, and remove unwanted biases introduced during model training | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Reinstein, Ilan |e verfasserin |4 aut | |
| 700 | 1 | |a Marin, Marina |e verfasserin |4 aut | |
| 700 | 1 | |a Burk-Rafel, Jesse |e verfasserin |0 (orcid)https://orcid.org/0000-0003-3785-2154 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Academic medicine : journal of the Association of American Medical Colleges |d 1989 |g 98(2023), 9 vom: 01. Sept., Seite 1018-1021 |w (DE-627)NLM012947180 |x 1938-808X |7 nnns |
| 773 | 1 | 8 | |g volume:98 |g year:2023 |g number:9 |g day:01 |g month:09 |g pages:1018-1021 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1097/ACM.0000000000005210 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 98 |j 2023 |e 9 |b 01 |c 09 |h 1018-1021 | ||