Auditing RadExam : Employing Psychometrics to Improve Exam Quality
Copyright © 2020 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved..
INTRODUCTION: RadExam is a question item and exam database jointly developed by the Association of Program Directors in Radiology and the American College of Radiology to provide formative resident assessment, offering performance metrics benchmarked against institutional and national resident performance. Beyond resident performance, data is available on question and exam performance. Despite considerable investment in the education and training of its question writers and editors and meticulous attention to current psychometrically validated methods, it was anticipated a minority of exam questions would still perform poorly. Audits were performed to identify these questions, identify reasons for poor performance, and modify or replace so-affected questions. Exam performance was also assessed.
METHODS: Two audits were performed, the first after the February-May 2018 RadExam pilot phase, and the second nearly 1 year after the full implementation of RadExam. In each audit, RadExam subspecialty editors evaluated all exam questions and exams using statistical data: question and test number of administrations, question p value, question Discrimination Index (DI), question Bloom's taxonomy learning level, exam P-value, and the number of image-based questions in each exam. Identified questions were modified or removed and replaced.
RESULTS: Audit 1 was performed after the administration of 3114 exams comprised of 2520 questions administered across 100 residency programs. Audit 1 identified 617 questions with DI <0.1 and 565 questions with unacceptable P-values, all of which were modified or replaced. Audit 2 was performed after the administration of 16,416 exams, comprised of 2,507 questions. Audit 2 identified 229 questions with DI <0.1 and 290 questions with unacceptable P-values, representing a 49.1% decrease in total flagged questions compared to Audit 1. Statistically significant decreases were seen in questions with both DI and P-values outside of the desired range across nearly all subspecialties.
CONCLUSION: The positive impact of our audit system on question and exam performance was reflected in a significant decrease in the number of questions flagged and improved overall exam performance in Audit 2. This illustrates the positive impact of Audit 1.
| Errataetall: |
CommentIn: Acad Radiol. 2021 Oct;28(10):1399-1400. - PMID 33965310 |
|---|---|
| Medienart: |
E-Artikel |
| Erscheinungsjahr: |
2021 |
|---|---|
| Erschienen: |
2021 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:28 |
|---|---|
| Enthalten in: |
Academic radiology - 28(2021), 10 vom: 13. Okt., Seite 1389-1398 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Smith, Elana B [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Educational assessment |
|---|
| Anmerkungen: |
Date Completed 08.10.2021 Date Revised 08.10.2021 published: Print-Electronic CommentIn: Acad Radiol. 2021 Oct;28(10):1399-1400. - PMID 33965310 Citation Status MEDLINE |
|---|
| doi: |
10.1016/j.acra.2020.05.037 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM312513054 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM312513054 | ||
| 003 | DE-627 | ||
| 005 | 20231225144555.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231225s2021 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.acra.2020.05.037 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1041.xml |
| 035 | |a (DE-627)NLM312513054 | ||
| 035 | |a (NLM)32674906 | ||
| 035 | |a (PII)S1076-6332(20)30347-0 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Smith, Elana B |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Auditing RadExam |b Employing Psychometrics to Improve Exam Quality |
| 264 | 1 | |c 2021 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 08.10.2021 | ||
| 500 | |a Date Revised 08.10.2021 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a CommentIn: Acad Radiol. 2021 Oct;28(10):1399-1400. - PMID 33965310 | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a Copyright © 2020 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. | ||
| 520 | |a INTRODUCTION: RadExam is a question item and exam database jointly developed by the Association of Program Directors in Radiology and the American College of Radiology to provide formative resident assessment, offering performance metrics benchmarked against institutional and national resident performance. Beyond resident performance, data is available on question and exam performance. Despite considerable investment in the education and training of its question writers and editors and meticulous attention to current psychometrically validated methods, it was anticipated a minority of exam questions would still perform poorly. Audits were performed to identify these questions, identify reasons for poor performance, and modify or replace so-affected questions. Exam performance was also assessed | ||
| 520 | |a METHODS: Two audits were performed, the first after the February-May 2018 RadExam pilot phase, and the second nearly 1 year after the full implementation of RadExam. In each audit, RadExam subspecialty editors evaluated all exam questions and exams using statistical data: question and test number of administrations, question p value, question Discrimination Index (DI), question Bloom's taxonomy learning level, exam P-value, and the number of image-based questions in each exam. Identified questions were modified or removed and replaced | ||
| 520 | |a RESULTS: Audit 1 was performed after the administration of 3114 exams comprised of 2520 questions administered across 100 residency programs. Audit 1 identified 617 questions with DI <0.1 and 565 questions with unacceptable P-values, all of which were modified or replaced. Audit 2 was performed after the administration of 16,416 exams, comprised of 2,507 questions. Audit 2 identified 229 questions with DI <0.1 and 290 questions with unacceptable P-values, representing a 49.1% decrease in total flagged questions compared to Audit 1. Statistically significant decreases were seen in questions with both DI and P-values outside of the desired range across nearly all subspecialties | ||
| 520 | |a CONCLUSION: The positive impact of our audit system on question and exam performance was reflected in a significant decrease in the number of questions flagged and improved overall exam performance in Audit 2. This illustrates the positive impact of Audit 1 | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Educational assessment | |
| 650 | 4 | |a Graduate medical education | |
| 650 | 4 | |a Internship and residency | |
| 650 | 4 | |a Medical education | |
| 650 | 4 | |a Radiology education | |
| 700 | 1 | |a Lewis, Petra |e verfasserin |4 aut | |
| 700 | 1 | |a Benefield, Thad |e verfasserin |4 aut | |
| 700 | 1 | |a Catanzano, Tara M |e verfasserin |4 aut | |
| 700 | 1 | |a Khan, Muhammad J |e verfasserin |4 aut | |
| 700 | 1 | |a Nyberg, Eric |e verfasserin |4 aut | |
| 700 | 1 | |a Jordan, Sheryl |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Academic radiology |d 1995 |g 28(2021), 10 vom: 13. Okt., Seite 1389-1398 |w (DE-627)NLM087676818 |x 1878-4046 |7 nnns |
| 773 | 1 | 8 | |g volume:28 |g year:2021 |g number:10 |g day:13 |g month:10 |g pages:1389-1398 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.acra.2020.05.037 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 28 |j 2021 |e 10 |b 13 |c 10 |h 1389-1398 | ||