AUCReshaping : improved sensitivity at high-specificity
© 2023. The Author(s)..
The evaluation of deep-learning (DL) systems typically relies on the Area under the Receiver-Operating-Curve (AU-ROC) as a performance metric. However, AU-ROC, in its holistic form, does not sufficiently consider performance within specific ranges of sensitivity and specificity, which are critical for the intended operational context of the system. Consequently, two systems with identical AU-ROC values can exhibit significantly divergent real-world performance. This issue is particularly pronounced in the context of anomaly detection tasks, a commonly employed application of DL systems across various research domains, including medical imaging, industrial automation, manufacturing, cyber security, fraud detection, and drug research, among others. The challenge arises from the heavy class imbalance in training datasets, with the abnormality class often incurring a considerably higher misclassification cost compared to the normal class. Traditional DL systems address this by adjusting the weighting of the cost function or optimizing for specific points along the ROC curve. While these approaches yield reasonable results in many cases, they do not actively seek to maximize performance for the desired operating point. In this study, we introduce a novel technique known as AUCReshaping, designed to reshape the ROC curve exclusively within the specified sensitivity and specificity range, by optimizing sensitivity at a predetermined specificity level. This reshaping is achieved through an adaptive and iterative boosting mechanism that allows the network to focus on pertinent samples during the learning process. We primarily investigated the impact of AUCReshaping in the context of abnormality detection tasks, specifically in Chest X-Ray (CXR) analysis, followed by breast mammogram and credit card fraud detection tasks. The results reveal a substantial improvement, ranging from 2 to 40%, in sensitivity at high-specificity levels for binary classification tasks.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:13 |
|---|---|
| Enthalten in: |
Scientific reports - 13(2023), 1 vom: 30. Nov., Seite 21097 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Bhat, Sheethal [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Completed 04.12.2023 Date Revised 21.12.2023 published: Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1038/s41598-023-48482-x |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM365277223 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM365277223 | ||
| 003 | DE-627 | ||
| 005 | 20231227135421.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1038/s41598-023-48482-x |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1234.xml |
| 035 | |a (DE-627)NLM365277223 | ||
| 035 | |a (NLM)38036602 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Bhat, Sheethal |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a AUCReshaping |b improved sensitivity at high-specificity |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 04.12.2023 | ||
| 500 | |a Date Revised 21.12.2023 | ||
| 500 | |a published: Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © 2023. The Author(s). | ||
| 520 | |a The evaluation of deep-learning (DL) systems typically relies on the Area under the Receiver-Operating-Curve (AU-ROC) as a performance metric. However, AU-ROC, in its holistic form, does not sufficiently consider performance within specific ranges of sensitivity and specificity, which are critical for the intended operational context of the system. Consequently, two systems with identical AU-ROC values can exhibit significantly divergent real-world performance. This issue is particularly pronounced in the context of anomaly detection tasks, a commonly employed application of DL systems across various research domains, including medical imaging, industrial automation, manufacturing, cyber security, fraud detection, and drug research, among others. The challenge arises from the heavy class imbalance in training datasets, with the abnormality class often incurring a considerably higher misclassification cost compared to the normal class. Traditional DL systems address this by adjusting the weighting of the cost function or optimizing for specific points along the ROC curve. While these approaches yield reasonable results in many cases, they do not actively seek to maximize performance for the desired operating point. In this study, we introduce a novel technique known as AUCReshaping, designed to reshape the ROC curve exclusively within the specified sensitivity and specificity range, by optimizing sensitivity at a predetermined specificity level. This reshaping is achieved through an adaptive and iterative boosting mechanism that allows the network to focus on pertinent samples during the learning process. We primarily investigated the impact of AUCReshaping in the context of abnormality detection tasks, specifically in Chest X-Ray (CXR) analysis, followed by breast mammogram and credit card fraud detection tasks. The results reveal a substantial improvement, ranging from 2 to 40%, in sensitivity at high-specificity levels for binary classification tasks | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Mansoor, Awais |e verfasserin |4 aut | |
| 700 | 1 | |a Georgescu, Bogdan |e verfasserin |4 aut | |
| 700 | 1 | |a Panambur, Adarsh B |e verfasserin |4 aut | |
| 700 | 1 | |a Ghesu, Florin C |e verfasserin |4 aut | |
| 700 | 1 | |a Islam, Saahil |e verfasserin |4 aut | |
| 700 | 1 | |a Packhäuser, Kai |e verfasserin |4 aut | |
| 700 | 1 | |a Rodríguez-Salas, Dalia |e verfasserin |4 aut | |
| 700 | 1 | |a Grbic, Sasa |e verfasserin |4 aut | |
| 700 | 1 | |a Maier, Andreas |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Scientific reports |d 2011 |g 13(2023), 1 vom: 30. Nov., Seite 21097 |w (DE-627)NLM215703936 |x 2045-2322 |7 nnns |
| 773 | 1 | 8 | |g volume:13 |g year:2023 |g number:1 |g day:30 |g month:11 |g pages:21097 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1038/s41598-023-48482-x |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 13 |j 2023 |e 1 |b 30 |c 11 |h 21097 | ||