Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
© 2022 The Authors..
Purpose: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles. Approach: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures. Results: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation. Conclusion: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2022 |
|---|---|
| Erschienen: |
2022 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:9 |
|---|---|
| Enthalten in: |
Journal of medical imaging (Bellingham, Wash.) - 9(2022), 5 vom: 07. Sept., Seite 052407 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Noothout, Julia M H [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Deep learning |
|---|
| Anmerkungen: |
Date Revised 29.05.2023 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1117/1.JMI.9.5.052407 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM342145398 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM342145398 | ||
| 003 | DE-627 | ||
| 005 | 20231226013423.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2022 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1117/1.JMI.9.5.052407 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1140.xml |
| 035 | |a (DE-627)NLM342145398 | ||
| 035 | |a (NLM)35692896 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Noothout, Julia M H |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation |
| 264 | 1 | |c 2022 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 29.05.2023 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a © 2022 The Authors. | ||
| 520 | |a Purpose: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles. Approach: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures. Results: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation. Conclusion: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a deep learning | |
| 650 | 4 | |a ensembles | |
| 650 | 4 | |a knowledge distillation | |
| 650 | 4 | |a segmentation | |
| 700 | 1 | |a Lessmann, Nikolas |e verfasserin |4 aut | |
| 700 | 1 | |a van Eede, Matthijs C |e verfasserin |4 aut | |
| 700 | 1 | |a van Harten, Louis D |e verfasserin |4 aut | |
| 700 | 1 | |a Sogancioglu, Ecem |e verfasserin |4 aut | |
| 700 | 1 | |a Heslinga, Friso G |e verfasserin |4 aut | |
| 700 | 1 | |a Veta, Mitko |e verfasserin |4 aut | |
| 700 | 1 | |a van Ginneken, Bram |e verfasserin |4 aut | |
| 700 | 1 | |a Išgum, Ivana |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Journal of medical imaging (Bellingham, Wash.) |d 2014 |g 9(2022), 5 vom: 07. Sept., Seite 052407 |w (DE-627)NLM244927499 |x 2329-4302 |7 nnns |
| 773 | 1 | 8 | |g volume:9 |g year:2022 |g number:5 |g day:07 |g month:09 |g pages:052407 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1117/1.JMI.9.5.052407 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 9 |j 2022 |e 5 |b 07 |c 09 |h 052407 | ||