Sensor fusion for the robust detection of facial regions of neonates using neural networks / Johanna Gleichauf, Lukas Hennemann, Fabian B. Fahlbusch, Oliver Hofmann, Christine Niebler and Alexander Koelpin
The monitoring of vital signs and increasing patient comfort are cornerstones of modern neonatal intensive care. Commonly used monitoring methods are based on skin contact which can cause irritations and discomfort in preterm neonates. Therefore, non-contact approaches are the subject of current research aiming to resolve this dichotomy. Robust neonatal face detection is essential for the reliable detection of heart rate, respiratory rate and body temperature. While solutions for adult face detection are established, the unique neonatal proportions require a tailored approach. Additionally, sufficient open-source data of neonates on the NICU is lacking. We set out to train neural networks with the thermal-RGB-fusion data of neonates. We propose a novel indirect fusion approach including the sensor fusion of a thermal and RGB camera based on a 3D time-of-flight (ToF) camera. Unlike other approaches, this method is tailored for close distances encountered in neonatal incubators. Two neural networks were used with the fusion data and compared to RGB and thermal networks. For the class “head” we reached average precision values of 0.9958 (RetinaNet) and 0.9455 (YOLOv3) for the fusion data. Compared with the literature, similar precision was achieved, but we are the first to train a neural network with fusion data of neonates. The advantage of this approach is in calculating the detection area directly from the fusion image for the RGB and thermal modality. This increases data efficiency by 66%. Our results will facilitate the future development of non-contact monitoring to further improve the standard of care for preterm neonates..
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:23 |
|---|---|
| Enthalten in: |
Sensors - 23(2023), 10, Artikel-ID 4910, Seite 1-29 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Gleichauf, Johanna [VerfasserIn] |
|---|
| Links: |
nbn-resolving.de [kostenfrei] |
|---|
| Themen: |
Face detection |
|---|
| Anmerkungen: |
Sonstige Körperschaft: Technische Universität Hamburg Sonstige Körperschaft: Technische Universität Hamburg, Institut für Hochfrequenztechnik |
|---|
| Umfang: |
Illustrationen, Diagramme |
|---|
| doi: |
10.15480/882.5152 10.3390/s23104910 |
|---|---|
| Weitere IDs: |
urn:nbn:de:gbv:830-882.0220865 11420/15355 |
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
1846992680 |
|---|
| LEADER | 01000naa a2200265 4500 | ||
|---|---|---|---|
| 001 | 1846992680 | ||
| 003 | DE-627 | ||
| 005 | 20230531080858.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 230531s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a urn:nbn:de:gbv:830-882.0220865 |2 urn | |
| 024 | 7 | |a 10.15480/882.5152 |2 doi | |
| 024 | 7 | |a 10.3390/s23104910 |2 doi | |
| 024 | 7 | |a 11420/15355 |2 hdl | |
| 035 | |a (DE-627)1846992680 | ||
| 035 | |a (DE-599)KXP1846992680 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 082 | 0 | |a 530: Physik | |
| 082 | 0 | |a 600: Technik | |
| 082 | 0 | |a 610: Medizin | |
| 082 | 0 | |a 620: Ingenieurwissenschaften | |
| 100 | 1 | |a Gleichauf, Johanna |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Sensor fusion for the robust detection of facial regions of neonates using neural networks |c Johanna Gleichauf, Lukas Hennemann, Fabian B. Fahlbusch, Oliver Hofmann, Christine Niebler and Alexander Koelpin |
| 264 | 1 | |c 2023 | |
| 300 | |b Illustrationen, Diagramme | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Sonstige Körperschaft: Technische Universität Hamburg | ||
| 500 | |a Sonstige Körperschaft: Technische Universität Hamburg, Institut für Hochfrequenztechnik | ||
| 506 | 0 | |q DE-830 |a Open Access |e Controlled Vocabulary for Access Rights |u http://purl.org/coar/access_right/c_abf2 | |
| 520 | |a The monitoring of vital signs and increasing patient comfort are cornerstones of modern neonatal intensive care. Commonly used monitoring methods are based on skin contact which can cause irritations and discomfort in preterm neonates. Therefore, non-contact approaches are the subject of current research aiming to resolve this dichotomy. Robust neonatal face detection is essential for the reliable detection of heart rate, respiratory rate and body temperature. While solutions for adult face detection are established, the unique neonatal proportions require a tailored approach. Additionally, sufficient open-source data of neonates on the NICU is lacking. We set out to train neural networks with the thermal-RGB-fusion data of neonates. We propose a novel indirect fusion approach including the sensor fusion of a thermal and RGB camera based on a 3D time-of-flight (ToF) camera. Unlike other approaches, this method is tailored for close distances encountered in neonatal incubators. Two neural networks were used with the fusion data and compared to RGB and thermal networks. For the class “head” we reached average precision values of 0.9958 (RetinaNet) and 0.9455 (YOLOv3) for the fusion data. Compared with the literature, similar precision was achieved, but we are the first to train a neural network with fusion data of neonates. The advantage of this approach is in calculating the detection area directly from the fusion image for the RGB and thermal modality. This increases data efficiency by 66%. Our results will facilitate the future development of non-contact monitoring to further improve the standard of care for preterm neonates. | ||
| 540 | |q DE-830 |a Namensnennung 4.0 International |f CC BY 4.0 |2 cc |u https://creativecommons.org/licenses/by/4.0/ | ||
| 650 | 7 | |a non-contact monitoring |2 DSpace | |
| 650 | 7 | |a neonates |2 DSpace | |
| 650 | 7 | |a sensor fusion |2 DSpace | |
| 650 | 7 | |a neural network |2 DSpace | |
| 650 | 7 | |a face detection |2 DSpace | |
| 700 | 1 | |a Hennemann, Lukas |e verfasserin |4 aut | |
| 700 | 1 | |a Fahlbusch, Fabian B. |d 1978- |e verfasserin |0 (DE-588)135785030 |0 (DE-627)571260098 |0 (DE-576)300645716 |4 aut | |
| 700 | 1 | |a Hofmann, Oliver |e verfasserin |4 aut | |
| 700 | 1 | |a Niebler, Christine |d 1980- |e verfasserin |0 (DE-588)136559018 |0 (DE-627)584508328 |0 (DE-576)301081409 |4 aut | |
| 700 | 1 | |a Kölpin, Alexander |e verfasserin |0 (DE-588)1197764402 |0 (DE-627)1679494147 |4 aut | |
| 710 | 2 | |a Technische Universität Hamburg |0 (DE-588)1112763473 |0 (DE-627)866918418 |0 (DE-576)476770564 |4 oth | |
| 710 | 2 | |a Technische Universität Hamburg |b Institut für Hochfrequenztechnik |0 (DE-588)112555729X |0 (DE-627)880221658 |0 (DE-576)483486655 |4 oth | |
| 773 | 0 | 8 | |i Enthalten in |t Sensors |d Basel : MDPI, 2001 |g 23(2023), 10, Artikel-ID 4910, Seite 1-29 |h Online-Ressource |w (DE-627)331640910 |w (DE-600)2052857-7 |w (DE-576)281205191 |x 1424-8220 |7 nnns |
| 773 | 1 | 8 | |g volume:23 |g year:2023 |g number:10 |g elocationid:4910 |g pages:1-29 |
| 856 | 4 | 0 | |u http://nbn-resolving.de/urn:nbn:de:gbv:830-882.0220865 |x Resolving-System |z kostenfrei |
| 856 | 4 | 0 | |u https://doi.org/10.15480/882.5152 |x Resolving-System |z kostenfrei |
| 856 | 4 | 0 | |u http://hdl.handle.net/11420/15355 |x Resolving-System |z kostenfrei |
| 856 | 4 | 0 | |u https://doi.org/10.3390/s23104910 |x Resolving-System |
| 912 | |a GBV_USEFLAG_U | ||
| 912 | |a GBV_ILN_23 | ||
| 912 | |a ISIL_DE-830 | ||
| 912 | |a SYSFLAG_1 | ||
| 912 | |a GBV_KXP | ||
| 912 | |a GBV_ILN_20 | ||
| 912 | |a GBV_ILN_22 | ||
| 912 | |a GBV_ILN_24 | ||
| 912 | |a GBV_ILN_31 | ||
| 912 | |a GBV_ILN_39 | ||
| 912 | |a GBV_ILN_40 | ||
| 912 | |a GBV_ILN_60 | ||
| 912 | |a GBV_ILN_62 | ||
| 912 | |a GBV_ILN_63 | ||
| 912 | |a GBV_ILN_65 | ||
| 912 | |a GBV_ILN_69 | ||
| 912 | |a GBV_ILN_70 | ||
| 912 | |a GBV_ILN_73 | ||
| 912 | |a GBV_ILN_95 | ||
| 912 | |a GBV_ILN_105 | ||
| 912 | |a GBV_ILN_110 | ||
| 912 | |a GBV_ILN_151 | ||
| 912 | |a GBV_ILN_161 | ||
| 912 | |a GBV_ILN_170 | ||
| 912 | |a GBV_ILN_206 | ||
| 912 | |a GBV_ILN_213 | ||
| 912 | |a GBV_ILN_230 | ||
| 912 | |a GBV_ILN_285 | ||
| 912 | |a GBV_ILN_293 | ||
| 912 | |a GBV_ILN_370 | ||
| 912 | |a GBV_ILN_602 | ||
| 912 | |a GBV_ILN_2005 | ||
| 912 | |a GBV_ILN_2009 | ||
| 912 | |a GBV_ILN_2011 | ||
| 912 | |a GBV_ILN_2014 | ||
| 912 | |a GBV_ILN_2055 | ||
| 912 | |a GBV_ILN_2057 | ||
| 912 | |a GBV_ILN_2111 | ||
| 912 | |a GBV_ILN_2507 | ||
| 912 | |a GBV_ILN_4012 | ||
| 912 | |a GBV_ILN_4037 | ||
| 912 | |a GBV_ILN_4112 | ||
| 912 | |a GBV_ILN_4125 | ||
| 912 | |a GBV_ILN_4126 | ||
| 912 | |a GBV_ILN_4249 | ||
| 912 | |a GBV_ILN_4305 | ||
| 912 | |a GBV_ILN_4306 | ||
| 912 | |a GBV_ILN_4307 | ||
| 912 | |a GBV_ILN_4313 | ||
| 912 | |a GBV_ILN_4322 | ||
| 912 | |a GBV_ILN_4323 | ||
| 912 | |a GBV_ILN_4324 | ||
| 912 | |a GBV_ILN_4325 | ||
| 912 | |a GBV_ILN_4335 | ||
| 912 | |a GBV_ILN_4338 | ||
| 912 | |a GBV_ILN_4367 | ||
| 912 | |a GBV_ILN_4700 | ||
| 912 | |a GBV_ILN_2403 | ||
| 912 | |a GBV_ILN_2403 | ||
| 912 | |a ISIL_DE-LFER | ||
| 935 | |i DSpace | ||
| 936 | u | w | |d 23 |j 2023 |e 10 |i 4910 |h 1-29 |
| 951 | |a AR | ||
| 952 | |d 23 |j 2023 |e 10 |i 4910 |h 1-29 | ||
| 953 | |2 045F |a 530: Physik | ||
| 953 | |2 045F |a 600: Technik | ||
| 953 | |2 045F |a 610: Medizin | ||
| 953 | |2 045F |a 620: Ingenieurwissenschaften | ||
| 980 | |2 23 |1 01 |x 0830 |b 4327478377 |h tuhh |k Elektronischer Volltext |u f |y z |z 31-05-23 | ||
| 980 | |2 2403 |1 01 |x DE-LFER |b 4339978515 |c 00 |f --%%-- |d --%%-- |e n |j --%%-- |y l01 |z 17-06-23 | ||
| 981 | |2 23 |1 01 |x 0830 |r https://doi.org/10.15480/882.5152 |z LF | ||
| 981 | |2 2403 |1 01 |x DE-LFER |r https://doi.org/10.3390/s23104910 | ||
| 981 | |2 2403 |1 01 |x DE-LFER |r http://nbn-resolving.de/urn:nbn:de:gbv:830-882.0220865 | ||
| 995 | |2 23 |1 01 |x 0830 |a tuhh | ||
| 995 | |2 23 |1 01 |x 0830 |a tubdok | ||