Self-Adaptive Training : Bridging Supervised and Self-Supervised Learning
We propose self-adaptive training-a unified training algorithm that dynamically calibrates and enhances training processes by model predictions without incurring an extra computational cost-to advance both supervised and self-supervised learning of deep neural networks. We analyze the training dynamics of deep networks on training data that are corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of any label information, highlighting that model predictions could substantially benefit the training processes: self-adaptive training improves the generalization of deep networks under noise and enhances the self-supervised representation learning. The analysis also sheds light on understanding deep learning, e.g., a potential explanation of the recently-discovered double-descent phenomenon in empirical risk minimization and the collapsing issue of the state-of-the-art self-supervised learning algorithms. Experiments on the CIFAR, STL, and ImageNet datasets verify the effectiveness of our approach in three applications: classification with label noise, selective classification, and linear evaluation. To facilitate future research, the code has been made publicly available at https://github.com/LayneH/self-adaptive-training.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:46 |
|---|---|
| Enthalten in: |
IEEE transactions on pattern analysis and machine intelligence - 46(2024), 3 vom: 16. Feb., Seite 1362-1377 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Huang, Lang [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Revised 07.02.2024 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1109/TPAMI.2022.3217792 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM348206194 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM348206194 | ||
| 003 | DE-627 | ||
| 005 | 20240207231957.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TPAMI.2022.3217792 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1283.xml |
| 035 | |a (DE-627)NLM348206194 | ||
| 035 | |a (NLM)36306295 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Huang, Lang |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Self-Adaptive Training |b Bridging Supervised and Self-Supervised Learning |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 07.02.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a We propose self-adaptive training-a unified training algorithm that dynamically calibrates and enhances training processes by model predictions without incurring an extra computational cost-to advance both supervised and self-supervised learning of deep neural networks. We analyze the training dynamics of deep networks on training data that are corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of any label information, highlighting that model predictions could substantially benefit the training processes: self-adaptive training improves the generalization of deep networks under noise and enhances the self-supervised representation learning. The analysis also sheds light on understanding deep learning, e.g., a potential explanation of the recently-discovered double-descent phenomenon in empirical risk minimization and the collapsing issue of the state-of-the-art self-supervised learning algorithms. Experiments on the CIFAR, STL, and ImageNet datasets verify the effectiveness of our approach in three applications: classification with label noise, selective classification, and linear evaluation. To facilitate future research, the code has been made publicly available at https://github.com/LayneH/self-adaptive-training | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Zhang, Chao |e verfasserin |4 aut | |
| 700 | 1 | |a Zhang, Hongyang |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t IEEE transactions on pattern analysis and machine intelligence |d 1979 |g 46(2024), 3 vom: 16. Feb., Seite 1362-1377 |w (DE-627)NLM098212257 |x 1939-3539 |7 nnns |
| 773 | 1 | 8 | |g volume:46 |g year:2024 |g number:3 |g day:16 |g month:02 |g pages:1362-1377 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1109/TPAMI.2022.3217792 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 46 |j 2024 |e 3 |b 16 |c 02 |h 1362-1377 | ||