FMixCutMatch for semi-supervised deep learning
Copyright © 2020 Elsevier Ltd. All rights reserved..
Mixed sample augmentation (MSA) has witnessed great success in the research area of semi-supervised learning (SSL) and is performed by mixing two training samples as an augmentation strategy to effectively smooth the training space. Following the insights on the efficacy of cut-mix in particular, we propose FMixCut, an MSA that combines Fourier space-based data mixing (FMix) and the proposed Fourier space-based data cutting (FCut) for labeled and unlabeled data augmentation. Specifically, for the SSL task, our approach first generates soft pseudo-labels using the model's previous predictions. The model is then trained to penalize the outputs of the FMix-generated samples so that they are consistent with their mixed soft pseudo-labels. In addition, we propose to use FCut, a new Cutout-based data augmentation strategy that adopts the two masked sample pairs from FMix for weighted cross-entropy minimization. Furthermore, by implementing two regularization techniques, namely, batch label distribution entropy maximization and sample confidence entropy minimization, we further boost the training efficiency. Finally, we introduce a dynamic labeled-unlabeled data mixing (DDM) strategy to further accelerate the convergence of the model. Combining the above process, we finally call our SSL approach as "FMixCutMatch", in short FMCmatch. As a result, the proposed FMCmatch achieves state-of-the-art performance on CIFAR-10/100, SVHN and Mini-Imagenet across a variety of SSL conditions with the CNN-13, WRN-28-2 and ResNet-18 networks. In particular, our method achieves a 4.54% test error on CIFAR-10 with 4K labels under the CNN-13 and a 41.25% Top-1 test error on Mini-Imagenet with 10K labels under the ResNet-18. Our codes for reproducing these results are publicly available at https://github.com/biuyq/FMixCutMatch.
Medienart: |
E-Artikel |
---|
Erscheinungsjahr: |
2021 |
---|---|
Erschienen: |
2021 |
Enthalten in: |
Zur Gesamtaufnahme - volume:133 |
---|---|
Enthalten in: |
Neural networks : the official journal of the International Neural Network Society - 133(2021) vom: 01. Jan., Seite 166-176 |
Sprache: |
Englisch |
---|
Beteiligte Personen: |
Wei, Xiang [VerfasserIn] |
---|
Links: |
---|
Themen: |
Journal Article |
---|
Anmerkungen: |
Date Completed 15.02.2021 Date Revised 15.02.2021 published: Print-Electronic Citation Status MEDLINE |
---|
doi: |
10.1016/j.neunet.2020.10.018 |
---|
funding: |
|
---|---|
Förderinstitution / Projekttitel: |
|
PPN (Katalog-ID): |
NLM317847325 |
---|
LEADER | 01000naa a22002652 4500 | ||
---|---|---|---|
001 | NLM317847325 | ||
003 | DE-627 | ||
005 | 20231225164124.0 | ||
007 | cr uuu---uuuuu | ||
008 | 231225s2021 xx |||||o 00| ||eng c | ||
024 | 7 | |a 10.1016/j.neunet.2020.10.018 |2 doi | |
028 | 5 | 2 | |a pubmed24n1059.xml |
035 | |a (DE-627)NLM317847325 | ||
035 | |a (NLM)33217685 | ||
035 | |a (PII)S0893-6080(20)30381-6 | ||
040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
041 | |a eng | ||
100 | 1 | |a Wei, Xiang |e verfasserin |4 aut | |
245 | 1 | 0 | |a FMixCutMatch for semi-supervised deep learning |
264 | 1 | |c 2021 | |
336 | |a Text |b txt |2 rdacontent | ||
337 | |a ƒaComputermedien |b c |2 rdamedia | ||
338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
500 | |a Date Completed 15.02.2021 | ||
500 | |a Date Revised 15.02.2021 | ||
500 | |a published: Print-Electronic | ||
500 | |a Citation Status MEDLINE | ||
520 | |a Copyright © 2020 Elsevier Ltd. All rights reserved. | ||
520 | |a Mixed sample augmentation (MSA) has witnessed great success in the research area of semi-supervised learning (SSL) and is performed by mixing two training samples as an augmentation strategy to effectively smooth the training space. Following the insights on the efficacy of cut-mix in particular, we propose FMixCut, an MSA that combines Fourier space-based data mixing (FMix) and the proposed Fourier space-based data cutting (FCut) for labeled and unlabeled data augmentation. Specifically, for the SSL task, our approach first generates soft pseudo-labels using the model's previous predictions. The model is then trained to penalize the outputs of the FMix-generated samples so that they are consistent with their mixed soft pseudo-labels. In addition, we propose to use FCut, a new Cutout-based data augmentation strategy that adopts the two masked sample pairs from FMix for weighted cross-entropy minimization. Furthermore, by implementing two regularization techniques, namely, batch label distribution entropy maximization and sample confidence entropy minimization, we further boost the training efficiency. Finally, we introduce a dynamic labeled-unlabeled data mixing (DDM) strategy to further accelerate the convergence of the model. Combining the above process, we finally call our SSL approach as "FMixCutMatch", in short FMCmatch. As a result, the proposed FMCmatch achieves state-of-the-art performance on CIFAR-10/100, SVHN and Mini-Imagenet across a variety of SSL conditions with the CNN-13, WRN-28-2 and ResNet-18 networks. In particular, our method achieves a 4.54% test error on CIFAR-10 with 4K labels under the CNN-13 and a 41.25% Top-1 test error on Mini-Imagenet with 10K labels under the ResNet-18. Our codes for reproducing these results are publicly available at https://github.com/biuyq/FMixCutMatch | ||
650 | 4 | |a Journal Article | |
650 | 4 | |a Mixed sample augmentation | |
650 | 4 | |a Regularization | |
650 | 4 | |a Semi-supervised learning | |
650 | 4 | |a Soft pseudo-labels | |
700 | 1 | |a Wei, Xiaotao |e verfasserin |4 aut | |
700 | 1 | |a Kong, Xiangyuan |e verfasserin |4 aut | |
700 | 1 | |a Lu, Siyang |e verfasserin |4 aut | |
700 | 1 | |a Xing, Weiwei |e verfasserin |4 aut | |
700 | 1 | |a Lu, Wei |e verfasserin |4 aut | |
773 | 0 | 8 | |i Enthalten in |t Neural networks : the official journal of the International Neural Network Society |d 1996 |g 133(2021) vom: 01. Jan., Seite 166-176 |w (DE-627)NLM087746824 |x 1879-2782 |7 nnns |
773 | 1 | 8 | |g volume:133 |g year:2021 |g day:01 |g month:01 |g pages:166-176 |
856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.neunet.2020.10.018 |3 Volltext |
912 | |a GBV_USEFLAG_A | ||
912 | |a GBV_NLM | ||
951 | |a AR | ||
952 | |d 133 |j 2021 |b 01 |c 01 |h 166-176 |