SigmoReLU: An Improvement Activation Function by Combining Sigmoid and ReLU<strong> </strong>
Two of the most common activation functions (AF) in deep neural networks (DNN) training are Sigmoid and ReLU. Sigmoid was tend to be more popular the previous decades, but it was suffering with the common vanishing gradient problems. ReLU has resolved these problems by using zero gradient and not tiny values for negative weights and the value &ldquo;1&rdquo; for all positives. Although it significant resolves the vanishing of the gradients, it poses new issues with dying neurons of the zero values. Recent approaches for improvements are in a similar direction by just proposing variations of the AF, such as Leaky ReLU (LReLU), while maintaining the solution within the same unresolved gradient problems. In this paper, the combining of the Sigmoid and ReLU in one single function is proposed, as a way to take the advantages of the two. The experimental results demonstrate that by using the ReLU&rsquo;s gradient solution on positive weights, and Sigmoid&rsquo;s gradient solution on negatives, has a significant improvement on performance of training Neural Networks on image classification of diseases such as COVID-19, text and tabular data classification tasks on five different datasets..
| Medienart: |
|
|---|
Preprint| Erscheinungsjahr: |
2021 |
|---|---|
| Erschienen: |
2021 |
| Enthalten in: |
Preprints.org - (2021) vom: 10. Juni Zur Gesamtaufnahme - year:2021 |
|---|
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Mastromichalakis, Stamatis [VerfasserIn] |
|---|
| Links: |
Volltext [kostenfrei] |
|---|
| Themen: |
|---|
| doi: |
10.20944/preprints202106.0252.v1 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
preprintsorg031970117 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | preprintsorg031970117 | ||
| 003 | DE-627 | ||
| 005 | 20230429204026.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 210611s2021 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.20944/preprints202106.0252.v1 |2 doi | |
| 035 | |a (DE-627)preprintsorg031970117 | ||
| 035 | |a (preprintsorg)10.20944/preprints202106.0252 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Mastromichalakis, Stamatis |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a SigmoReLU: An Improvement Activation Function by Combining Sigmoid and ReLU<strong> </strong> |
| 264 | 1 | |c 2021 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 520 | |a Two of the most common activation functions (AF) in deep neural networks (DNN) training are Sigmoid and ReLU. Sigmoid was tend to be more popular the previous decades, but it was suffering with the common vanishing gradient problems. ReLU has resolved these problems by using zero gradient and not tiny values for negative weights and the value &ldquo;1&rdquo; for all positives. Although it significant resolves the vanishing of the gradients, it poses new issues with dying neurons of the zero values. Recent approaches for improvements are in a similar direction by just proposing variations of the AF, such as Leaky ReLU (LReLU), while maintaining the solution within the same unresolved gradient problems. In this paper, the combining of the Sigmoid and ReLU in one single function is proposed, as a way to take the advantages of the two. The experimental results demonstrate that by using the ReLU&rsquo;s gradient solution on positive weights, and Sigmoid&rsquo;s gradient solution on negatives, has a significant improvement on performance of training Neural Networks on image classification of diseases such as COVID-19, text and tabular data classification tasks on five different datasets. | ||
| 650 | 4 | |a Engineering | |
| 650 | 4 | |a 620 | |
| 773 | 0 | 8 | |i Enthalten in |t Preprints.org |g (2021) vom: 10. Juni |
| 773 | 1 | 8 | |g year:2021 |g day:10 |g month:06 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.20944/preprints202106.0252.v1 |z kostenfrei |3 Volltext |
| 912 | |a preprintsorg | ||
| 912 | |a SSG-OLC-PHA | ||
| 951 | |a AR | ||
| 952 | |j 2021 |b 10 |c 06 | ||