False Correlation Reduction for Offline Reinforcement Learning
Offline reinforcement learning (RL) harnesses the power of massive datasets for resolving sequential decision problems. Most existing papers only discuss defending against out-of-distribution (OOD) actions while we investigate a broader issue, the false correlations between epistemic uncertainty and decision-making, an essential factor that causes suboptimality. In this paper, we propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL). The proposed algorithm introduces an annealing behavior cloning regularizer to help produce a high-quality estimation of uncertainty which is critical for eliminating false correlations from suboptimality. Theoretically, we justify the rationality of the proposed method and prove its convergence to the optimal policy with a sublinear rate under mild assumptions.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:46 |
|---|---|
| Enthalten in: |
IEEE transactions on pattern analysis and machine intelligence - 46(2024), 2 vom: 27. Jan., Seite 1199-1211 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Deng, Zhihong [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Revised 12.01.2024 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1109/TPAMI.2023.3328397 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM363950591 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM363950591 | ||
| 003 | DE-627 | ||
| 005 | 20240114234752.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TPAMI.2023.3328397 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1258.xml |
| 035 | |a (DE-627)NLM363950591 | ||
| 035 | |a (NLM)37903051 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Deng, Zhihong |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a False Correlation Reduction for Offline Reinforcement Learning |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 12.01.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a Offline reinforcement learning (RL) harnesses the power of massive datasets for resolving sequential decision problems. Most existing papers only discuss defending against out-of-distribution (OOD) actions while we investigate a broader issue, the false correlations between epistemic uncertainty and decision-making, an essential factor that causes suboptimality. In this paper, we propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL). The proposed algorithm introduces an annealing behavior cloning regularizer to help produce a high-quality estimation of uncertainty which is critical for eliminating false correlations from suboptimality. Theoretically, we justify the rationality of the proposed method and prove its convergence to the optimal policy with a sublinear rate under mild assumptions | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Fu, Zuyue |e verfasserin |4 aut | |
| 700 | 1 | |a Wang, Lingxiao |e verfasserin |4 aut | |
| 700 | 1 | |a Yang, Zhuoran |e verfasserin |4 aut | |
| 700 | 1 | |a Bai, Chenjia |e verfasserin |4 aut | |
| 700 | 1 | |a Zhou, Tianyi |e verfasserin |4 aut | |
| 700 | 1 | |a Wang, Zhaoran |e verfasserin |4 aut | |
| 700 | 1 | |a Jiang, Jing |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t IEEE transactions on pattern analysis and machine intelligence |d 1979 |g 46(2024), 2 vom: 27. Jan., Seite 1199-1211 |w (DE-627)NLM098212257 |x 1939-3539 |7 nnns |
| 773 | 1 | 8 | |g volume:46 |g year:2024 |g number:2 |g day:27 |g month:01 |g pages:1199-1211 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1109/TPAMI.2023.3328397 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 46 |j 2024 |e 2 |b 27 |c 01 |h 1199-1211 | ||