NExT-OOD : Overcoming Dual Multiple-Choice VQA Biases
In recent years, multiple-choice Visual Question Answering (VQA) has become topical and achieved remarkable progress. However, most pioneer multiple-choice VQA models are heavily driven by statistical correlations in datasets, which cannot perform well on multimodal understanding and suffer from poor generalization. In this paper, we identify two kinds of spurious correlations, i.e., a Vision-Answer bias (VA bias) and a Question-Answer bias (QA bias). To systematically and scientifically study these biases, we construct a new video question answering (videoQA) benchmark NExT-OOD in OOD setting and propose a graph-based cross-sample method for bias reduction. Specifically, the NExT-OOD is designed to quantify models' generalizability and measure their reasoning ability comprehensively. It contains three sub-datasets including NExT-OOD-VA, NExT-OOD-QA, and NExT-OOD-VQA, which are designed for the VA bias, QA bias, and VA&QA bias, respectively. We evaluate several existing multiple-choice VQA models on our NExT-OOD, and illustrate that their performance degrades significantly compared with the results obtained on the original multiple-choice VQA dataset. Besides, to mitigate the VA bias and QA bias, we explicitly consider the cross-sample information and design a contrastive graph matching loss in our approach, which provides adequate debiasing guidance from the perspective of whole dataset, and encourages the model to focus on multimodal contents instead of spurious statistical regularities. Extensive experimental results illustrate that our method significantly outperforms other bias reduction strategies, demonstrating the effectiveness and generalizability of the proposed approach.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:46 |
|---|---|
| Enthalten in: |
IEEE transactions on pattern analysis and machine intelligence - 46(2024), 4 vom: 01. März, Seite 1913-1931 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Zhang, Xi [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Revised 07.03.2024 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1109/TPAMI.2023.3269429 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM35597424X |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM35597424X | ||
| 003 | DE-627 | ||
| 005 | 20240307231908.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TPAMI.2023.3269429 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1319.xml |
| 035 | |a (DE-627)NLM35597424X | ||
| 035 | |a (NLM)37093718 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Zhang, Xi |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a NExT-OOD |b Overcoming Dual Multiple-Choice VQA Biases |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 07.03.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a In recent years, multiple-choice Visual Question Answering (VQA) has become topical and achieved remarkable progress. However, most pioneer multiple-choice VQA models are heavily driven by statistical correlations in datasets, which cannot perform well on multimodal understanding and suffer from poor generalization. In this paper, we identify two kinds of spurious correlations, i.e., a Vision-Answer bias (VA bias) and a Question-Answer bias (QA bias). To systematically and scientifically study these biases, we construct a new video question answering (videoQA) benchmark NExT-OOD in OOD setting and propose a graph-based cross-sample method for bias reduction. Specifically, the NExT-OOD is designed to quantify models' generalizability and measure their reasoning ability comprehensively. It contains three sub-datasets including NExT-OOD-VA, NExT-OOD-QA, and NExT-OOD-VQA, which are designed for the VA bias, QA bias, and VA&QA bias, respectively. We evaluate several existing multiple-choice VQA models on our NExT-OOD, and illustrate that their performance degrades significantly compared with the results obtained on the original multiple-choice VQA dataset. Besides, to mitigate the VA bias and QA bias, we explicitly consider the cross-sample information and design a contrastive graph matching loss in our approach, which provides adequate debiasing guidance from the perspective of whole dataset, and encourages the model to focus on multimodal contents instead of spurious statistical regularities. Extensive experimental results illustrate that our method significantly outperforms other bias reduction strategies, demonstrating the effectiveness and generalizability of the proposed approach | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Zhang, Feifei |e verfasserin |4 aut | |
| 700 | 1 | |a Xu, Changsheng |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t IEEE transactions on pattern analysis and machine intelligence |d 1979 |g 46(2024), 4 vom: 01. März, Seite 1913-1931 |w (DE-627)NLM098212257 |x 1939-3539 |7 nnns |
| 773 | 1 | 8 | |g volume:46 |g year:2024 |g number:4 |g day:01 |g month:03 |g pages:1913-1931 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1109/TPAMI.2023.3269429 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 46 |j 2024 |e 4 |b 01 |c 03 |h 1913-1931 | ||