Knowledge-Based Embodied Question Answering
In this paper, we propose a novel Knowledge-based Embodied Question Answering (K-EQA) task, in which the agent intelligently explores the environment to answer various questions with the knowledge. Different from explicitly specifying the target object in the question as existing EQA work, the agent can resort to external knowledge to understand more complicated question such as "Please tell me what are objects used to cut food in the room?", in which the agent must know the knowledge such as "knife is used for cutting food". To address this K-EQA problem, a novel framework based on neural program synthesis reasoning is proposed, where the joint reasoning of the external knowledge and 3D scene graph is performed to realize navigation and question answering. Especially, the 3D scene graph can provide the memory to store the visual information of visited scenes, which significantly improves the efficiency for the multi-turn question answering. Experimental results have demonstrated that the proposed framework is capable of answering more complicated and realistic questions in the embodied environment. The proposed method is also applicable to multi-agent scenarios.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:45 |
|---|---|
| Enthalten in: |
IEEE transactions on pattern analysis and machine intelligence - 45(2023), 10 vom: 17. Okt., Seite 11948-11960 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Tan, Sinan [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Revised 06.09.2023 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1109/TPAMI.2023.3277206 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM356984214 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM356984214 | ||
| 003 | DE-627 | ||
| 005 | 20231226071619.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TPAMI.2023.3277206 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1189.xml |
| 035 | |a (DE-627)NLM356984214 | ||
| 035 | |a (NLM)37195849 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Tan, Sinan |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Knowledge-Based Embodied Question Answering |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 06.09.2023 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a In this paper, we propose a novel Knowledge-based Embodied Question Answering (K-EQA) task, in which the agent intelligently explores the environment to answer various questions with the knowledge. Different from explicitly specifying the target object in the question as existing EQA work, the agent can resort to external knowledge to understand more complicated question such as "Please tell me what are objects used to cut food in the room?", in which the agent must know the knowledge such as "knife is used for cutting food". To address this K-EQA problem, a novel framework based on neural program synthesis reasoning is proposed, where the joint reasoning of the external knowledge and 3D scene graph is performed to realize navigation and question answering. Especially, the 3D scene graph can provide the memory to store the visual information of visited scenes, which significantly improves the efficiency for the multi-turn question answering. Experimental results have demonstrated that the proposed framework is capable of answering more complicated and realistic questions in the embodied environment. The proposed method is also applicable to multi-agent scenarios | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Ge, Mengmeng |e verfasserin |4 aut | |
| 700 | 1 | |a Guo, Di |e verfasserin |4 aut | |
| 700 | 1 | |a Liu, Huaping |e verfasserin |4 aut | |
| 700 | 1 | |a Sun, Fuchun |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t IEEE transactions on pattern analysis and machine intelligence |d 1979 |g 45(2023), 10 vom: 17. Okt., Seite 11948-11960 |w (DE-627)NLM098212257 |x 1939-3539 |7 nnns |
| 773 | 1 | 8 | |g volume:45 |g year:2023 |g number:10 |g day:17 |g month:10 |g pages:11948-11960 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1109/TPAMI.2023.3277206 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 45 |j 2023 |e 10 |b 17 |c 10 |h 11948-11960 | ||