Deep M2CDL : Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network
For multi-modal image processing, network interpretability is essential due to the complicated dependency across modalities. Recently, a promising research direction for interpretable network is to incorporate dictionary learning into deep learning through unfolding strategy. However, the existing multi-modal dictionary learning models are both single-layer and single-scale, which restricts the representation ability. In this paper, we first introduce a multi-scale multi-modal convolutional dictionary learning ( M2CDL) model, which is performed in a multi-layer strategy, to associate different image modalities in a coarse-to-fine manner. Then, we propose a unified framework namely Deep M2CDL derived from the M2CDL model for both multi-modal image restoration (MIR) and multi-modal image fusion (MIF) tasks. The network architecture of Deep M2CDL fully matches the optimization steps of the M2CDL model, which makes each network module with good interpretability. Different from handcrafted priors, both the dictionary and sparse feature priors are learned through the network. The performance of the proposed Deep M2CDL is evaluated on a wide variety of MIR and MIF tasks, which shows the superiority of it over many state-of-the-art methods both quantitatively and qualitatively. In addition, we also visualize the multi-modal sparse features and dictionary filters learned from the network, which demonstrates the good interpretability of the Deep M2CDL network.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:46 |
|---|---|
| Enthalten in: |
IEEE transactions on pattern analysis and machine intelligence - 46(2024), 5 vom: 18. Apr., Seite 2770-2787 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Deng, Xin [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Revised 05.04.2024 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1109/TPAMI.2023.3334624 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM364746785 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM364746785 | ||
| 003 | DE-627 | ||
| 005 | 20240405233224.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TPAMI.2023.3334624 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1366.xml |
| 035 | |a (DE-627)NLM364746785 | ||
| 035 | |a (NLM)37983156 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Deng, Xin |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Deep M2CDL |b Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 05.04.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a For multi-modal image processing, network interpretability is essential due to the complicated dependency across modalities. Recently, a promising research direction for interpretable network is to incorporate dictionary learning into deep learning through unfolding strategy. However, the existing multi-modal dictionary learning models are both single-layer and single-scale, which restricts the representation ability. In this paper, we first introduce a multi-scale multi-modal convolutional dictionary learning ( M2CDL) model, which is performed in a multi-layer strategy, to associate different image modalities in a coarse-to-fine manner. Then, we propose a unified framework namely Deep M2CDL derived from the M2CDL model for both multi-modal image restoration (MIR) and multi-modal image fusion (MIF) tasks. The network architecture of Deep M2CDL fully matches the optimization steps of the M2CDL model, which makes each network module with good interpretability. Different from handcrafted priors, both the dictionary and sparse feature priors are learned through the network. The performance of the proposed Deep M2CDL is evaluated on a wide variety of MIR and MIF tasks, which shows the superiority of it over many state-of-the-art methods both quantitatively and qualitatively. In addition, we also visualize the multi-modal sparse features and dictionary filters learned from the network, which demonstrates the good interpretability of the Deep M2CDL network | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Xu, Jingyi |e verfasserin |4 aut | |
| 700 | 1 | |a Gao, Fangyuan |e verfasserin |4 aut | |
| 700 | 1 | |a Sun, Xiancheng |e verfasserin |4 aut | |
| 700 | 1 | |a Xu, Mai |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t IEEE transactions on pattern analysis and machine intelligence |d 1979 |g 46(2024), 5 vom: 18. Apr., Seite 2770-2787 |w (DE-627)NLM098212257 |x 1939-3539 |7 nnns |
| 773 | 1 | 8 | |g volume:46 |g year:2024 |g number:5 |g day:18 |g month:04 |g pages:2770-2787 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1109/TPAMI.2023.3334624 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 46 |j 2024 |e 5 |b 18 |c 04 |h 2770-2787 | ||