Exploring the Temporal Consistency of Arbitrary Style Transfer : A Channelwise Perspective
Arbitrary image stylization by neural networks has become a popular topic, and video stylization is attracting more attention as an extension of image stylization. However, when image stylization methods are applied to videos, unsatisfactory results that suffer from severe flickering effects appear. In this article, we conducted a detailed and comprehensive analysis of the cause of such flickering effects. Systematic comparisons among typical neural style transfer approaches show that the feature migration modules for state-of-the-art (SOTA) learning systems are ill-conditioned and could lead to a channelwise misalignment between the input content representations and the generated frames. Unlike traditional methods that relieve the misalignment via additional optical flow constraints or regularization modules, we focus on keeping the temporal consistency by aligning each output frame with the input frame. To this end, we propose a simple yet efficient multichannel correlation network (MCCNet), to ensure that output frames are directly aligned with inputs in the hidden feature space while maintaining the desired style patterns. An inner channel similarity loss is adopted to eliminate side effects caused by the absence of nonlinear operations such as softmax for strict alignment. Furthermore, to improve the performance of MCCNet under complex light conditions, we introduce an illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in arbitrary video and image style transfer tasks. Code is available at https://github.com/kongxiuxiu/MCCNetV2.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:PP |
|---|---|
| Enthalten in: |
IEEE transactions on neural networks and learning systems - PP(2023) vom: 06. Jan. |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Kong, Xiaoyu [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Revised 17.02.2024 published: Print-Electronic Citation Status Publisher |
|---|
| doi: |
10.1109/TNNLS.2022.3230084 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM355232715 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM355232715 | ||
| 003 | DE-627 | ||
| 005 | 20240217232150.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TNNLS.2022.3230084 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1297.xml |
| 035 | |a (DE-627)NLM355232715 | ||
| 035 | |a (NLM)37018565 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Kong, Xiaoyu |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Exploring the Temporal Consistency of Arbitrary Style Transfer |b A Channelwise Perspective |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 17.02.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status Publisher | ||
| 520 | |a Arbitrary image stylization by neural networks has become a popular topic, and video stylization is attracting more attention as an extension of image stylization. However, when image stylization methods are applied to videos, unsatisfactory results that suffer from severe flickering effects appear. In this article, we conducted a detailed and comprehensive analysis of the cause of such flickering effects. Systematic comparisons among typical neural style transfer approaches show that the feature migration modules for state-of-the-art (SOTA) learning systems are ill-conditioned and could lead to a channelwise misalignment between the input content representations and the generated frames. Unlike traditional methods that relieve the misalignment via additional optical flow constraints or regularization modules, we focus on keeping the temporal consistency by aligning each output frame with the input frame. To this end, we propose a simple yet efficient multichannel correlation network (MCCNet), to ensure that output frames are directly aligned with inputs in the hidden feature space while maintaining the desired style patterns. An inner channel similarity loss is adopted to eliminate side effects caused by the absence of nonlinear operations such as softmax for strict alignment. Furthermore, to improve the performance of MCCNet under complex light conditions, we introduce an illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in arbitrary video and image style transfer tasks. Code is available at https://github.com/kongxiuxiu/MCCNetV2 | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Deng, Yingying |e verfasserin |4 aut | |
| 700 | 1 | |a Tang, Fan |e verfasserin |4 aut | |
| 700 | 1 | |a Dong, Weiming |e verfasserin |4 aut | |
| 700 | 1 | |a Ma, Chongyang |e verfasserin |4 aut | |
| 700 | 1 | |a Chen, Yongyong |e verfasserin |4 aut | |
| 700 | 1 | |a He, Zhenyu |e verfasserin |4 aut | |
| 700 | 1 | |a Xu, Changsheng |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t IEEE transactions on neural networks and learning systems |d 2012 |g PP(2023) vom: 06. Jan. |w (DE-627)NLM23236897X |x 2162-2388 |7 nnns |
| 773 | 1 | 8 | |g volume:PP |g year:2023 |g day:06 |g month:01 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1109/TNNLS.2022.3230084 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d PP |j 2023 |b 06 |c 01 | ||