Exploring sparsity in graph transformers
Copyright © 2024 Elsevier Ltd. All rights reserved..
Graph Transformers (GTs) have achieved impressive results on various graph-related tasks. However, the huge computational cost of GTs hinders their deployment and application, especially in resource-constrained environments. Therefore, in this paper, we explore the feasibility of sparsifying GTs, a significant yet under-explored topic. We first discuss the redundancy of GTs based on the characteristics of existing GT models, and then propose a comprehensive Graph Transformer SParsification (GTSP) framework that helps to reduce the computational complexity of GTs from four dimensions: the input graph data, attention heads, model layers, and model weights. Specifically, GTSP designs differentiable masks for each individual compressible component, enabling effective end-to-end pruning. We examine our GTSP through extensive experiments on prominent GTs, including GraphTrans, Graphormer, and GraphGPS. The experimental results demonstrate that GTSP effectively reduces computational costs, with only marginal decreases in accuracy or, in some instances, even improvements. For example, GTSP results in a 30% reduction in Floating Point Operations while contributing to a 1.8% increase in Area Under the Curve accuracy on the OGBG-HIV dataset. Furthermore, we provide several insights on the characteristics of attention heads and the behavior of attention mechanisms, all of which have immense potential to inspire future research endeavors in this domain. Our code is available at https://github.com/LiuChuang0059/GTSP.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:174 |
|---|---|
| Enthalten in: |
Neural networks : the official journal of the International Neural Network Society - 174(2024) vom: 01. Apr., Seite 106265 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Liu, Chuang [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Graph classification |
|---|
| Anmerkungen: |
Date Completed 15.04.2024 Date Revised 15.04.2024 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1016/j.neunet.2024.106265 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM370418786 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM370418786 | ||
| 003 | DE-627 | ||
| 005 | 20240415233609.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240331s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.neunet.2024.106265 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1376.xml |
| 035 | |a (DE-627)NLM370418786 | ||
| 035 | |a (NLM)38552351 | ||
| 035 | |a (PII)S0893-6080(24)00189-8 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Liu, Chuang |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Exploring sparsity in graph transformers |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 15.04.2024 | ||
| 500 | |a Date Revised 15.04.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a Copyright © 2024 Elsevier Ltd. All rights reserved. | ||
| 520 | |a Graph Transformers (GTs) have achieved impressive results on various graph-related tasks. However, the huge computational cost of GTs hinders their deployment and application, especially in resource-constrained environments. Therefore, in this paper, we explore the feasibility of sparsifying GTs, a significant yet under-explored topic. We first discuss the redundancy of GTs based on the characteristics of existing GT models, and then propose a comprehensive Graph Transformer SParsification (GTSP) framework that helps to reduce the computational complexity of GTs from four dimensions: the input graph data, attention heads, model layers, and model weights. Specifically, GTSP designs differentiable masks for each individual compressible component, enabling effective end-to-end pruning. We examine our GTSP through extensive experiments on prominent GTs, including GraphTrans, Graphormer, and GraphGPS. The experimental results demonstrate that GTSP effectively reduces computational costs, with only marginal decreases in accuracy or, in some instances, even improvements. For example, GTSP results in a 30% reduction in Floating Point Operations while contributing to a 1.8% increase in Area Under the Curve accuracy on the OGBG-HIV dataset. Furthermore, we provide several insights on the characteristics of attention heads and the behavior of attention mechanisms, all of which have immense potential to inspire future research endeavors in this domain. Our code is available at https://github.com/LiuChuang0059/GTSP | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Graph classification | |
| 650 | 4 | |a Graph sparse training | |
| 650 | 4 | |a Graph transformers | |
| 650 | 4 | |a Model pruning | |
| 700 | 1 | |a Zhan, Yibing |e verfasserin |4 aut | |
| 700 | 1 | |a Ma, Xueqi |e verfasserin |4 aut | |
| 700 | 1 | |a Ding, Liang |e verfasserin |4 aut | |
| 700 | 1 | |a Tao, Dapeng |e verfasserin |4 aut | |
| 700 | 1 | |a Wu, Jia |e verfasserin |4 aut | |
| 700 | 1 | |a Hu, Wenbin |e verfasserin |4 aut | |
| 700 | 1 | |a Du, Bo |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Neural networks : the official journal of the International Neural Network Society |d 1996 |g 174(2024) vom: 01. Apr., Seite 106265 |w (DE-627)NLM087746824 |x 1879-2782 |7 nnns |
| 773 | 1 | 8 | |g volume:174 |g year:2024 |g day:01 |g month:04 |g pages:106265 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.neunet.2024.106265 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 174 |j 2024 |b 01 |c 04 |h 106265 | ||