Two-stream vision transformer based multi-label recognition for TCM prescriptions construction
Copyright © 2024. Published by Elsevier Ltd..
Traditional Chinese medicine (TCM) observation diagnosis images (including facial and tongue images) provide essential human body information, holding significant importance in clinical medicine for diagnosis and treatment. TCM prescriptions, known for their simplicity, non-invasiveness, and low side effects, have been widely applied worldwide. Exploring automated herbal prescription construction based on visual diagnosis holds vital value in delving into the correlation between external features and herbal prescriptions and offering medical services in mobile healthcare systems. To effectively integrate multi-perspective visual diagnosis images and automate prescription construction, this study proposes a multi-herb recommendation framework based on Visual Transformer and multi-label classification. The framework comprises three key components: image encoder, label embedding module, and cross-modal fusion classification module. The image encoder employs a dual-stream Visual Transformer to learn dependencies between different regions of input images, capturing both local and global features. The label embedding module utilizes Graph Convolutional Networks to capture associations between diverse herbal labels. Finally, two Multi-Modal Factorized Bilinear modules are introduced as effective components to fuse cross-modal vectors, creating an end-to-end multi-label image-herb recommendation model. Through experimentation with real facial and tongue images and generating prescription data closely resembling real samples. The precision is 50.06 %, the recall rate is 48.33 %, and the F1-score is 49.18 %. This study validates the feasibility of automated herbal prescription construction from the perspective of visual diagnosis. Simultaneously, it provides valuable insights for constructing herbal prescriptions automatically from more physical information.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:170 |
|---|---|
| Enthalten in: |
Computers in biology and medicine - 170(2024) vom: 20. Feb., Seite 107920 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Zhao, Zijuan [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Facial and tongue images |
|---|
| Anmerkungen: |
Date Completed 28.02.2024 Date Revised 28.02.2024 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1016/j.compbiomed.2024.107920 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM367349639 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM367349639 | ||
| 003 | DE-627 | ||
| 005 | 20240229170720.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240121s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.compbiomed.2024.107920 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1310.xml |
| 035 | |a (DE-627)NLM367349639 | ||
| 035 | |a (NLM)38244474 | ||
| 035 | |a (PII)S0010-4825(24)00004-0 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Zhao, Zijuan |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Two-stream vision transformer based multi-label recognition for TCM prescriptions construction |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 28.02.2024 | ||
| 500 | |a Date Revised 28.02.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a Copyright © 2024. Published by Elsevier Ltd. | ||
| 520 | |a Traditional Chinese medicine (TCM) observation diagnosis images (including facial and tongue images) provide essential human body information, holding significant importance in clinical medicine for diagnosis and treatment. TCM prescriptions, known for their simplicity, non-invasiveness, and low side effects, have been widely applied worldwide. Exploring automated herbal prescription construction based on visual diagnosis holds vital value in delving into the correlation between external features and herbal prescriptions and offering medical services in mobile healthcare systems. To effectively integrate multi-perspective visual diagnosis images and automate prescription construction, this study proposes a multi-herb recommendation framework based on Visual Transformer and multi-label classification. The framework comprises three key components: image encoder, label embedding module, and cross-modal fusion classification module. The image encoder employs a dual-stream Visual Transformer to learn dependencies between different regions of input images, capturing both local and global features. The label embedding module utilizes Graph Convolutional Networks to capture associations between diverse herbal labels. Finally, two Multi-Modal Factorized Bilinear modules are introduced as effective components to fuse cross-modal vectors, creating an end-to-end multi-label image-herb recommendation model. Through experimentation with real facial and tongue images and generating prescription data closely resembling real samples. The precision is 50.06 %, the recall rate is 48.33 %, and the F1-score is 49.18 %. This study validates the feasibility of automated herbal prescription construction from the perspective of visual diagnosis. Simultaneously, it provides valuable insights for constructing herbal prescriptions automatically from more physical information | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Facial and tongue images | |
| 650 | 4 | |a Graph convolutional network | |
| 650 | 4 | |a Multi-label image recognition | |
| 650 | 4 | |a Prescriptions construction | |
| 650 | 4 | |a Visual transformer | |
| 700 | 1 | |a Qiang, Yan |e verfasserin |4 aut | |
| 700 | 1 | |a Yang, Fenghao |e verfasserin |4 aut | |
| 700 | 1 | |a Hou, Xiao |e verfasserin |4 aut | |
| 700 | 1 | |a Zhao, Juanjuan |e verfasserin |4 aut | |
| 700 | 1 | |a Song, Kai |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Computers in biology and medicine |d 1970 |g 170(2024) vom: 20. Feb., Seite 107920 |w (DE-627)NLM000382272 |x 1879-0534 |7 nnns |
| 773 | 1 | 8 | |g volume:170 |g year:2024 |g day:20 |g month:02 |g pages:107920 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.compbiomed.2024.107920 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 170 |j 2024 |b 20 |c 02 |h 107920 | ||