DLI-IT : a deep learning approach to drug label identification through image and text embedding

BACKGROUND: Drug label, or packaging insert play a significant role in all the operations from production through drug distribution channels to the end consumer. Image of the label also called Display Panel or label could be used to identify illegal, illicit, unapproved and potentially dangerous drugs. Due to the time-consuming process and high labor cost of investigation, an artificial intelligence-based deep learning model is necessary for fast and accurate identification of the drugs.

METHODS: In addition to image-based identification technology, we take advantages of rich text information on the pharmaceutical package insert of drug label images. In this study, we developed the Drug Label Identification through Image and Text embedding model (DLI-IT) to model text-based patterns of historical data for detection of suspicious drugs. In DLI-IT, we first trained a Connectionist Text Proposal Network (CTPN) to crop the raw image into sub-images based on the text. The texts from the cropped sub-images are recognized independently through the Tesseract OCR Engine and combined as one document for each raw image. Finally, we applied universal sentence embedding to transform these documents into vectors and find the most similar reference images to the test image through the cosine similarity.

RESULTS: We trained the DLI-IT model on 1749 opioid and 2365 non-opioid drug label images. The model was then tested on 300 external opioid drug label images, the result demonstrated our model achieves up-to 88% of the precision in drug label identification, which outperforms previous image-based or text-based identification method by up-to 35% improvement.

CONCLUSION: To conclude, by combining Image and Text embedding analysis under deep learning framework, our DLI-IT approach achieved a competitive performance in advancing drug label identification.

Medienart:

E-Artikel

Erscheinungsjahr:

2020

Erschienen:

2020

Enthalten in:

Zur Gesamtaufnahme - volume:20

Enthalten in:

BMC medical informatics and decision making - 20(2020), 1 vom: 15. Apr., Seite 68

Sprache:

Englisch

Beteiligte Personen:

Liu, Xiangwen [VerfasserIn]
Meehan, Joe [VerfasserIn]
Tong, Weida [VerfasserIn]
Wu, Leihong [VerfasserIn]
Xu, Xiaowei [VerfasserIn]
Xu, Joshua [VerfasserIn]

Links:

Volltext

Themen:

Daily-med
Deep learning
Drug labeling
Image recognition
Journal Article
Neural network
Opioid drug
Pharmaceutical Preparations
Pharmaceutical packaging
Scene text detection
Semantic similarity
Similarity identification

Anmerkungen:

Date Completed 14.12.2020

Date Revised 14.12.2020

published: Electronic

Citation Status MEDLINE

doi:

10.1186/s12911-020-1078-3

funding:

Förderinstitution / Projekttitel:

PPN (Katalog-ID):

NLM30878376X