Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study
Guardado en:
| Publicado en: | Journal of Medical Internet Research vol. 27 (2025), p. e66220 |
|---|---|
| Autor principal: | |
| Otros Autores: | , , , , , , , , , , , , , , , , , , , , |
| Publicado: |
Gunther Eysenbach MD MPH, Associate Professor
|
| Materias: | |
| Acceso en línea: | Citation/Abstract Full Text + Graphics Full Text - PDF |
| Etiquetas: |
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3222367931 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 1438-8871 | ||
| 024 | 7 | |a 10.2196/66220 |2 doi | |
| 035 | |a 3222367931 | ||
| 045 | 2 | |b d20250101 |b d20251231 | |
| 100 | 1 | |a Das, Sudeshna | |
| 245 | 1 | |a Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study | |
| 260 | |b Gunther Eysenbach MD MPH, Associate Professor |c 2025 | ||
| 513 | |a Journal Article | ||
| 520 | 3 | |a Background:The increasing use of social media to share lived and living experiences of substance use presents a unique opportunity to obtain information on side effects, use patterns, and opinions on novel psychoactive substances. However, due to the large volume of data, obtaining useful insights through natural language processing technologies such as large language models is challenging.Objective:This paper aims to develop a retrieval-augmented generation (RAG) architecture for medical question answering pertaining to clinicians’ queries on emerging issues associated with health-related topics, using user-generated medical information on social media.Methods:We proposed a two-layer RAG framework for query-focused answer generation and evaluated a proof of concept for the framework in the context of query-focused summary generation from social media forums, focusing on emerging drug-related information. Our modular framework generates individual summaries followed by an aggregated summary to answer medical queries from large amounts of user-generated social media data in an efficient manner. We compared the performance of a quantized large language model (Nous-Hermes-2-7B-DPO), deployable in low-resource settings, with GPT-4. For this proof-of-concept study, we used user-generated data from Reddit to answer clinicians’ questions on the use of xylazine and ketamine.Results:Our framework achieves comparable median scores in terms of relevance, length, hallucination, coverage, and coherence when evaluated using GPT-4 and Nous-Hermes-2-7B-DPO, evaluated for 20 queries with 76 samples. There was no statistically significant difference between GPT-4 and Nous-Hermes-2-7B-DPO for coverage (Mann-Whitney U=733.0; n1=37; n2=39; P=.89 two-tailed), coherence (U=670.0; n1=37; n2=39; P=.49 two-tailed), relevance (U=662.0; n1=37; n2=39; P=.15 two-tailed), length (U=672.0; n1=37; n2=39; P=.55 two-tailed), and hallucination (U=859.0; n1=37; n2=39; P=.01 two-tailed). A statistically significant difference was noted for the Coleman-Liau Index (U=307.5; n1=20; n2=16; P<.001 two-tailed).Conclusions:Our RAG framework can effectively answer medical questions about targeted topics and can be deployed in resource-constrained settings. | |
| 651 | 4 | |a United States--US | |
| 653 | |a Readability | ||
| 653 | |a Drug overdose | ||
| 653 | |a Drug withdrawal | ||
| 653 | |a Multimedia | ||
| 653 | |a Hypotheses | ||
| 653 | |a Substance abuse | ||
| 653 | |a Ketamine | ||
| 653 | |a Social media | ||
| 653 | |a Retrieval | ||
| 653 | |a Side effects | ||
| 653 | |a Frame analysis | ||
| 653 | |a Likert scale | ||
| 653 | |a Data collection | ||
| 653 | |a Queries | ||
| 653 | |a Large language models | ||
| 653 | |a Augmentation | ||
| 653 | |a Retrieval performance measures | ||
| 653 | |a Information retrieval | ||
| 653 | |a Coherence | ||
| 653 | |a Clinical information | ||
| 653 | |a Answers | ||
| 653 | |a Data processing | ||
| 653 | |a Models | ||
| 653 | |a Mass media | ||
| 653 | |a Attitudes | ||
| 653 | |a Question answer sequences | ||
| 653 | |a Drug effects | ||
| 653 | |a Natural language processing | ||
| 653 | |a Information | ||
| 653 | |a Language | ||
| 653 | |a Language modeling | ||
| 700 | 1 | |a Yao Ge | |
| 700 | 1 | |a Guo, Yuting | |
| 700 | 1 | |a Rajwal, Swati | |
| 700 | 1 | |a Hairston, JaMor | |
| 700 | 1 | |a Powell, Jeanne | |
| 700 | 1 | |a Walker, Drew | |
| 700 | 1 | |a Peddireddy, Snigdha | |
| 700 | 1 | |a Lakamana, Sahithi | |
| 700 | 1 | |a Bozkurt, Selen | |
| 700 | 1 | |a Reyna, Matthew | |
| 700 | 1 | |a Sameni, Reza | |
| 700 | 1 | |a Xiao, Yunyu | |
| 700 | 1 | |a Kim, Sangmi | |
| 700 | 1 | |a Chandler, Rasheeta | |
| 700 | 1 | |a Hernandez, Natalie | |
| 700 | 1 | |a Mowery, Danielle | |
| 700 | 1 | |a Wightman, Rachel | |
| 700 | 1 | |a Love, Jennifer | |
| 700 | 1 | |a Spadaro, Anthony | |
| 700 | 1 | |a Perrone, Jeanmarie | |
| 700 | 1 | |a Sarker, Abeed | |
| 773 | 0 | |t Journal of Medical Internet Research |g vol. 27 (2025), p. e66220 | |
| 786 | 0 | |d ProQuest |t Library Science Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3222367931/abstract/embedded/L8HZQI7Z43R0LA5T?source=fedsrch |
| 856 | 4 | 0 | |3 Full Text + Graphics |u https://www.proquest.com/docview/3222367931/fulltextwithgraphics/embedded/L8HZQI7Z43R0LA5T?source=fedsrch |
| 856 | 4 | 0 | |3 Full Text - PDF |u https://www.proquest.com/docview/3222367931/fulltextPDF/embedded/L8HZQI7Z43R0LA5T?source=fedsrch |