Beeld op de omslag

Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems

Bewaard in:
Gepubliceerd in:Machine Learning and Knowledge Extraction vol. 7, no. 1 (2025), p. 15
Hoofdauteur: Lakatos, Róbert
Andere auteurs: Pollner, Péter, Hajdu, András, Joó, Tamás
Gepubliceerd in:
MDPI AG
Onderwerpen:
United States > US
Language
Work stations
Accuracy
Datasets
Performance evaluation
Large language models
Knowledge
Benchmarks
Retrieval
Machine translation
Performance degradation
Coronaviruses
Chatbots
Meteors
Online toegang:Citation/Abstract
Full Text + Graphics
Full Text - PDF
Tags: Voeg label toe
Geen labels, Wees de eerste die dit record labelt!
Samenvatting:Generative large language models (LLMs) have revolutionized the development of knowledge-based systems, enabling new possibilities in applications like ChatGPT, Bing, and Gemini. Two key strategies for domain adaptation in these systems are Domain-Specific Fine-Tuning (DFT) and Retrieval-Augmented Generation (RAG). In this study, we evaluate the performance of RAG and DFT on several LLM architectures, including GPT-J-6B, OPT-6.7B, LLaMA, and LLaMA-2. We use the ROUGE, BLEU, and METEOR scores to evaluate the performance of the models. We also measure the performance of the models with our own designed cosine similarity-based Coverage Score (CS). Our results, based on experiments across multiple datasets, show that RAG-based systems consistently outperform those fine-tuned with DFT. Specifically, RAG models outperform DFT by an average of 17% in ROUGE, 13% in BLEU, and 36% in CS. At the same time, DFT achieves only a modest advantage in METEOR, suggesting slightly better creative capabilities. We also highlight the challenges of integrating RAG with DFT, as such integration can lead to performance degradation. Furthermore, we propose a simplified RAG-based architecture that maximizes efficiency and reduces hallucination, underscoring the advantages of RAG in building reliable, domain-adapted knowledge systems.
ISSN:2504-4990
DOI:10.3390/make7010015
Bron:Advanced Technologies & Aerospace Database