ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries
Αποθηκεύτηκε σε:
| Εκδόθηκε σε: | arXiv.org (Dec 18, 2024), p. n/a |
|---|---|
| Κύριος συγγραφέας: | |
| Άλλοι συγγραφείς: | , , , , , , |
| Έκδοση: |
Cornell University Library, arXiv.org
|
| Θέματα: | |
| Διαθέσιμο Online: | Citation/Abstract Full text outside of ProQuest |
| Ετικέτες: |
Δεν υπάρχουν, Καταχωρήστε ετικέτα πρώτοι!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3119817256 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2331-8422 | ||
| 035 | |a 3119817256 | ||
| 045 | 0 | |b d20241218 | |
| 100 | 1 | |a Maharaj, Kishan | |
| 245 | 1 | |a ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries | |
| 260 | |b Cornell University Library, arXiv.org |c Dec 18, 2024 | ||
| 513 | |a Working Paper | ||
| 520 | 3 | |a Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarization. However, LLMs are prone to hallucination-outputs that stray from intended meanings. Detecting hallucinations in code summarization is especially difficult due to the complex interplay between programming and natural languages. We introduce a first-of-its-kind dataset with \(\sim\)10K samples, curated specifically for hallucination detection in code summarization. We further propose a novel Entity Tracing Framework (ETF) that a) utilizes static program analysis to identify code entities from the program and b) uses LLMs to map and verify these entities and their intents within generated code summaries. Our experimental analysis demonstrates the effectiveness of the framework, leading to a 0.73 F1 score. This approach provides an interpretable method for detecting hallucinations by grounding entities, allowing us to evaluate summary accuracy. | |
| 653 | |a Program verification (computers) | ||
| 653 | |a Large language models | ||
| 653 | |a Summaries | ||
| 653 | |a Tracing | ||
| 653 | |a Task complexity | ||
| 700 | 1 | |a Munigala, Vitobha | |
| 700 | 1 | |a Tamilselvam, Srikanth G | |
| 700 | 1 | |a Kumar, Prince | |
| 700 | 1 | |a Sen, Sayandeep | |
| 700 | 1 | |a Kodeswaran, Palani | |
| 700 | 1 | |a Mishra, Abhijit | |
| 700 | 1 | |a Bhattacharyya, Pushpak | |
| 773 | 0 | |t arXiv.org |g (Dec 18, 2024), p. n/a | |
| 786 | 0 | |d ProQuest |t Engineering Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3119817256/abstract/embedded/6A8EOT78XXH2IG52?source=fedsrch |
| 856 | 4 | 0 | |3 Full text outside of ProQuest |u http://arxiv.org/abs/2410.14748 |