Design choices made by LLM-based test generators prevent them from finding bugs
Kaydedildi:
| Yayımlandı: | arXiv.org (Dec 18, 2024), p. n/a |
|---|---|
| Yazar: | |
| Diğer Yazarlar: | |
| Baskı/Yayın Bilgisi: |
Cornell University Library, arXiv.org
|
| Konular: | |
| Online Erişim: | Citation/Abstract Full text outside of ProQuest |
| Etiketler: |
Etiket eklenmemiş, İlk siz ekleyin!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3147264606 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2331-8422 | ||
| 035 | |a 3147264606 | ||
| 045 | 0 | |b d20241218 | |
| 100 | 1 | |a Noble Saji Mathews | |
| 245 | 1 | |a Design choices made by LLM-based test generators prevent them from finding bugs | |
| 260 | |b Cornell University Library, arXiv.org |c Dec 18, 2024 | ||
| 513 | |a Working Paper | ||
| 520 | 3 | |a There is an increasing amount of research and commercial tools for automated test case generation using Large Language Models (LLMs). This paper critically examines whether recent LLM-based test generation tools, such as Codium CoverAgent and CoverUp, can effectively find bugs or unintentionally validate faulty code. Considering bugs are only exposed by failing test cases, we explore the question: can these tools truly achieve the intended objectives of software testing when their test oracles are designed to pass? Using real human-written buggy code as input, we evaluate these tools, showing how LLM-generated tests can fail to detect bugs and, more alarmingly, how their design can worsen the situation by validating bugs in the generated test suite and rejecting bug-revealing tests. These findings raise important questions about the validity of the design behind LLM-based test generation tools and their impact on software quality and test suite reliability. | |
| 653 | |a Software reliability | ||
| 653 | |a Debugging | ||
| 653 | |a Large language models | ||
| 653 | |a Software testing | ||
| 700 | 1 | |a Nagappan, Meiyappan | |
| 773 | 0 | |t arXiv.org |g (Dec 18, 2024), p. n/a | |
| 786 | 0 | |d ProQuest |t Engineering Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3147264606/abstract/embedded/6A8EOT78XXH2IG52?source=fedsrch |
| 856 | 4 | 0 | |3 Full text outside of ProQuest |u http://arxiv.org/abs/2412.14137 |