SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks
Guardat en:
| Publicat a: | arXiv.org (Dec 21, 2024), p. n/a |
|---|---|
| Autor principal: | |
| Altres autors: | , |
| Publicat: |
Cornell University Library, arXiv.org
|
| Matèries: | |
| Accés en línia: | Citation/Abstract Full text outside of ProQuest |
| Etiquetes: |
Sense etiquetes, Sigues el primer a etiquetar aquest registre!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3148978562 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2331-8422 | ||
| 035 | |a 3148978562 | ||
| 045 | 0 | |b d20241221 | |
| 100 | 1 | |a Fröhling, Leon | |
| 245 | 1 | |a SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks | |
| 260 | |b Cornell University Library, arXiv.org |c Dec 21, 2024 | ||
| 513 | |a Working Paper | ||
| 520 | 3 | |a With the release of ever more capable large language models (LLMs), researchers in NLP and related disciplines have started to explore the usability of LLMs for a wide variety of different annotation tasks. Very recently, a lot of this attention has shifted to tasks that are subjective in nature. Given that the latest generations of LLMs have digested and encoded extensive knowledge about different human subpopulations and individuals, the hope is that these models can be trained, tuned or prompted to align with a wide range of different human perspectives. While researchers already evaluate the success of this alignment via surveys and tests, there is a lack of resources to evaluate the alignment on what oftentimes matters the most in NLP; the actual downstream tasks. To fill this gap we present SubData, a Python library that offers researchers working on topics related to subjectivity in annotation tasks a convenient way of collecting, combining and using a range of suitable datasets. | |
| 653 | |a Datasets | ||
| 653 | |a Python | ||
| 653 | |a Alignment | ||
| 653 | |a Annotations | ||
| 653 | |a Large language models | ||
| 700 | 1 | |a Bernardelle, Pietro | |
| 700 | 1 | |a Demartini, Gianluca | |
| 773 | 0 | |t arXiv.org |g (Dec 21, 2024), p. n/a | |
| 786 | 0 | |d ProQuest |t Engineering Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3148978562/abstract/embedded/6A8EOT78XXH2IG52?source=fedsrch |
| 856 | 4 | 0 | |3 Full text outside of ProQuest |u http://arxiv.org/abs/2412.16783 |