HITgram: A Platform for Experimenting with n-gram Language Models
Sábháilte in:
| Foilsithe in: | arXiv.org (Dec 14, 2024), p. n/a |
|---|---|
| Príomhchruthaitheoir: | |
| Rannpháirtithe: | , , , , |
| Foilsithe / Cruthaithe: |
Cornell University Library, arXiv.org
|
| Ábhair: | |
| Rochtain ar líne: | Citation/Abstract Full text outside of ProQuest |
| Clibeanna: |
Níl clibeanna ann, Bí ar an gcéad duine le clib a chur leis an taifead seo!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3145904375 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2331-8422 | ||
| 035 | |a 3145904375 | ||
| 045 | 0 | |b d20241214 | |
| 100 | 1 | |a Dasgupta, Shibaranjani | |
| 245 | 1 | |a HITgram: A Platform for Experimenting with n-gram Language Models | |
| 260 | |b Cornell University Library, arXiv.org |c Dec 14, 2024 | ||
| 513 | |a Working Paper | ||
| 520 | 3 | |a Large language models (LLMs) are powerful but resource intensive, limiting accessibility. HITgram addresses this gap by offering a lightweight platform for n-gram model experimentation, ideal for resource-constrained environments. It supports unigrams to 4-grams and incorporates features like context sensitive weighting, Laplace smoothing, and dynamic corpus management to e-hance prediction accuracy, even for unseen word sequences. Experiments demonstrate HITgram's efficiency, achieving 50,000 tokens/second and generating 2-grams from a 320MB corpus in 62 seconds. HITgram scales efficiently, constructing 4-grams from a 1GB file in under 298 seconds on an 8 GB RAM system. Planned enhancements include multilingual support, advanced smoothing, parallel processing, and model saving, further broadening its utility. | |
| 653 | |a Parallel processing | ||
| 653 | |a Smoothing | ||
| 653 | |a Large language models | ||
| 700 | 1 | |a Maity, Chandan | |
| 700 | 1 | |a Mukherjee, Somdip | |
| 700 | 1 | |a Singh, Rohan | |
| 700 | 1 | |a Dutta, Diptendu | |
| 700 | 1 | |a Jana, Debasish | |
| 773 | 0 | |t arXiv.org |g (Dec 14, 2024), p. n/a | |
| 786 | 0 | |d ProQuest |t Engineering Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3145904375/abstract/embedded/ZKJTFFSVAI7CB62C?source=fedsrch |
| 856 | 4 | 0 | |3 Full text outside of ProQuest |u http://arxiv.org/abs/2412.10717 |