Puzzle: Distillation-Based NAS for Inference-Optimized LLMs

Guardado en:

書目詳細資料
發表在:	arXiv.org (Dec 8, 2024), p. n/a
主要作者:	Akhiad Bercovich
其他作者:	Ronen, Tomer, Abramovich, Talor, Ailon, Nir, Nave Assaf, Dabbah, Mohammad, Galil, Ido, Geifman, Amnon, Geifman, Yonatan, Golan, Izhak, Haber, Netanel, Karpas, Ehud, Koren, Roi, Levy, Itay, Molchanov, Pavlo, Mor, Shahar, Zach, Moshe, Nabwani, Najeeb, Puny, Omri, Rubin, Ran, Schen, Itamar, Shahaf, Ido, Tropp, Oren, Omer Ullman Argov, Zilberstein, Ran, El-Yaniv, Ran
出版:	Cornell University Library, arXiv.org
主題:	Computing costs Integer programming Mixed integer Computer architecture Large language models Graphics processing units Hardware Constraints Parameters Inference
在線閱讀:	Citation/Abstract Full text outside of ProQuest
標簽:	添加標簽沒有標簽, 成為第一個標記此記錄!

實物特徵
Resumen:	Large language models (LLMs) have demonstrated remarkable capabilities, but their adoption is limited by high computational costs during inference. While increasing parameter counts enhances accuracy, it also widens the gap between state-of-the-art capabilities and practical deployability. We present Puzzle, a framework to accelerate LLM inference on specific hardware while preserving their capabilities. Through an innovative application of neural architecture search (NAS) at an unprecedented scale, Puzzle systematically optimizes models with tens of billions of parameters under hardware constraints. Our approach utilizes blockwise local knowledge distillation (BLD) for parallel architecture exploration and employs mixed-integer programming for precise constraint optimization. We demonstrate the real-world impact of our framework through Llama-3.1-Nemotron-51B-Instruct (Nemotron-51B), a publicly available model derived from Llama-3.1-70B-Instruct. Nemotron-51B achieves a 2.17x inference throughput speedup, fitting on a single NVIDIA H100 GPU while preserving 98.4% of the original model's capabilities. Nemotron-51B currently stands as the most accurate language model capable of inference on a single GPU with large batch sizes. Remarkably, this transformation required just 45B training tokens, compared to over 15T tokens used for the 70B model it was derived from. This establishes a new paradigm where powerful models can be optimized for efficient deployment with only negligible compromise of their capabilities, demonstrating that inference performance, not parameter count alone, should guide model selection. With the release of Nemotron-51B and the presentation of the Puzzle framework, we provide practitioners immediate access to state-of-the-art language modeling capabilities at significantly reduced computational costs.
ISSN:	2331-8422
Fuente:	Engineering Database