Ata uhi

AI Computing Systems for Large Language Models Training

I tiakina i:
I whakaputaina i:Journal of Computer Science and Technology vol. 40, no. 1 (Jan 2025), p. 6
I whakaputaina:
Springer Nature B.V.
Ngā marau:
Large language models
Artificial intelligence
Graphics processing units
Hardware
Tensors
Algorithms
Encoders-Decoders
Complexity
Machine learning
Infrastructure
Distributed memory
Software
Accelerators
Distributed processing
Memory management
Urunga tuihono:Citation/Abstract
Full Text - PDF
Ngā Tūtohu: Tāpirihia he Tūtohu
Kāore He Tūtohu, Me noho koe te mea tuatahi ki te tūtohu i tēnei pūkete!
Whakarāpopotonga:In this paper, we present a comprehensive overview of artificial intelligence (AI) computing systems for large language models (LLMs) training. The rapid advancement of LLMs in recent years, coupled with the widespread adoption of algorithms and applications such as BERT, ChatGPT, and DeepSeek, has sparked significant interest in this field. We classify LLMs into encoder-only, encoder-decoder, and decoder-only models, and briefly analyze their training and inference processes to emphasize their substantial need for computational resources. These operations depend heavily on AI-specific accelerators like GPUs (graphics processing units), TPUs (tensor processing units), and MLUs (machine learning units). However, as the gap widens between the increasing complexity of LLMs and the current capabilities of accelerators, it becomes essential to adopt heterogeneous computing systems optimized for distributed environments to manage the growing computational and memory requirements of LLMs. We delve into the execution and scheduling of LLM algorithms, underlining the critical role of distributed computing strategies, memory management enhancements, and boosting computational efficiency. This paper clarifies the complex relationship between algorithm design, hardware infrastructure, and software optimization, and provides an in-depth understanding of both the software and hardware infrastructure supporting LLMs training, offering insights into the challenges and potential avenues for future development and deployment.
ISSN:1000-9000
1860-4749
DOI:10.1007/s11390-024-4178-1
Puna:ABI/INFORM Global