Optimizing Ring AllReduce for Sparse Data
Guardado en:
| Udgivet i: | ProQuest Dissertations and Theses (2026) |
|---|---|
| Hovedforfatter: | |
| Udgivet: |
ProQuest Dissertations & Theses
|
| Fag: | |
| Online adgang: | Citation/Abstract Full Text - PDF |
| Tags: |
Ingen Tags, Vær først til at tagge denne postø!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3286136717 | ||
| 003 | UK-CbPIL | ||
| 020 | |a 9798270248345 | ||
| 035 | |a 3286136717 | ||
| 045 | 2 | |b d20260101 |b d20261231 | |
| 084 | |a 66569 |2 nlm | ||
| 100 | 1 | |a Arunachalam, Anshul | |
| 245 | 1 | |a Optimizing Ring AllReduce for Sparse Data | |
| 260 | |b ProQuest Dissertations & Theses |c 2026 | ||
| 513 | |a Dissertation/Thesis | ||
| 520 | 3 | |a The distributed training of machine learning models via gradient descent is generally conducted by iteratively computing the local gradients of a loss function and aggregating them across all processors. Communicating these gradients during aggregation is often a major cost but sparsification techniques can greatly improve efficiency. One such technique, Top-k gradient compression, ensures that only the k largest components of each local gradient are sent. However, effectively scaling this method can be challenging. The standard ring AllReduce algorithm, which is frequently used to aggregate dense gradients, lacks a counterpart that is optimized for sparse data. Notably, ring algorithms are contention-free, which generally make them easier to scale than other collective communication algorithms. Thus, in practice, the ring AllGather algorithm, which can be trivially adapted for sparse data, may be used instead, even though its bandwidth costs are proportional to the number of utilized processors (unlike ring AllReduce). To provide a more scalable contention-free alternative, we present a variant of ring AllReduce that has been better optimized for sparse data. We compare it to the standard dense ring AllReduce and ring AllGather algorithms, and we evaluate it empirically using gradients sampled from fine-tuning Llama 2 7b. | |
| 653 | |a Computer science | ||
| 653 | |a Computer engineering | ||
| 653 | |a Information science | ||
| 773 | 0 | |t ProQuest Dissertations and Theses |g (2026) | |
| 786 | 0 | |d ProQuest |t Publicly Available Content Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3286136717/abstract/embedded/7BTGNMKEMPT1V9Z2?source=fedsrch |
| 856 | 4 | 0 | |3 Full Text - PDF |u https://www.proquest.com/docview/3286136717/fulltextPDF/embedded/7BTGNMKEMPT1V9Z2?source=fedsrch |