Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization

Na minha lista:

Detalhes bibliográficos
Publicado no:	arXiv.org (Dec 24, 2024), p. n/a
Autor principal:	Shen, Qianli
Outros Autores:	Wang, Yezhen, Yang, Zhouhao, Li, Xiang, Wang, Haonan, Zhang, Yang, Scarlett, Jonathan, Zhu, Zhanxing, Kawaguchi, Kenji
Publicado em:	Cornell University Library, arXiv.org
Assuntos:	Approximation Algorithms Deep learning System effectiveness Machine learning Optimization Distributed processing
Acesso em linha:	Citation/Abstract Full text outside of ProQuest
Tags:	Adicionar Tag Sem tags, seja o primeiro a adicionar uma tag!

Descrição
Resumo:	Bi-level optimization (BO) has become a fundamental mathematical framework for addressing hierarchical machine learning problems. As deep learning models continue to grow in size, the demand for scalable bi-level optimization solutions has become increasingly critical. Traditional gradient-based bi-level optimization algorithms, due to their inherent characteristics, are ill-suited to meet the demands of large-scale applications. In this paper, we introduce \(\textbf{F}\)orward \(\textbf{G}\)radient \(\textbf{U}\)nrolling with \(\textbf{F}\)orward \(\textbf{F}\)radient, abbreviated as \((\textbf{FG})^2\textbf{U}\), which achieves an unbiased stochastic approximation of the meta gradient for bi-level optimization. \((\text{FG})^2\text{U}\) circumvents the memory and approximation issues associated with classical bi-level optimization approaches, and delivers significantly more accurate gradient estimates than existing large-scale bi-level optimization approaches. Additionally, \((\text{FG})^2\text{U}\) is inherently designed to support parallel computing, enabling it to effectively leverage large-scale distributed computing systems to achieve significant computational efficiency. In practice, \((\text{FG})^2\text{U}\) and other methods can be strategically placed at different stages of the training process to achieve a more cost-effective two-phase paradigm. Further, \((\text{FG})^2\text{U}\) is easy to implement within popular deep learning frameworks, and can be conveniently adapted to address more challenging zeroth-order bi-level optimization scenarios. We provide a thorough convergence analysis and a comprehensive practical discussion for \((\text{FG})^2\text{U}\), complemented by extensive empirical evaluations, showcasing its superior performance in diverse large-scale bi-level optimization tasks. Code is available at https://github.com/ShenQianli/FG2U.
ISSN:	2331-8422
Fonte:	Engineering Database