Зображення обкладинки

SimCost: cost-effective resource provision prediction and recommendation for spark workloads

Збережено в:
Опубліковано в::Distributed and Parallel Databases vol. 42, no. 1 (Mar 2024), p. 73
Автор: Chen, Yuxing
Інші автори: Hoque, Mohammad A., Xu, Pengfei, Lu, Jiaheng, Tarkoma, Sasu
Опубліковано:
Springer Nature B.V.
Предмети:
Workload
Big Data
Machine learning
Simulation
Algorithms
Resource utilization
Salience
Cloud computing
Workloads
Monte Carlo simulation
Cost effectiveness
Онлайн доступ:Citation/Abstract
Full Text
Full Text - PDF
Теги: Додати тег
Немає тегів, Будьте першим, хто поставить тег для цього запису!
Короткий огляд:Spark is one of the most popular big data analytical platforms. To save time, achieve high resource utilization, and remain cost-effective for Spark jobs, it is challenging but imperative for data scientists to configure suitable resource portions.In this paper, we investigate the proper parameter values that meet workloads’ performance requirements with minimized resource cost and resource utilization time. We propose SimCost, a simulation-based cost model, to predict the performance of jobs accurately. We achieve low-cost training by taking advantage of simulation framework, i.e., Monte Carlo simulation, which uses a small amount of data and resources to make a reliable prediction for larger datasets and clusters. Our method’s salient feature is that it allows us to invest low training costs while obtaining an accurate prediction. Through empirical experiments with 12 benchmark workloads, we show that the cost model yields less than 5% error on average prediction accuracy, and the recommendation achieves up to 6x resource cost saving.
ISSN:0926-8782
1573-7578
DOI:10.1007/s10619-023-07436-y
Джерело:Advanced Technologies & Aerospace Database