Obálka

SimCost: cost-effective resource provision prediction and recommendation for spark workloads

Uloženo v:
Vydáno v:Distributed and Parallel Databases vol. 42, no. 1 (Mar 2024), p. 73
Hlavní autor: Chen, Yuxing
Další autoři: Hoque, Mohammad A., Xu, Pengfei, Lu, Jiaheng, Tarkoma, Sasu
Vydáno:
Springer Nature B.V.
Témata:
Workload
Big Data
Machine learning
Simulation
Algorithms
Resource utilization
Salience
Cloud computing
Workloads
Monte Carlo simulation
Cost effectiveness
On-line přístup:Citation/Abstract
Full Text
Full Text - PDF
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Abstrakt:Spark is one of the most popular big data analytical platforms. To save time, achieve high resource utilization, and remain cost-effective for Spark jobs, it is challenging but imperative for data scientists to configure suitable resource portions.In this paper, we investigate the proper parameter values that meet workloads’ performance requirements with minimized resource cost and resource utilization time. We propose SimCost, a simulation-based cost model, to predict the performance of jobs accurately. We achieve low-cost training by taking advantage of simulation framework, i.e., Monte Carlo simulation, which uses a small amount of data and resources to make a reliable prediction for larger datasets and clusters. Our method’s salient feature is that it allows us to invest low training costs while obtaining an accurate prediction. Through empirical experiments with 12 benchmark workloads, we show that the cost model yields less than 5% error on average prediction accuracy, and the recommendation achieves up to 6x resource cost saving.
ISSN:0926-8782
1573-7578
DOI:10.1007/s10619-023-07436-y
Zdroj:Advanced Technologies & Aerospace Database