Imagen de Portada

Performance models of data parallel DAG workflows for large scale data analytics

Guardado en:
Publicado en:Distributed and Parallel Databases vol. 41, no. 3 (Sep 2023), p. 299
Autor principal: Shi, Juwei
Otros Autores: Lu, Jiaheng
Publicado:
Springer Nature B.V.
Materias:
Skewness
Workloads
Estimates
Distributed processing
Estimation
Resource allocation
Acceso en línea:Citation/Abstract
Full Text
Full Text - PDF
Etiquetas: Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
Resumen:Directed Acyclic Graph (DAG) workflows are widely used for large-scale data analytics in cluster-based distributed computing systems. The performance model for a DAG on data-parallel frameworks (e.g., MapReduce) is a research challenge because the allocation of preemptable system resources among parallel jobs may dynamically vary during execution. This resource allocation variation during execution makes it difficult to accurately estimate the execution time. In this paper, we tackle this challenge by proposing a new cost model, called Bottleneck Oriented Estimation (BOE), to estimate the allocation of preemptable resources by identifying the bottleneck to accurately predict task execution time. For a DAG workflow, we propose a state-based approach to iteratively use the resource allocation property among stages to estimate the overall execution plan. Furthermore, to handle the skewness of various jobs, we refine the model with the order statistics theory to improve estimation accuracy. Extensive experiments were performed to validate these cost models with HiBench and TPC-H workloads. The BOE model outperforms the state-of-the-art models by a factor of five for task execution time estimation. For the refined skew-aware model, the average prediction error is under 3%<inline-graphic specific-use="web" mime-subtype="GIF" xlink:href="10619_2023_7425_Article_IEq1.gif" /> when estimating the execution time of 51 hybrid analytics (HiBench) and query (TPC-H) DAG workflows.
ISSN:0926-8782
1573-7578
DOI:10.1007/s10619-023-07425-1
Fuente:Advanced Technologies & Aerospace Database