Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

Qi Fan
  • Fonction : Auteur
  • PersonId : 1090908
Li Ma
  • Fonction : Auteur
Yihui Feng
  • Fonction : Auteur
  • PersonId : 1201074
Yaliang Li
  • Fonction : Auteur
  • PersonId : 1201075
Kai Zeng
  • Fonction : Auteur
  • PersonId : 1201076
Jingren Zhou
  • Fonction : Auteur
  • PersonId : 1201077

Résumé

Big data processing at the production scale presents a highly complex environment for resource optimization (RO), a problem crucial for meeting performance goals and budgetary constraints of analytical users. The RO problem is challenging because it involves a set of decisions (the partition count, placement of parallel instances on machines, and resource allocation to each instance), requires multi-objective optimization (MOO), and is compounded by the scale and complexity of big data systems while having to meet stringent time constraints for scheduling. This paper presents a MaxCompute based integrated system to support multi-objective resource optimization via ne-grained instance-level modeling and optimization. We propose a new architecture that breaks RO into a series of simpler problems, new ne-grained predictive models, and novel optimization methods that exploit these models to make effective instance-level RO decisions well under a second. Evaluation using production workloads shows that our new RO system could reduce 37-72% latency and 43-78% cost at the same time, compared to the current optimizer and scheduler, while running in 0.02-0.23s.

Domaines

Informatique
Fichier principal
Vignette du fichier
VLDB2022-Udao.pdf (2.26 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03897397 , version 1 (13-12-2022)

Identifiants

  • HAL Id : hal-03897397 , version 1

Citer

Chenghao Lyu, Qi Fan, Fei Song, Arnab Sinha, Yanlei Diao, et al.. Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing. VLDB 2022 - 48th International Conference on Very Large Databases, Sep 2022, Sydney, Australia. ⟨hal-03897397⟩
41 Consultations
48 Téléchargements

Partager

Gmail Facebook X LinkedIn More