Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2019

Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning

Résumé

As storage hierarchies are getting deeper on modern high-performance computing systems, intelligent data placement strategies that can choose the optimal storage tier dynamically is the key to realize the potential of hierarchical storage architecture. However, providing a general solution that can be applied in different storage architectures and diverse applications is challenging. In this paper, we propose adaptive storage learner (ASL), which explores the idea of using machine learning techniques to mine the relationship between data placement strategies and I/O performance under varied workflow characteristics and system status, and uses the learned model to choose the optimal storage tier intelligently. We implement a prototype and integrate it into an existing data management system. Empirical comparison based on real scientific workflows tests shows that ASL is capable of combining workflow characteristics and real-time system status to make optimal data placement decisions.
Fichier principal
Vignette du fichier
486810_1_En_23_Chapter.pdf (340.5 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03770550 , version 1 (06-09-2022)

Licence

Paternité

Identifiants

Citer

Peng Cheng, Yutong Lu, Yunfei Du, Zhiguang Chen, Yang Liu. Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning. 16th IFIP International Conference on Network and Parallel Computing (NPC), Aug 2019, Hohhot, China. pp.289-302, ⟨10.1007/978-3-030-30709-7_23⟩. ⟨hal-03770550⟩
25 Consultations
104 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More