Optimization of RDMA-Based HDFS Data Distribution Mechanism - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Optimization of RDMA-Based HDFS Data Distribution Mechanism

Résumé

Hadoop Distributed File System (short for HDFS) is a high availability file system designed to run on commodity hardware. It uses multiple replicas to ensure high reliability, and many data are transmitted between storage nodes. The performance of data transmission has a great impact on the latency of writing operations. Remote Direct Memory Access (short for RDMA) is a protocol with low latency and high through which is running on the Infiniband network. When HDFS runs on the Infiniband network, the default protocol IPoIB can not take advantage of the high-speed network. The latency of the writing process is similar to a TCP/IP network. In this paper, we present a new RDMA-based HDFS writing mechanism. It enables DataNodes to read data parallelly from the Client through RDMA. And by using RDMA primitive, the transmission latency is slower than the original TCP/IP protocol. The experiments show that our approach reduces the latency of the writing process by 10.11%–40.81% compared with the original HDFS.
Fichier principal
Vignette du fichier
511910_1_En_22_Chapter.pdf (2.1 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03768747 , version 1 (04-09-2022)

Licence

Paternité

Identifiants

Citer

Xiao Zhang, Binbin Liu, Junhao Zhao, Cong Dong. Optimization of RDMA-Based HDFS Data Distribution Mechanism. 17th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2020, Zhengzhou, China. pp.251-262, ⟨10.1007/978-3-030-79478-1_22⟩. ⟨hal-03768747⟩
22 Consultations
13 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More