Application and Implementation of Batch File Transfer in Redis Storage - Inria - Institut national de recherche en sciences et technologies du numérique
Conference Papers Year : 2017

Application and Implementation of Batch File Transfer in Redis Storage

Abstract

In the face of massive information, batch processing of files is an important way of information transmission and storage, and the application is quite common. With the increasing demand for batch files processing reliability and speed, and the problem of low storage efficiency for current batch file processing, the paper proposes a storage method that combine distributed storage system HDFS file storage advantage and Redis cache technology to form a rapid batch merge files. The files that meet the conditions are merged into the Sequence File and stored in the HDFS. The multiple linear regression analysis method is used to determine the load factor, so that the load balancing is adjusted and the Redis cache hash data is used to ensure the efficiency. Through experiments on the corresponding file platform for file upload, query, delete and memory usage, we analysis batch processing method and non-batch method comparatively. It can be concluded that compared with the non-batch direct upload file to HDFS way, improved batch file processing method can process files more faster and ensure the stability and reliability of the file at the same time.
Fichier principal
Vignette du fichier
978-3-319-68121-4_24_Chapter.pdf (107.39 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Loading...

Dates and versions

hal-01820921 , version 1 (22-06-2018)

Licence

Identifiers

Cite

Hu Meng, Yongsheng Pan, Lang Sun. Application and Implementation of Batch File Transfer in Redis Storage. 2nd International Conference on Intelligence Science (ICIS), Oct 2017, Shanghai, China. pp.228-233, ⟨10.1007/978-3-319-68121-4_24⟩. ⟨hal-01820921⟩
154 View
184 Download

Altmetric

Share

More