Application and Implementation of Batch File Transfer in Redis Storage - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2017

Application and Implementation of Batch File Transfer in Redis Storage

Résumé

In the face of massive information, batch processing of files is an important way of information transmission and storage, and the application is quite common. With the increasing demand for batch files processing reliability and speed, and the problem of low storage efficiency for current batch file processing, the paper proposes a storage method that combine distributed storage system HDFS file storage advantage and Redis cache technology to form a rapid batch merge files. The files that meet the conditions are merged into the Sequence File and stored in the HDFS. The multiple linear regression analysis method is used to determine the load factor, so that the load balancing is adjusted and the Redis cache hash data is used to ensure the efficiency. Through experiments on the corresponding file platform for file upload, query, delete and memory usage, we analysis batch processing method and non-batch method comparatively. It can be concluded that compared with the non-batch direct upload file to HDFS way, improved batch file processing method can process files more faster and ensure the stability and reliability of the file at the same time.
Fichier principal
Vignette du fichier
978-3-319-68121-4_24_Chapter.pdf (107.39 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01820921 , version 1 (22-06-2018)

Licence

Identifiants

Citer

Hu Meng, Yongsheng Pan, Lang Sun. Application and Implementation of Batch File Transfer in Redis Storage. 2nd International Conference on Intelligence Science (ICIS), Oct 2017, Shanghai, China. pp.228-233, ⟨10.1007/978-3-319-68121-4_24⟩. ⟨hal-01820921⟩
158 Consultations
189 Téléchargements

Altmetric

Partager

More