Application and Implementation of Batch File Transfer in Redis Storage
Abstract
In the face of massive information, batch processing of files is an important way of information transmission and storage, and the application is quite common. With the increasing demand for batch files processing reliability and speed, and the problem of low storage efficiency for current batch file processing, the paper proposes a storage method that combine distributed storage system HDFS file storage advantage and Redis cache technology to form a rapid batch merge files. The files that meet the conditions are merged into the Sequence File and stored in the HDFS. The multiple linear regression analysis method is used to determine the load factor, so that the load balancing is adjusted and the Redis cache hash data is used to ensure the efficiency. Through experiments on the corresponding file platform for file upload, query, delete and memory usage, we analysis batch processing method and non-batch method comparatively. It can be concluded that compared with the non-batch direct upload file to HDFS way, improved batch file processing method can process files more faster and ensure the stability and reliability of the file at the same time.
Domains
Computer Science [cs]Origin | Files produced by the author(s) |
---|
Loading...