Computer Science ›› 2014, Vol. 41 ›› Issue (7): 30-35.doi: 10.11896/j.issn.1002-137X.2014.07.005

Previous Articles     Next Articles

Design and Implementation of File Prefetching Module Oriented to Distributed File System

SHI Ming,LIU Yi and TANG Ge-shi   

  • Online:2018-11-14 Published:2018-11-14

Abstract: How to provide a stable and efficient file I/O performance for the upper application and computing,is the performance research hotspot oriented to distributed file system.This paper analyzed the mechanism in the design of the distributed file system on the common features,presented a general-purpose file prefetching heuristic module,and selected HDFS platform system to implement.The heuristic file prefetching module services the upper application and accomplishes the implementation in the internal of distributed file system,using the method of establishing prefetching thread pool within the file system,and the data not block as prefetching unit.This idea has certain universality,and is suitable for a variety of distributed file systems.Experimental results show that the heuristic file prefetching method can enhance the distributed file system I/O performance effectively.

Key words: Distributed file system,File prefetching,Heuristic,HDFS

[1] Yue Yin-liang,Feng Dan,Wang Juang,et al.High AvailabilityStorage System Based on Two-level Metadata Management[C]∥Proceedings of Frontier of Computer Science and Technology(FCST 2007).2007:41-48
[2] Mackey G,Sehrish S,Wang Jun.Improving Metadata Management for Small Files in HDFS[C]∥Proceedings of Cluster Computing and Workshops.2009:1-4
[3] Yu Wei-kuan,Oral H S,Canon R S,et al.Empirical Analysis of a Large-Scale Hierarchical Storage System[C]∥Euro-Par 2008-Parallel Processing.2008:130-140
[4] Yu Wei-kuan,Jeffrey S V,et al.Performance Characterizationand Optimization of Parallel I/O on the Cray XT[C]∥The 22nd IEEE International Parallel and Distributed Processing Symposium(IPDPS2008).2008:1-11
[5] Dittrich J,Quiane R J A,Jindal A,et al.Hadoop++:Making a Yellow Elephant Run Like a Cheetah(Without It Even Noticing)[J].Proceedings of the VLDB Endowment,2010(3):515-529
[6] OMalley O.The Anatomy of Hadoop I/O Pipeline[EB/OL].http://developer.yahoo.com/
[7] Liu Xu-hui,Han Ji-zhong,Zhong Yun-qin,et al.Implementing WebGIS on Hadoop:A case study of improving small file I/O performance on HDFS[C]∥Cluster Computing and Workshops(CLUSTER’09).2009:1-8
[8] Dong Bo,Qiu Jie,Zheng Qing-hua,et al.A Novel Approach to Improving the Efficiency of Storing and Accessing Small Files on Hadoop:A Case Study by PowerPoint Files[C]∥Services Computing Conference(SCC).2009:65-72

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!