Computer Science ›› 2013, Vol. 40 ›› Issue (Z11): 270-273.

Previous Articles     Next Articles

Data Placement Algorithm for Large-scale Storage System

ZHENG Sheng and LI Tong   

  • Online:2018-11-16 Published:2018-11-16

Abstract: With the era of big data coming,t PB and EB even ZB-level dataset makes storage system scalable.Traditional data distribution algorithm was confronted with serious challenge because of different performance storage devices added and the old ones quitted,even multiple devices failed simultaneously.A new hash mapping algorithm was proposed which supports the node weight and multi-replica and also considers node failure and node overload.The algorithm can adapt dynamically to change of storage nodes and promises data even distribution probabilistically for different performance nodes.Besides,the one can effectively deal with node failure and node overload which can improve the availability and performance of the system.

Key words: Distributed file system,Scalability,Data placement,Data migration

[1] Goel A,Shahabi C,Yao D S,et al.SCADDAR:An efficient randomized technique to reorganize continuous media blocks [C]∥Proc of the 18th Int Conf on Data Engineering(ICDE 02).Piscataway,NJ:IEEE,2002:473-482
[2] Litwin W, Risch T.LH*g:a high-availability scalable distributed data structure by record grouping[J].IEEE Transactions on Knowledge and Data Engineering,2002,14(4):923-927
[3] 刘仲,周兴铭,等.基于动态区映射的数据对象布局算法[J].软件学报,2005,16(11):1886-1893
[4] Honicky R J,Miller E L.A fast algorithm for online placement and reorganization of replicated data[C]∥Dongarra J,ed.Proc.of the 17th Int’l Parallel & Distributed Processing Symp.Nice:IEEE Computer Society,2003
[5] Honicky R J,Miller E L.Replication under scalable hashing:A family of algorithms for scalable DecentRalized data distribution[C]∥Proceedings of the 18th International Pallel & Distributed Processing Symposium.Santa Fe,NM,2004
[6] 穆飞,薛巍,舒继武,等.一种面向大规模存储系统的数据副本映射算法[J].计算机研究与发展,2009,3:492-497
[7] 罗象宏舒继武.存储系统中的纠删码研究综述[J].计算机研究与展,2012,9(1):1-11
[8] Peter S,Gerhard W,Peter Z.Data partitioning and load balancing in parallel disk systems[J].The VLDB Journal,1998(7):48-66
[9] 潘承洞,潘承彪,等.初等数论(第三版)[M].北京:北京大学出版社,2013
[10] 郑胜,郝毫毫.基于贝努力大数定律的数据分布算法[J].计算机工程,2009,10(19):59-61

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!