Computer Science ›› 2017, Vol. 44 ›› Issue (Z11): 397-399.doi: 10.11896/j.issn.1002-137X.2017.11A.083

Previous Articles     Next Articles

Improvement of HDFS Balanced Placement Strategy

YUAN Li-na   

  • Online:2018-12-01 Published:2018-12-01

Abstract: The default data replica placement policy for HDFS is only measured using a single metric based on disk space,the true load balancing of each node cannot be realized. This paper proposed an improvement strategy of the load balancing based on performance through five aspects,such as disk space,CPU processing power,memory processing power,disk read/write,bandwidth,and a load capacity model was defined.Experimental results indicate that the improvement strategy is better than the default policy.

Key words: HDFS,Balanced placement strategy,Load capacity model

[1] 周江,等.面向大数据分析的分布式文件系统关键技术[J].计算机研究与发展,2014,51(2):382-394.
[2] 罗鹏,等.HDFS数据存放策略的研究与改进[J].计算机工程与设计,2014,35(4):1127-1131.
[3] 孙知信,等.基于云计算的数据存储技术研究[J].南京邮电大学学报,2014,34(4):13-19.
[4] 王海荣,等.基于Hadoop的海量数据存储系统设计[J].科技通报,2014,30(9):127-130.
[5] 张柄虹,等.空间高效的分布式数据存储方案[J].计算机应用研究,2015,32(5):1508-1511.
[6] 马晓亭.数字图书馆大数据分布式存储架构模式与策略研究[J].新世纪图书馆,2015(5):43-46.
[7] 康承昆,等.一种基于多衡量指标的HDFS负载均衡算法[J].四川大学学报,2014,51(6):1163-1169.
[8] 熊安萍,等.一种基于混合索引的HDFS小文件存储策略[J].重庆邮电大学学报,2015,27(1):97-102.
[9] 英昌甜,等.一种面向低延迟的内存HDFS数据存储策略[J].微电子学与计算机,2014,31(11):160-166.
[10] 尹颖,等.HDFS中高效存储小文件的方法[J].计算机工程与设计,2015,36(2):406-409.
[11] 卢美莲,等.基于CMM模型的HDFS负载均衡策略[J].北京邮电大学学报,2014,37(5):20-25.
[12] 孟祥萍,等.基于hadoop云平台的智能电网HDFS资源存储技术研究[J].电测与仪表,2014,51(19):24-30.
[13] 张华伟,等.基于多目标优化的云存储副本分布策略的研究[J].计算机科学,2015,42(4):44-50.
[14] 肖达,苏丽裕,王俊龙,等.CSFS:云存储服务支撑的文件系统设计与实现[J].北京邮电大学学报,2015,38(6):77-82.
[15] 罗芳,等.基于多属性的海量Web数据关联存储及检索系统[J].计算机工程与科学,2014,36(3):404-410.
[16] 宋宝燕,等.基于范德蒙码的HDFS优化存储策略研究[J].计算机学报,2015,38(9):1825-1837.
[17] 肖玉泽,等.HDFS下海量小文件高效存储于索引方法[J].小型微型计算机系统,2015,36(10):2218-2223.
[18] MISHNE G,DALTON J,LI Z H,et al.Fast data in the era of big data:Twitter’s real-time related query suggestion architecture[C]∥Proceedings of the ACM SIGMOD International Conference on Management of Data.2013:1147-1157.

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!