计算机科学 ›› 2015, Vol. 42 ›› Issue (5): 24-27.doi: 10.11896/j.issn.1002-137X.2015.05.005

• 综述 • 上一篇    下一篇

不同MapReduce运行系统的性能测试与分析

易修文,李天瑞,张钧波,滕 飞   

  1. 西南交通大学信息科学与技术学院 成都610031,西南交通大学信息科学与技术学院 成都610031,西南交通大学信息科学与技术学院 成都610031,西南交通大学信息科学与技术学院 成都610031
  • 出版日期:2018-11-14 发布日期:2018-11-14
  • 基金资助:
    本文受国家自然科学基金(61175047,3),国家自然科学基金联合基金(U1230117),四川省科技支撑计划项目(2012RZ0009),中央高校基本科研业务费专项资金(SWJTU11ZT08)资助

Performance Testing and Analysis among Different MapReduce Runtime Systems

YI Xiu-wen, LI Tian-rui, ZHANG Jun-bo and TENG Fei   

  • Online:2018-11-14 Published:2018-11-14

摘要: 随着云计算技术的发展,许多MapReduce运行系统被开发出来,如Hadoop、Phoenix和Twister等。直观上,Hadoop具有很强的可扩展性、稳定性,适合处理大规模离线应用;Phoenix具有运行速度快等优点,适合处理数据密集型任务;Twister是轻量级的迭代系统,非常适合迭代式的应用。不同的应用在不同的MapReduce运行系统中有着不同的性能。通过测试不同应用在这些运行系统上的性能,给出了实验比较和性能分析,从而为大数据处理时选择合适的并行编程模型提供依据。

关键词: 云计算,MapReduce,Hadoop,Phoenix,Twister

Abstract: With the development of cloud computing technology,several implementations which adopt MapReduce mo-del,e.g.,Hadoop,Phoenix and twister,have been developed.Hadoop has high scalability and stability,thus is suitable for handling large-scale off-line applications.The primary advantage of Phoenix,which is especially appropriate for data-intensive tasks,is its processing speed.Twister,a lightweight iterative runtime system,is designed for iterative applications.Different applications produce different levels of performance on different MapReduce runtime systems.By testing various applications using the aforementioned runtime systems,the experimental comparison and performance analysis were presented,providing the basis for the selection of parallel programming models for big data processing.

Key words: Cloud computing,MapReduce,Hadoop,Phoenix,Twister

[1] Pan Y,Zhang J B.Parallel Programming on Cloud Computing Platforms:Challenges and Solutions[J].KITCS/FTRA Journal of Convergence,2012,3(4):23-28〖ZK)
[2] Armbrust M,Fox A,Griffith R,et al.A view of cloud computing[J].Communications of the ACM,2010,53(4):50-58
[3] Dean J,Ghemawat S.MapReduce:simplified data processing on large clusters[J].Communications of the ACM,2008,51(1):107-113
[4] White T.Hadoop:the definitive guide(2nd ed)[M].O’Reilly,2012
[5] Talbot J,Yoo R M,Kozyrakis C.Phoenix++:modular MapReduce for shared-memory systems[C]∥Proceedings of the Se-cond International Workshop on MapReduce and its Applications.ACM,2011:9-16
[6] Ekanayake J,Li H,Zhang B,et al.Twister:a runtime for iterative mapreduce[C]∥Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing.ACM,2010:810-818
[7] He B,Fang W,Luo Q,et al.Mars:a MapReduce framework on graphics processors[C]∥Proceedings of the 17th international conference on Parallel architectures and compilation techniques.ACM,2008:260-269
[8] Zaharia M,Chowdhury M,Das T,et al.Resilient distributeddatasets:A fault-tolerant abstraction for in-memory cluster computing[C]∥Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation.USENIX Association,2012:2
[9] Kang L Y,Wang X Y,Bai R J.Analysis of MapReduce Principle and Its Main Implementation Platforms[J].New Technology of Library and Information Service,2012(02):60-67
[10] Li J J,Li Q,Tian B.The Analysis and Comparison between MapReduce Implementations[J/OL].http://www.paper.edu.cn/html/releasepaper/2011/11/464.2011
[11] Li J J,C J,W D,et al.Survey of MapReduce Parallel Programming Model[J].ACTA Electronica Sinica,2011(11):2635-2642
[12] Borthakur D.The hadoop distributed file system:Architectureand design.http://hadoop.apache.ogr/docs/ro.18.1/hdsf_design.html
[13] Thusoo A,Sarma J S,Jain N,et al.Hive:a warehousing solution over a map-reduce framework[J].Proceedings of the VLDB Endowment,2009,2(2):1626-1629
[14] Olston C,Reed B,Srivastava U,et al.Pig latin:a not-so-foreign language for data processing[C]∥Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data.ACM,2008:1099-1110
[15] Abouzeid A,Bajda-Pawlikowski K,Abadi D,et al.HadoopDB:an architectural hybrid of MapReduce and DBMS technologies for analytical workloads[J].Proceedings of the VLDB Endowment,2009,2(1):922-933
[16] Anil R,Dunning T,Friedman E.Mahout in action[M].Manning publications,2011

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!