Computer Science ›› 2018, Vol. 45 ›› Issue (6): 67-71,95.doi: 10.11896/j.issn.1002-137X.2018.06.011

• WISA2023 • Previous Articles     Next Articles

Monitoring and Dispatching Service for Heterogeneous Big Data Computing Frameworks

HU Ya-peng, DING Wei-long, WANG Gui-ling   

  1. Data Engineering Institute,North China University of Technology,Beijing 100144,China;
    Beijing Key Laboratory on Integration and Analysis of Large-scale Stream Data,Beijing 100144,China
  • Received:2017-03-11 Online:2018-06-15 Published:2018-07-24

Abstract: Various types of large data computing frameworks have their own management methods.The operation of traditional monitoring and scheduling service in heterogeneous environment is limited by the global status of cluster.It not only wastes resource of cluster,but also suffers long executive latencies of job.To solve these problems above,this paper presented an integrated monitoring and dynamic scheduling management service for heterogeneous big data computing framework.The service can monitor multiple types of computing framework automatically and provide integrated dispatching for diverse computing jobs.The work was implemented on Hadoop and Storm.The experimental results show that the service can reduce the complexity of manual operation in heterogeneous environment and improve job scheduling efficiency.

Key words: Job scheduling, Cluster monitoring, Management service, Job submission

CLC Number: 

  • TP315
[1]DONG B,SHEN Q,XIAO D B.Research on Monitoring Method of Cloud Computing Cluster Server System[J].Computer Engineering&Science,2012,34(10):69-71.(in Chinese)
董波,沈青,肖德宝.云计算集群服务器系统监控方法的研究[J].计算机工程与科学,2012,34(10):69-71.
[2]DARLING C L,GERNAEY M E,KALDESTAD H S,et al. Dynamic monitor and controller of availability of a load-balancing cluster,US 7,296,268 B2[P].2007-11-13.
[3]YUAN K.Design and Implementation of Monitoring System Based on Cloud[D].Wuhan:Huazhong University of Science and Technology,2012.(in Chinese)
袁凯.云计算环境下的监控系统设计与实现[D].武汉:华中科技大学,2012.
[4]LIU J,LIU F,ANSARI N.Monitoring and analyzing big traffic data of a large-scale cellular network with Hadoop[J].IEEE Network,2014,28(4):32-39.
[5]GUO X H,LI R Z,ZHANG Q,et al.Application research on distributed Zabbix network monitoring system[J].Journal on Communications,2013,34(Z2):95-98.(in Chinese)
郭晓慧,李润知,张茜,等.基于Zabbix的分布式服务器监控应用研究[J].通信学报,2013,34(Z2):95-98.
[6]YU G F,WANG Y C ,ZHUANG L Y,et al.Response Delay Predictive and Load Scheduling Control for the Cluster Server[J].Computer Systems Applications,2007,30(7):26-29.(in Chinese)
于国防,王耀才,庄立运,等.集群服务器响应延时预测及其负载调度控制[J].计算机系统应用,2007,30(7):26-29.
[7]XU G,YU W,CHEN Z.A Cloud Computing Based System for Cyber Security Management[J].International Journal of Parallel,Emergent and Distributed System,2016,30(1):29-45.
[8]ARAVINTH S S,BEGAM A H,SHANMUGAPRIYAA S,et al.An Efficient Hadoop Frameworks SQOOP and Ambari for Big Data Processing[J].International Journal for Innovative Reasarch in Science & Technology,2015,1(10):252-255.
[9]ZAHARIA M,BORTHAKUR D,SARMA J S,et al.JobSche-duling for Multi-User MapReduce Cluster:Technical Report No.UCB/EECS-2009-55[R].2009.
[10]WANG X T,SHEN D R,NIE T Z,et al.Batch-Job Scheduling in Shared MapReduce Environment[J].Journal of Computer Research and Development,2013,50(Suppl.):332-341.(in Chinese)
王习特,申德荣,聂铁铮,等.共享的MapReduce环境下批量作业的调度算法研究[J].计算机研究与发展,2013,50(Supp.):332-341.
[11]LI Q M,ZHANG S X,LU L,et al.A Job Scheduling Algorithm and Hybrid Scheduling Method on Hadoop Platform[J].Journal of Computer Research and Development,2013,50(Suppl.):361-368.(in Chinese)
李千目,张晟骁,陆路,等.一种Hadoop平台下的调度算法及混合调度策略[J].计算机研究与发展,2013,50(Suppl.):361-368.
[12]LI B F,ZHU Y Z,WEI R H.Implementation of Load Balancing Technology on Heterogeneous Beowulf System[J].Computer Technology And Development,2008,18(7):61-65.(in Chinese)
李丙锋,祝永志,魏榕晖.异构Beowulf系统负载均衡技术的研究与实现[J].计算机技术与发展,2008,18(7):61-65.
[13]TANG S,LEE B.DynamincMR:A Dynamic Slot Allocation Optimization Framework for MapReduce Clusters[J].IEEETransa-ctions on Cloud Computing,2014,2(3):333-346.
[14]TIAN G Z,XIAO C B,XU Z S,et al.Hybrid Scheduling Strategy for Multiple DAGS Workflow in Heterogeneous System[J].Journal of Software,2012,23(10):2720-2734.(in Chinese)
田国忠,肖创柏,徐竹胜,等.异构分布式环境下多DAG工作流的混合调度策略[J].软件学报,2012,23(10):2720-2734.
[15]ZAHARIA M,KONWINSKI A,JOSEPH A D,et al.Improving mapreduce performance in heterogeneous environments[C]//Usenix Conference on Operating Systems Design & Implementation.2008:29-42.
[16]XU C,LIU H,TAN L.New Mechanism of Monitoring on Hadoop Cloud Platform[J].Computer Science,2013,40(1):112-117.(in Chinese)
许丞,刘洪,谭良.Hadoop云平台的一种新的任务调度和监控机制[J].计算机科学,2013,40(1):112-117.
[1] LI Zhi-jia, HU Xiang, JIAO Li and WANG Wei-feng. Performance Evaluation of Job Scheduling and InfiniBand Network Interconnection in High Performance Computing System Based on Stochastic Petri Nets [J]. Computer Science, 2015, 42(1): 33-37,46.
[2] . Research on Improving Hadoop Job Scheduling Based on Learning Approach [J]. Computer Science, 2012, 39(Z6): 220-222,256.
[3] . Research of Job Scheduling Strategy of High-performance Computer Based on Adaptive Power Management [J]. Computer Science, 2012, 39(10): 313-317.
[4] ZHUO Cui-min LI Lu-qun. Dynamic Job Scheduling Model of Mobile Sensor Sink in Wireless Sensor [J]. Computer Science, 2011, 38(Z10): 356-358.
[5] HE Jun-Mei ,ZOU Xian-Chun (Faculty of Computer and Information Science, Southwest University, Chongqing 400715). [J]. Computer Science, 2007, 34(7): 282-283.
[6] ZHANG Guo-Bin, PAN Jin-Gui (State Key Lab. for Novel Software,Nanjing University, Nanjing 210093). [J]. Computer Science, 2007, 34(7): 279-281.
[7] . [J]. Computer Science, 2007, 34(6): 128-130.
[8] WU Dai-Xian, YANG Juan, QIU Yu-Hui (The Faculty of Computer and Information Science, SWU,Chongqing 400715). [J]. Computer Science, 2007, 34(3): 254-255.
[9] . [J]. Computer Science, 2006, 33(2): 35-37.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] . [J]. Computer Science, 2018, 1(1): 1 .
[2] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75, 88 .
[3] XIA Qing-xun and ZHUANG Yi. Remote Attestation Mechanism Based on Locality Principle[J]. Computer Science, 2018, 45(4): 148 -151, 162 .
[4] LI Bai-shen, LI Ling-zhi, SUN Yong and ZHU Yan-qin. Intranet Defense Algorithm Based on Pseudo Boosting Decision Tree[J]. Computer Science, 2018, 45(4): 157 -162 .
[5] WANG Huan, ZHANG Yun-feng and ZHANG Yan. Rapid Decision Method for Repairing Sequence Based on CFDs[J]. Computer Science, 2018, 45(3): 311 -316 .
[6] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[7] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[8] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[9] LIU Qin. Study on Data Quality Based on Constraint in Computer Forensics[J]. Computer Science, 2018, 45(4): 169 -172 .
[10] ZHONG Fei and YANG Bin. License Plate Detection Based on Principal Component Analysis Network[J]. Computer Science, 2018, 45(3): 268 -273 .