Computer Science ›› 2018, Vol. 45 ›› Issue (6): 67-71,95.doi: 10.11896/j.issn.1002-137X.2018.06.011

• WISA2023 • Previous Articles     Next Articles

Monitoring and Dispatching Service for Heterogeneous Big Data Computing Frameworks

HU Ya-peng, DING Wei-long, WANG Gui-ling   

  1. Data Engineering Institute,North China University of Technology,Beijing 100144,China;
    Beijing Key Laboratory on Integration and Analysis of Large-scale Stream Data,Beijing 100144,China
  • Received:2017-03-11 Online:2018-06-15 Published:2018-07-24

Abstract: Various types of large data computing frameworks have their own management methods.The operation of traditional monitoring and scheduling service in heterogeneous environment is limited by the global status of cluster.It not only wastes resource of cluster,but also suffers long executive latencies of job.To solve these problems above,this paper presented an integrated monitoring and dynamic scheduling management service for heterogeneous big data computing framework.The service can monitor multiple types of computing framework automatically and provide integrated dispatching for diverse computing jobs.The work was implemented on Hadoop and Storm.The experimental results show that the service can reduce the complexity of manual operation in heterogeneous environment and improve job scheduling efficiency.

Key words: Job scheduling, Cluster monitoring, Management service, Job submission

CLC Number: 

  • TP315
[1]DONG B,SHEN Q,XIAO D B.Research on Monitoring Method of Cloud Computing Cluster Server System[J].Computer Engineering&Science,2012,34(10):69-71.(in Chinese)
[2]DARLING C L,GERNAEY M E,KALDESTAD H S,et al. Dynamic monitor and controller of availability of a load-balancing cluster,US 7,296,268 B2[P].2007-11-13.
[3]YUAN K.Design and Implementation of Monitoring System Based on Cloud[D].Wuhan:Huazhong University of Science and Technology,2012.(in Chinese)
[4]LIU J,LIU F,ANSARI N.Monitoring and analyzing big traffic data of a large-scale cellular network with Hadoop[J].IEEE Network,2014,28(4):32-39.
[5]GUO X H,LI R Z,ZHANG Q,et al.Application research on distributed Zabbix network monitoring system[J].Journal on Communications,2013,34(Z2):95-98.(in Chinese)
[6]YU G F,WANG Y C ,ZHUANG L Y,et al.Response Delay Predictive and Load Scheduling Control for the Cluster Server[J].Computer Systems Applications,2007,30(7):26-29.(in Chinese)
[7]XU G,YU W,CHEN Z.A Cloud Computing Based System for Cyber Security Management[J].International Journal of Parallel,Emergent and Distributed System,2016,30(1):29-45.
[8]ARAVINTH S S,BEGAM A H,SHANMUGAPRIYAA S,et al.An Efficient Hadoop Frameworks SQOOP and Ambari for Big Data Processing[J].International Journal for Innovative Reasarch in Science & Technology,2015,1(10):252-255.
[9]ZAHARIA M,BORTHAKUR D,SARMA J S,et al.JobSche-duling for Multi-User MapReduce Cluster:Technical Report No.UCB/EECS-2009-55[R].2009.
[10]WANG X T,SHEN D R,NIE T Z,et al.Batch-Job Scheduling in Shared MapReduce Environment[J].Journal of Computer Research and Development,2013,50(Suppl.):332-341.(in Chinese)
[11]LI Q M,ZHANG S X,LU L,et al.A Job Scheduling Algorithm and Hybrid Scheduling Method on Hadoop Platform[J].Journal of Computer Research and Development,2013,50(Suppl.):361-368.(in Chinese)
[12]LI B F,ZHU Y Z,WEI R H.Implementation of Load Balancing Technology on Heterogeneous Beowulf System[J].Computer Technology And Development,2008,18(7):61-65.(in Chinese)
[13]TANG S,LEE B.DynamincMR:A Dynamic Slot Allocation Optimization Framework for MapReduce Clusters[J].IEEETransa-ctions on Cloud Computing,2014,2(3):333-346.
[14]TIAN G Z,XIAO C B,XU Z S,et al.Hybrid Scheduling Strategy for Multiple DAGS Workflow in Heterogeneous System[J].Journal of Software,2012,23(10):2720-2734.(in Chinese)
[15]ZAHARIA M,KONWINSKI A,JOSEPH A D,et al.Improving mapreduce performance in heterogeneous environments[C]//Usenix Conference on Operating Systems Design & Implementation.2008:29-42.
[16]XU C,LIU H,TAN L.New Mechanism of Monitoring on Hadoop Cloud Platform[J].Computer Science,2013,40(1):112-117.(in Chinese)
[1] LI Zhi-jia, HU Xiang, JIAO Li and WANG Wei-feng. Performance Evaluation of Job Scheduling and InfiniBand Network Interconnection in High Performance Computing System Based on Stochastic Petri Nets [J]. Computer Science, 2015, 42(1): 33-37,46.
[2] . Research on Improving Hadoop Job Scheduling Based on Learning Approach [J]. Computer Science, 2012, 39(Z6): 220-222,256.
[3] . Research of Job Scheduling Strategy of High-performance Computer Based on Adaptive Power Management [J]. Computer Science, 2012, 39(10): 313-317.
[4] ZHUO Cui-min LI Lu-qun. Dynamic Job Scheduling Model of Mobile Sensor Sink in Wireless Sensor [J]. Computer Science, 2011, 38(Z10): 356-358.
[5] HE Jun-Mei ,ZOU Xian-Chun (Faculty of Computer and Information Science, Southwest University, Chongqing 400715). [J]. Computer Science, 2007, 34(7): 282-283.
[6] ZHANG Guo-Bin, PAN Jin-Gui (State Key Lab. for Novel Software,Nanjing University, Nanjing 210093). [J]. Computer Science, 2007, 34(7): 279-281.
[7] . [J]. Computer Science, 2007, 34(6): 128-130.
[8] WU Dai-Xian, YANG Juan, QIU Yu-Hui (The Faculty of Computer and Information Science, SWU,Chongqing 400715). [J]. Computer Science, 2007, 34(3): 254-255.
[9] . [J]. Computer Science, 2006, 33(2): 35-37.
Full text



[1] . [J]. Computer Science, 2018, 1(1): 1 .
[2] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75, 88 .
[3] XIA Qing-xun and ZHUANG Yi. Remote Attestation Mechanism Based on Locality Principle[J]. Computer Science, 2018, 45(4): 148 -151, 162 .
[4] LI Bai-shen, LI Ling-zhi, SUN Yong and ZHU Yan-qin. Intranet Defense Algorithm Based on Pseudo Boosting Decision Tree[J]. Computer Science, 2018, 45(4): 157 -162 .
[5] WANG Huan, ZHANG Yun-feng and ZHANG Yan. Rapid Decision Method for Repairing Sequence Based on CFDs[J]. Computer Science, 2018, 45(3): 311 -316 .
[6] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[7] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[8] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[9] LIU Qin. Study on Data Quality Based on Constraint in Computer Forensics[J]. Computer Science, 2018, 45(4): 169 -172 .
[10] ZHONG Fei and YANG Bin. License Plate Detection Based on Principal Component Analysis Network[J]. Computer Science, 2018, 45(3): 268 -273 .