计算机科学 ›› 2013, Vol. 40 ›› Issue (1): 112-117.

• 信息安全 • 上一篇    下一篇

Hadoop云平台的一种新的任务调度和监控机制

许 丞,刘 洪,谭 良   

  1. (四川师范大学计算机科学学院 成都610068);(中国科学院计算技术研究所 北京100190)
  • 出版日期:2018-11-16 发布日期:2018-11-16

New Mechanism of Monitoring on Hadoop Cloud Platform

  • Online:2018-11-16 Published:2018-11-16

摘要: 云平台任务监控与资源调度机制是云平台的核心功能之一。Hadoop云平台中任务监控和资源管理的任务是由JobTracker负责处理,并通过slaw节点向其发送心跳消息来实现。这种方式导致JobTracker的负载过重,降低了Hadoop云平台的工作效率,限制了Hadoop云平台的规模。提出了一种新的任务监控方案,该方案将JobTracker 的任务监控和资源管理功能分离,任务监控功能仍由Job Tracker节点完成,资源管理功能由新增的资源管理节点完成,Job Tracker通过增量更新的算法将任务调度所需的对象信息动态同步到资源管理节点上,资源管理节点根据心跳消息进行任务分配,并将分配结果返回给JobTracker节点。实验结果表明,本方案不仅通过监控节点实现了任务的监控,增加了监控的灵活性和鲁棒性,而且降低了Jobtracker节点的负担,可有效提高Hadoop云平台的工作效率和规模。

关键词: 云计算,Hadoop,任务监控,任务调度,资源管理,增量更新算法

Abstract: Job monitoring and resource managing mechanism in cloud computing platform arc the core functions. The tasks of job monitoring and resource managing in Hadoop are assigned to JobTracker which is implemented on receiving heartbeat message sending by slave node, so JobTracker is the bottleneck of Hadoop. A new mechanism of job monitoring and resource managing scheme was presented. In this scheme,job monitoring and resource managing arc separated from original JobTracker, but job monitoring function is still assigned to JobTracker node, and the resource managing function is accomplish by newly added resource monitoring node. JobTracker sends the necessary information of Objects to resource managing node by using a new method of delta updates algorithm. Resource managing node schedules tasks with heartbeat message, and returns the result to Job Tracker node. Experimental results show that the scheme implemenu monitoring node monitor job completely,makes JobTracker node more flexible and robust, reduces the loading of Job Tracker node, improves the efficiency of Hadoop platform.

Key words: Cloud computing, Hadoop, Task scheduling, Resource managing, Delta updating algorithm

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!