Computer Science ›› 2015, Vol. 42 ›› Issue (9): 45-49.doi: 10.11896/j.issn.1002-137X.2015.09.010

Previous Articles     Next Articles

Dynamic Scheduling Algorithm in Hadoop Platform

GAO Yan-fei, CHEN Jun-jie and QIANG Yan   

  • Online:2018-11-14 Published:2018-11-14

Abstract: With the increasing of the clusters and the user’s QoS in the cloud environment,it becomes much harder to meet the requirements of jobs and users using the traditional strategy.To adjust scheduler dynamically according to the status of the jobs and the resources,this paper proposed a dynamic scheduling method based on the job classification method in the Hadoop platform.The proposed method employs the Nave Bayesian method to classify the jobs in which the human inferences are added to preset the jobs’ weight according to the types.Then, the scheduling priority of the jobs is set dynamically using the utility function based on the user’s expected completing time and the estimated completed time of jobs.The experimental results show that the proposed method can not only reduce the classification time,but also improve the scheduling dynamics and user’s QoS greatly.

Key words: Human-computer interaction,Hadoop,Dynamic scheduling,Bayesian network,QoS

[1] 朱晓敏,祝江汉,马满好.异构集群系统中具有 QoS 需求的实时任务容错调度[J].软件学报,2011,22(7):1440-1456 Zhu Xiao-min,Zhu Jiang-han,Ma Man-chao.Fault-Tolerant Scheduling for Real-time Tasks with QoS Requirements on Heterogeneous Clusters[J].Journal of Software,2011,2(7):1440-1456
[2] Yang Y,Xiang P,Mantor M,et al.CPU-Assisted GPGPU on Fused CPU-GPU Architecture[C]∥the 8th International Symposium on High Performance Computer Architecture(HPCA-18).2012:103-114
[3] Dutta D,Joshi R C.A genetic:algorithm approach to cost-based multi-QoS job scheduling in cloud computing environment[C]∥ICWET.2011:422-427
[4] 王凯,侯紫峰.动态调整虚拟机权重参数的调度方法[J].计算机研究与发展,2012,48(11):2094-2102 Wang Kai,Hou Zi-feng.An Adaptive Scheduling Method of Weight Parameter Adjustment on Virtual Machines [J].Journal of Computer Research and Development,2012,8(11):2094-2102
[5] 王守信,张莉,李鹤松.一种基于云模型的主观信任评价方法[J].软件学报,2010,21(6):1341-1352 Wang Shou-xin,Zhang Li,Li He-song.Evaluation Approach of Subjective Trust Based on Cloud Model[J].Journal of Software,2010,1(6):1341-1352
[6] Rasooli A,Down D.An Adaptive Scheduling Algorithm for Dynamic Heterogeneous Hadoop Systems [C]∥Proceedings of the 2011 Conference of the Center for Advanced Studies on Collaborative Research.IBM Corp.,2011:30-44
[7] Chen Q,Zhang D,Guo M,et al.Samr:A self-adaptive mapreduce scheduling algorithm in heterogeneous environment[C]∥2010 IEEE 10th International Conference on Computer and Information Technology(CIT).IEEE,2010:2736-2743
[8] 陈全,邓倩妮.异构环境下动态的Map-Reduce调度[J].计算机工程与科学,2009,31(S1):168-171,175 Chen Quan,Deng Qian-ni.Self-Adaptive Map-reduce Scheduling Under Heterogeneous Environment[J].Computer Engineering & Science,2009,1(S1):168-171,5
[9] Mao Hong,Hu Sheng-qiu,Zhang Zhen-zhong,et al.A Load-Driven Task Scheduler with Adaptive DSC for MapReduce[C]∥GREENCOM.2011:28-33
[10] Kaushik R T,Bhandarkar M.Greenhdfs:towards an energy-conserving,storage-efficient,hybrid hadoop compute cluster[C]∥Proceedings of the USENIX Annual Technical Conference.2010
[11] Lu P,Lee Y C,Wang C,et al.Workload Characteristic Oriented Scheduler for MapReduce[C]∥2012 IEEE 18th International Conference on Parallel and Distributed Systems(ICPADS).IEEE,2012:156-163
[12] 李强,郝沁汾,肖利民,等.云计算中虚拟机放置的动态管理与多目标优化[J].计算机学报,2011,34(12):2253-2264 Li Qiang,Hao Qin-fen,Xiao Li-min,et al.An Integrated Approach to Automatic Management of Virtualized Resources in Cloud Environments [J].Chinese Journal of Computers,2011,4(12):2253-2264
[13] Sharma B,Chudnovsky V,Hellerstein J L,et al.Modeling and synthesizing task placement constraints in Google compute clusters[C]∥SOCC.2011
[14] Deng Ke-feng,Song Jun-qiang,Ren Kai-jun,et al.Graph-Cut Based Coscheduling Strategy Towards Efficient Execution of Scientific Workflows in Collaborative Cloud Environments[C]∥GRID.2011:34-41
[15] Dhok J,Varma V.Using pattern classification for task assignment in mapreduce[C]∥International Institute of Information Technology.Hyderabad,India,2005
[16] Polo J,Castillo C,Carrera D,et al.Resource-aware Adaptive Scheduling for MapReduce Clusters[C]∥ACM/IFIP/USENIX 12th International Middleware Conference(Middleware 2011).2011

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!