关键词: 数据本地性,抢占式,作业平均完成时间

Abstract: Local scheduling algorithm is an algorithm to improve data locality in Hadoop cluster environment.The nature of the scheduling strategy of the local scheduling algorithm is to improve the data locality,reduce network transmission and avoid congestion.However,due to the different completion time of the Map task,the waiting phenomenon of Reduce task affects the completion average time of the job,the completion time of the job is increased,and then the performance parameters of the system are not good.In this thesis,we proposed to integrate the preemptive scheduling based on the local requirement of the original algorithm.When the Reduce task waits,the task is supended and the resource is rleased to other Map tasks.Based on the above scheduling strategy,this thesis designed the qualitative scheduling of integrated preemptive strategy.In order to validate the improved algorithm,the local scheduling algorithm and the integrated preemptive local scheduling algorithm were compared by experiments.Experimental results show that,on the same data,the average completion time of the integrated preemptive local scheduling algorithm is significantly reduced.

Key words: Data locality,Preemptive,Average completion time for the job