关键词: 大数据,容错调度,自适应,推测的,MapReduce

Abstract: Current fault-tolerant scheduling algorithm for unresponsive task,based on static execution failed time threshold,can not adapt to dynamic cluster load of large data processing center.To address this issue,an adaptive execution failed time threshold method was proposed.Based on this method,an adaptive fault-tolerant scheduling algorithm (AFTS) for unresponsive task was designed.AFTS adjusts unresponsive task’s time threshold to be determined failure dynamically and to reduce the job response time,according to the information of job size,the size of individual tasks and the remaining operating time.A prototype system using AFTS is developed,on which the validation of the adaptive execution failed time threshold method and the evaluation of AFTS’s performance are carried out.It is shown that AFTS outperforms current fault-tolerant scheduling algorithm in term of the job response time.

Key words: Big data,Fault-tolerant scheduling algorithm,Adaptive,Speculative,MapReduce