Computer Science ›› 2016, Vol. 43 ›› Issue (Z11): 11-15.doi: 10.11896/j.issn.1002-137X.2016.11A.003

Previous Articles     Next Articles

Adaptive Fault-tolerant Scheduling Algorithm for Unresponsive Task Based on Speculation

CUI Yun-fei, WU Xiao-jin, DAI Ye, CHENG Xiao and GUO Gang   

  • Online:2018-12-01 Published:2018-12-01

Abstract: Current fault-tolerant scheduling algorithm for unresponsive task,based on static execution failed time threshold,can not adapt to dynamic cluster load of large data processing center.To address this issue,an adaptive execution failed time threshold method was proposed.Based on this method,an adaptive fault-tolerant scheduling algorithm (AFTS) for unresponsive task was designed.AFTS adjusts unresponsive task’s time threshold to be determined failure dynamically and to reduce the job response time,according to the information of job size,the size of individual tasks and the remaining operating time.A prototype system using AFTS is developed,on which the validation of the adaptive execution failed time threshold method and the evaluation of AFTS’s performance are carried out.It is shown that AFTS outperforms current fault-tolerant scheduling algorithm in term of the job response time.

Key words: Big data,Fault-tolerant scheduling algorithm,Adaptive,Speculative,MapReduce

[1] Dean J,Ghemawat S.MapReduce:simplified data processing on large clusters [J].Communications of the ACM,2008,51(1):107-113
[2] 陆嘉恒.Hadoop实战[M].机械工业出版社,2012
[3] Adaptive Scheduler[EB/OL].https://issues.apache.org/jira/browse/MAPREDUCE-1380,3
[4] Improve speculative execution [EB/OL].https://issues.apache.org/jira/browse/MAPREDUCE-2039,3
[5] Speculative execution for Reads [EB/OL].https://issues.apa-che.org/jira/browse/CASSANSRA-4705,3
[6] Looking for speculative tasks is very expensive [EB/OL].https://issues.apache.org/jira/browse/MAPREDUCE-4499,3
[7] Dinu F, Ng T S E.Understanding the Effects and Implications of Compute Node Related Failures in Hadoop[R].HPDC’12.2012:18-22
[8] Lee K H,Lee Y J,Choi H,et al.Parallel Data Processing with MapReduce:A Survey[J].SIGMOD Record,2011,0(4):11-20
[9] Matei Z,Andy K,Anthony D.Improving MapReduce Performance in Heterogeneous Environments[C]∥8th Usenix Symposium on Operating Systems Design and Implementation.2008
[10] ResourceManagerRest [EB/OL].http://hadoop.apace.org/docs/r0.23.6,2013

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!