Computer Science ›› 2016, Vol. 43 ›› Issue (Z11): 11-15.doi: 10.11896/j.issn.1002-137X.2016.11A.003

Previous Articles     Next Articles

Adaptive Fault-tolerant Scheduling Algorithm for Unresponsive Task Based on Speculation

CUI Yun-fei, WU Xiao-jin, DAI Ye, CHENG Xiao and GUO Gang   

  • Online:2018-12-01 Published:2018-12-01

Abstract: Current fault-tolerant scheduling algorithm for unresponsive task,based on static execution failed time threshold,can not adapt to dynamic cluster load of large data processing center.To address this issue,an adaptive execution failed time threshold method was proposed.Based on this method,an adaptive fault-tolerant scheduling algorithm (AFTS) for unresponsive task was designed.AFTS adjusts unresponsive task’s time threshold to be determined failure dynamically and to reduce the job response time,according to the information of job size,the size of individual tasks and the remaining operating time.A prototype system using AFTS is developed,on which the validation of the adaptive execution failed time threshold method and the evaluation of AFTS’s performance are carried out.It is shown that AFTS outperforms current fault-tolerant scheduling algorithm in term of the job response time.

Key words: Big data,Fault-tolerant scheduling algorithm,Adaptive,Speculative,MapReduce

[1] Dean J,Ghemawat S.MapReduce:simplified data processing on large clusters [J].Communications of the ACM,2008,51(1):107-113
[2] 陆嘉恒.Hadoop实战[M].机械工业出版社,2012
[3] Adaptive Scheduler[EB/OL].https://issues.apache.org/jira/browse/MAPREDUCE-1380,3
[4] Improve speculative execution [EB/OL].https://issues.apache.org/jira/browse/MAPREDUCE-2039,3
[5] Speculative execution for Reads [EB/OL].https://issues.apa-che.org/jira/browse/CASSANSRA-4705,3
[6] Looking for speculative tasks is very expensive [EB/OL].https://issues.apache.org/jira/browse/MAPREDUCE-4499,3
[7] Dinu F, Ng T S E.Understanding the Effects and Implications of Compute Node Related Failures in Hadoop[R].HPDC’12.2012:18-22
[8] Lee K H,Lee Y J,Choi H,et al.Parallel Data Processing with MapReduce:A Survey[J].SIGMOD Record,2011,0(4):11-20
[9] Matei Z,Andy K,Anthony D.Improving MapReduce Performance in Heterogeneous Environments[C]∥8th Usenix Symposium on Operating Systems Design and Implementation.2008
[10] ResourceManagerRest [EB/OL].http://hadoop.apace.org/docs/r0.23.6,2013

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75, 88 .
[2] XIA Qing-xun and ZHUANG Yi. Remote Attestation Mechanism Based on Locality Principle[J]. Computer Science, 2018, 45(4): 148 -151, 162 .
[3] LI Bai-shen, LI Ling-zhi, SUN Yong and ZHU Yan-qin. Intranet Defense Algorithm Based on Pseudo Boosting Decision Tree[J]. Computer Science, 2018, 45(4): 157 -162 .
[4] WANG Huan, ZHANG Yun-feng and ZHANG Yan. Rapid Decision Method for Repairing Sequence Based on CFDs[J]. Computer Science, 2018, 45(3): 311 -316 .
[5] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[6] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[7] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[8] LIU Qin. Study on Data Quality Based on Constraint in Computer Forensics[J]. Computer Science, 2018, 45(4): 169 -172 .
[9] ZHONG Fei and YANG Bin. License Plate Detection Based on Principal Component Analysis Network[J]. Computer Science, 2018, 45(3): 268 -273 .
[10] SHI Wen-jun, WU Ji-gang and LUO Yu-chun. Fast and Efficient Scheduling Algorithms for Mobile Cloud Offloading[J]. Computer Science, 2018, 45(4): 94 -99, 116 .