Computer Science ›› 2022, Vol. 49 ›› Issue (7): 271-279.doi: 10.11896/jsjkx.210600040

• Computer Network • Previous Articles     Next Articles

Server-reliability Task Offloading Strategy Based on Deep Deterministic Policy Gradient

LI Meng-fei1, MAO Ying-chi1,2, TU Zi-jian1, WANG Xuan1, XU Shu-fang1,2   

  1. 1 College of Computer and Information,Hohai University,Nanjing 210098,China
    2 Key Laboratory of Water Big Data Technology of Ministry of Water Resources,Nanjing 210098,China
  • Received:2021-06-04 Revised:2021-12-07 Online:2022-07-15 Published:2022-07-12
  • About author:LI Meng-fei,born in 1998,postgra-duate,is a member of China Computer Federation.Her main research interests include distributed computing,IoT and edge intelligence computing.
    MAO Ying-chi,born in 1976,Ph.D,professor,is a senior member of China Computer Federation.Her main research interests include distributed computing and edge computing.
  • Supported by:
    Key Research and Development Project of Jiangsu Province,China(BE2020729),Gusu Innovation Leading Ta-lentsSpecial Project(ZXL2020210),2020 Kunshan Zu Chongzhi Key Tack Project and Key Technology Project of China Huaneng Group(HNKJ19-H12, HNKJ20-H64).

Abstract: With the popularization of smart mobile devices,a new generation of mobile applications such as face recognition and virtual reality have gradually emerged.The limited computing power and battery capacity of mobile devices cannot support applications with high computing requirements and latency-sensitive applications.Therefore,mobile edge computing(MEC) is proposed to solve this problem.However,in the MEC environment,the reliability of the edge server is low,and the possible equipment failure will lead to the existing offloading decision failure,which increases the application response time and reduces the user experience.In view of the possible failure of edge servers,and considering that the deep deterministic policy gradient(DDPG) algorithm can better deal with the problem of high-dimensional action space through the network fitting strategy function,this paper proposes a server-reliability task offloading based on deep deterministic policy gradient(SRTO-DDPG).The main work is as follows.Firstly,the failure rate of application execution is reduced by duplicating subtasks for secondary offload.Secondly,the task offloading and resource allocation problems with server reliability constraints to minimize application delay are modeled as Markov decision process(MDP).Finally,an algorithm based on DDPG is used to solve the problem.Simulation results show that the SRTO-DDPG strategy can effectively interact with the environment to obtain the optimal offloading decision,and its perfor-mance is better than the local execution strategy(LE).Compared with the single location task offloading based on deep determi-nistic policy gradient(SLTO-DDPG),this strategy can achieve a low total delay of about 26.16% under reliability constraints,and can better adapt to the reliability problems of edge servers in multi-server scenarios.

Key words: Deep reinforcement learning, Dependent tasks, Mobile edge computing, Resource allocation, Task offloading

CLC Number: 

  • TP399
[1]AL-JANABI S,AL-SHOURBAJI I,SHOJAFAR M,et al.Mobile Cloud Computing:Challenges and Future Research Directions[C]//International Conference on the Developments on Systems Engineering.IEEE,2017:62-67.
[2]RANADHEERA S,MAGHSUDI S,HOSSAIN E.Mobile Edge Computation Offloading Using Game Theory and Reinforcement Learning[J].arXiv:1711.09012,2017.
[3]XU C B,LIU Y,LIU Y X,et al.MEC server selection scheme based on multiple indicators [J].Journal of Chongqing Univer-sity of Posts and Telecommunications(Natural Science Edition),2020,32(3):329-335.
[4]LIANG Y C,CAO B.Artificial neural network methods for task offloading in VANET cloud [J].Journal of Chongqing Univer-sity of Posts and Telecommunications(Natural Science Edition),2020,32(3):336-344.
[5]LI J,ZHANG Y P,PANG L,et al.Joint Resource Allocation and Task Scheduling in Mobile Edge Computing[J].Journal of Chongqing University of Technology(Natural Science),2020,34(11):156-163.
[6]CHEN L.Multicast resource allocation algorithm based on la-yered coding in sparse code multiple access systems[J].Journal of Chongqing University of Posts and Telecommunications(Na-tural Science Edition),2020,32(6):917-924.
[7]MAO Y Y,ZHANG J,BEN K,et al.Dynamic Computation Offloading for Mobile-Edge Computing with Energy Harvesting Devices[J].IEEE Journal on Selected Areas in Communications,2016,34(12):3590-3605.
[8]CEHN X,JIAO L,LI W,et al.Efficient Multi-User Computation Offloading for Mobile-Edge Cloud Computing[J].IEEE/ACM Transactions on Networking,2016,24(5):2795-2808.
[9]CAO H,CAI J.Distributed Multiuser Computation Offloading for Cloudlet-Based Mobile Cloud Computing:A Game-Theoretic Machine Learning Approach[J].IEEE Transactions on Vehicular Technology,2018:752-764.
[10]HUANG L,BI S,ZHANG Y J A.Deep Reinforcement Learning for Online Computation Offloading in Wireless Powered Mobile-Edge Computing Networks[J].IEEE Transactions on Mobile Computing,2019,19(11):2581-2593.
[11]GUO H,LIU J,ZHANG J.Computation Offloading for Multi-Access Mobile Edge Computing in Ultra-Dense Networks[J].IEEE Communications Magazine,2018,56(8):14-19.
[12]ZHU A,GUO S,MA M,et al.Computation Offloading forWorkflow in Mobile Edge Computing Based on Deep Q-Learning[C]//Wireless and Optical Communications Conference.IEEE,2019:1-5.
[13]CHEN X,ZHANG H,WU C,et al.Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning[J].IEEE Internet of Things Journal,2019,6(3):4005-4018.
[14]HUANG B,LI Y,LI Z,et al.Security and Cost-Aware Computation Offloading via Deep Reinforcement Learning in Mobile Edge Computing[J].Wireless Communications and Mobile Computing,2019,2019:1-20.
[15]SHARMA Y,JAVADI B,SI W,et al.Reliability and Energy Efficiency in Cloud Computing Systems:Survey and Taxonomy[J].Journal of Network and Computer Applications,2016,74:66-85.
[16]ALSENANI Y,CROSBY G,VELASCO T.SaRa:A Stochastic Model to Estimate Reliability of Edge Resources in Volunteer Cloud[C]//2018 IEEE International Conference on Edge Computing(EDGE).IEEE,2018:121-124.
[17]LORENZO B,GARCIA-ROIS J,LI X,et al.A Robust Dynamic Edge Network Architecture for the Internet-of-Things[J].IEEE Network,2018,32(1):8-15.
[18]AL-HABOB A A,IBRAHIM A,DOBRE O A,et al.Collision-Free Sequential Task Offloading for Mobile Edge Computing[J].IEEE Communications Letters,2020,24(1):71-75.
[19]LIU J,ZHANG Q.Offloading Schemes in Mobile Edge Computing for Ultra-Reliable Low Latency Communications[J].IEEE Access,2018,16:12825-12837.
[20]HUANG L,FENG X,QIAN L,et al.Deep ReinforcementLearning-Based Task Offloading and Resource Allocation for Mobile Edge Computing[C]//International Conference on Machine Learning and Intelligent Communications.2018:33-42.
[21]LI J,GAO H,LV T J,et al.Deep Reinforcement Learning based Computation Offloading and Resource Allocation for MEC[C]//Wireless Communications and Networking Conference.IEEE,2018:1-6.
[22]SHU C,ZHAO Z,HAN Y,et al.Dependency-Aware and Latency-Optimal Computation Offloading for Multi-User Edge Computing Networks[C]//2019 16th Annual IEEE International Conference on Sensing,Communication,and Networking.IEEE,2019:1-9.
[23]LIU J,ZHANG Q.Code-Partitioning Offloading Schemes inMobile Edge Computing for Augmented Reality[J].IEEE Access,2019,7:11222-11236.
[24]CHEN Z,WANG X.Decentralized Computation Offloading for Multi-User Mobile Edge Computing:A Deep Reinforcement Learning Approach[J].Journal on Wireless Communications and Networking,2020,188:1-21.
[1] YU Bin, LI Xue-hua, PAN Chun-yu, LI Na. Edge-Cloud Collaborative Resource Allocation Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2022, 49(7): 248-253.
[2] TANG Feng, FENG Xiang, YU Hui-qun. Multi-task Cooperative Optimization Algorithm Based on Adaptive Knowledge Transfer andResource Allocation [J]. Computer Science, 2022, 49(7): 254-262.
[3] FANG Tao, YANG Yang, CHEN Jia-xin. Optimization of Offloading Decisions in D2D-assisted MEC Networks [J]. Computer Science, 2022, 49(6A): 601-605.
[4] LIU Zhang-hui, ZHENG Hong-qiang, ZHANG Jian-shan, CHEN Zhe-yi. Computation Offloading and Deployment Optimization in Multi-UAV-Enabled Mobile Edge Computing Systems [J]. Computer Science, 2022, 49(6A): 619-627.
[5] XIE Wan-cheng, LI Bin, DAI Yue-yue. PPO Based Task Offloading Scheme in Aerial Reconfigurable Intelligent Surface-assisted Edge Computing [J]. Computer Science, 2022, 49(6): 3-11.
[6] ZHOU Tian-qing, YUE Ya-li. Multi-Task and Multi-Step Computation Offloading in Ultra-dense IoT Networks [J]. Computer Science, 2022, 49(6): 12-18.
[7] QIU Xu, BIAN Hao-bu, WU Ming-xiao, ZHU Xiao-rong. Study on Task Offloading Algorithm for Internet of Vehicles on Highway Based on 5G MillimeterWave Communication [J]. Computer Science, 2022, 49(6): 25-31.
[8] XU Hao, CAO Gui-jun, YAN Lu, LI Ke, WANG Zhen-hong. Wireless Resource Allocation Algorithm with High Reliability and Low Delay for Railway Container [J]. Computer Science, 2022, 49(6): 39-43.
[9] HONG Zhi-li, LAI Jun, CAO Lei, CHEN Xi-liang, XU Zhi-xiong. Study on Intelligent Recommendation Method of Dueling Network Reinforcement Learning Based on Regret Exploration [J]. Computer Science, 2022, 49(6): 149-157.
[10] SHEN Jia-fang, QIAN Li-ping, YANG Chao. Non-orthogonal Multiple Access and Multi-dimension Resource Optimization in EH Relay NB-IoT Networks [J]. Computer Science, 2022, 49(5): 279-286.
[11] LI Peng, YI Xiu-wen, QI De-kang, DUAN Zhe-wen, LI Tian-rui. Heating Strategy Optimization Method Based on Deep Learning [J]. Computer Science, 2022, 49(4): 263-268.
[12] OUYANG Zhuo, ZHOU Si-yuan, LYU Yong, TAN Guo-ping, ZHANG Yue, XIANG Liang-liang. DRL-based Vehicle Control Strategy for Signal-free Intersections [J]. Computer Science, 2022, 49(3): 46-51.
[13] ZHANG Hai-bo, ZHANG Yi-feng, LIU Kai-jian. Task Offloading,Migration and Caching Strategy in Internet of Vehicles Based on NOMA-MEC [J]. Computer Science, 2022, 49(2): 304-311.
[14] PAN Yan-na, FENG Xiang, YU Hui-qun. Competitive-Cooperative Coevolution for Large Scale Optimization with Computation Resource Allocation Pool [J]. Computer Science, 2022, 49(2): 182-190.
[15] DAI Shan-shan, LIU Quan. Action Constrained Deep Reinforcement Learning Based Safe Automatic Driving Method [J]. Computer Science, 2021, 48(9): 235-243.
Full text



No Suggested Reading articles found!