计算机科学

计算机科学 ›› 2022, Vol. 49 ›› Issue (7): 271-279.doi: 10.11896/jsjkx.210600040

• 计算机网络 • 上一篇    下一篇

基于深度确定性策略梯度的服务器可靠性任务卸载策略

李梦菲1, 毛莺池1,2, 屠子健1, 王瑄1, 徐淑芳1,2   

  1. 1 河海大学计算机与信息学院 南京210098
    2 水利部水利大数据技术重点实验室 南京210098
  • 收稿日期:2021-06-04 修回日期:2021-12-07 出版日期:2022-07-15 发布日期:2022-07-12
  • 通讯作者: 毛莺池(yingchimao@hhu.edu.cn)
  • 作者简介:(2028004563@qq.com)
  • 基金资助:
    江苏省重点研发项目(BE2020729);姑苏创新领军人才专项(ZXL2020210);2020年昆山祖冲之攻关计划项目;中国华能集团关键技术项目(HNKJ19-H12,HNKJ20-H64)

Server-reliability Task Offloading Strategy Based on Deep Deterministic Policy Gradient

LI Meng-fei1, MAO Ying-chi1,2, TU Zi-jian1, WANG Xuan1, XU Shu-fang1,2   

  1. 1 College of Computer and Information,Hohai University,Nanjing 210098,China
    2 Key Laboratory of Water Big Data Technology of Ministry of Water Resources,Nanjing 210098,China
  • Received:2021-06-04 Revised:2021-12-07 Online:2022-07-15 Published:2022-07-12
  • About author:LI Meng-fei,born in 1998,postgra-duate,is a member of China Computer Federation.Her main research interests include distributed computing,IoT and edge intelligence computing.
    MAO Ying-chi,born in 1976,Ph.D,professor,is a senior member of China Computer Federation.Her main research interests include distributed computing and edge computing.
  • Supported by:
    Key Research and Development Project of Jiangsu Province,China(BE2020729),Gusu Innovation Leading Ta-lentsSpecial Project(ZXL2020210),2020 Kunshan Zu Chongzhi Key Tack Project and Key Technology Project of China Huaneng Group(HNKJ19-H12, HNKJ20-H64).

PDF (PC)

490

摘要： 随着智能移动设备的普及,新一代移动应用如人脸识别、虚拟现实等逐渐兴起,但移动设备因计算能力和电池容量有限,无法支持这类计算需求高且延迟敏感的应用。因此,移动边缘计算被提出以解决该问题。然而,在MEC环境中,边缘服务器可靠性较低,若发生设备故障会导致已有的卸载决策失效,使得应用程序响应时间增加,用户体验感降低。针对边缘服务器可能发生故障的问题,同时考虑到深度确定性策略梯度算法通过网络拟合策略函数,可以较好地应对高维动作空间的问题,提出了基于深度确定性策略梯度的服务器可靠性任务卸载策略。首先,通过复制子任务进行二次卸载的方式来降低应用执行的失败率;其次,将服务器可靠性约束下最小化应用时延的任务卸载和资源分配问题建模为马尔可夫决策过程;最后,利用基于深度确定性策略梯度的算法来求解任务卸载策略。仿真结果表明,SRTO-DDPG策略能有效地与环境交互并获得最优卸载决策,其性能优于本地执行策略,且相比基于DDPG的单卸载地点任务卸载策略,所提策略在可靠性约束下能实现低约26.16%的总延迟,能够更好地适应多服务器场景中边缘服务器的可靠性问题。

关键词: 任务卸载, 深度强化学习, 依赖性任务, 移动边缘计算, 资源分配

Abstract: With the popularization of smart mobile devices,a new generation of mobile applications such as face recognition and virtual reality have gradually emerged.The limited computing power and battery capacity of mobile devices cannot support applications with high computing requirements and latency-sensitive applications.Therefore,mobile edge computing(MEC) is proposed to solve this problem.However,in the MEC environment,the reliability of the edge server is low,and the possible equipment failure will lead to the existing offloading decision failure,which increases the application response time and reduces the user experience.In view of the possible failure of edge servers,and considering that the deep deterministic policy gradient(DDPG) algorithm can better deal with the problem of high-dimensional action space through the network fitting strategy function,this paper proposes a server-reliability task offloading based on deep deterministic policy gradient(SRTO-DDPG).The main work is as follows.Firstly,the failure rate of application execution is reduced by duplicating subtasks for secondary offload.Secondly,the task offloading and resource allocation problems with server reliability constraints to minimize application delay are modeled as Markov decision process(MDP).Finally,an algorithm based on DDPG is used to solve the problem.Simulation results show that the SRTO-DDPG strategy can effectively interact with the environment to obtain the optimal offloading decision,and its perfor-mance is better than the local execution strategy(LE).Compared with the single location task offloading based on deep determi-nistic policy gradient(SLTO-DDPG),this strategy can achieve a low total delay of about 26.16% under reliability constraints,and can better adapt to the reliability problems of edge servers in multi-server scenarios.

Key words: Deep reinforcement learning, Dependent tasks, Mobile edge computing, Resource allocation, Task offloading

中图分类号: 

  • TP399
[1]AL-JANABI S,AL-SHOURBAJI I,SHOJAFAR M,et al.Mobile Cloud Computing:Challenges and Future Research Directions[C]//International Conference on the Developments on Systems Engineering.IEEE,2017:62-67.
[2]RANADHEERA S,MAGHSUDI S,HOSSAIN E.Mobile Edge Computation Offloading Using Game Theory and Reinforcement Learning[J].arXiv:1711.09012,2017.
[3]XU C B,LIU Y,LIU Y X,et al.MEC server selection scheme based on multiple indicators [J].Journal of Chongqing Univer-sity of Posts and Telecommunications(Natural Science Edition),2020,32(3):329-335.
[4]LIANG Y C,CAO B.Artificial neural network methods for task offloading in VANET cloud [J].Journal of Chongqing Univer-sity of Posts and Telecommunications(Natural Science Edition),2020,32(3):336-344.
[5]LI J,ZHANG Y P,PANG L,et al.Joint Resource Allocation and Task Scheduling in Mobile Edge Computing[J].Journal of Chongqing University of Technology(Natural Science),2020,34(11):156-163.
[6]CHEN L.Multicast resource allocation algorithm based on la-yered coding in sparse code multiple access systems[J].Journal of Chongqing University of Posts and Telecommunications(Na-tural Science Edition),2020,32(6):917-924.
[7]MAO Y Y,ZHANG J,BEN K,et al.Dynamic Computation Offloading for Mobile-Edge Computing with Energy Harvesting Devices[J].IEEE Journal on Selected Areas in Communications,2016,34(12):3590-3605.
[8]CEHN X,JIAO L,LI W,et al.Efficient Multi-User Computation Offloading for Mobile-Edge Cloud Computing[J].IEEE/ACM Transactions on Networking,2016,24(5):2795-2808.
[9]CAO H,CAI J.Distributed Multiuser Computation Offloading for Cloudlet-Based Mobile Cloud Computing:A Game-Theoretic Machine Learning Approach[J].IEEE Transactions on Vehicular Technology,2018:752-764.
[10]HUANG L,BI S,ZHANG Y J A.Deep Reinforcement Learning for Online Computation Offloading in Wireless Powered Mobile-Edge Computing Networks[J].IEEE Transactions on Mobile Computing,2019,19(11):2581-2593.
[11]GUO H,LIU J,ZHANG J.Computation Offloading for Multi-Access Mobile Edge Computing in Ultra-Dense Networks[J].IEEE Communications Magazine,2018,56(8):14-19.
[12]ZHU A,GUO S,MA M,et al.Computation Offloading forWorkflow in Mobile Edge Computing Based on Deep Q-Learning[C]//Wireless and Optical Communications Conference.IEEE,2019:1-5.
[13]CHEN X,ZHANG H,WU C,et al.Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning[J].IEEE Internet of Things Journal,2019,6(3):4005-4018.
[14]HUANG B,LI Y,LI Z,et al.Security and Cost-Aware Computation Offloading via Deep Reinforcement Learning in Mobile Edge Computing[J].Wireless Communications and Mobile Computing,2019,2019:1-20.
[15]SHARMA Y,JAVADI B,SI W,et al.Reliability and Energy Efficiency in Cloud Computing Systems:Survey and Taxonomy[J].Journal of Network and Computer Applications,2016,74:66-85.
[16]ALSENANI Y,CROSBY G,VELASCO T.SaRa:A Stochastic Model to Estimate Reliability of Edge Resources in Volunteer Cloud[C]//2018 IEEE International Conference on Edge Computing(EDGE).IEEE,2018:121-124.
[17]LORENZO B,GARCIA-ROIS J,LI X,et al.A Robust Dynamic Edge Network Architecture for the Internet-of-Things[J].IEEE Network,2018,32(1):8-15.
[18]AL-HABOB A A,IBRAHIM A,DOBRE O A,et al.Collision-Free Sequential Task Offloading for Mobile Edge Computing[J].IEEE Communications Letters,2020,24(1):71-75.
[19]LIU J,ZHANG Q.Offloading Schemes in Mobile Edge Computing for Ultra-Reliable Low Latency Communications[J].IEEE Access,2018,16:12825-12837.
[20]HUANG L,FENG X,QIAN L,et al.Deep ReinforcementLearning-Based Task Offloading and Resource Allocation for Mobile Edge Computing[C]//International Conference on Machine Learning and Intelligent Communications.2018:33-42.
[21]LI J,GAO H,LV T J,et al.Deep Reinforcement Learning based Computation Offloading and Resource Allocation for MEC[C]//Wireless Communications and Networking Conference.IEEE,2018:1-6.
[22]SHU C,ZHAO Z,HAN Y,et al.Dependency-Aware and Latency-Optimal Computation Offloading for Multi-User Edge Computing Networks[C]//2019 16th Annual IEEE International Conference on Sensing,Communication,and Networking.IEEE,2019:1-9.
[23]LIU J,ZHANG Q.Code-Partitioning Offloading Schemes inMobile Edge Computing for Augmented Reality[J].IEEE Access,2019,7:11222-11236.
[24]CHEN Z,WANG X.Decentralized Computation Offloading for Multi-User Mobile Edge Computing:A Deep Reinforcement Learning Approach[J].Journal on Wireless Communications and Networking,2020,188:1-21.
[1] 熊丽琴, 曹雷, 赖俊, 陈希亮.
基于值分解的多智能体深度强化学习综述
Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization
计算机科学, 2022, 49(9): 172-182. https://doi.org/10.11896/jsjkx.210800112
[2] 于滨, 李学华, 潘春雨, 李娜.
基于深度强化学习的边云协同资源分配算法
Edge-Cloud Collaborative Resource Allocation Algorithm Based on Deep Reinforcement Learning
计算机科学, 2022, 49(7): 248-253. https://doi.org/10.11896/jsjkx.210400219
[3] 唐枫, 冯翔, 虞慧群.
基于自适应知识迁移与资源分配的多任务协同优化算法
Multi-task Cooperative Optimization Algorithm Based on Adaptive Knowledge Transfer andResource Allocation
计算机科学, 2022, 49(7): 254-262. https://doi.org/10.11896/jsjkx.210600184
[4] 方韬, 杨旸, 陈佳馨.
D2D辅助移动边缘计算下的卸载策略优化
Optimization of Offloading Decisions in D2D-assisted MEC Networks
计算机科学, 2022, 49(6A): 601-605. https://doi.org/10.11896/jsjkx.210200114
[5] 刘漳辉, 郑鸿强, 张建山, 陈哲毅.
多无人机使能移动边缘计算系统中的计算卸载与部署优化
Computation Offloading and Deployment Optimization in Multi-UAV-Enabled Mobile Edge Computing Systems
计算机科学, 2022, 49(6A): 619-627. https://doi.org/10.11896/jsjkx.210600165
[6] 谢万城, 李斌, 代玥玥.
空中智能反射面辅助边缘计算中基于PPO的任务卸载方案
PPO Based Task Offloading Scheme in Aerial Reconfigurable Intelligent Surface-assisted Edge Computing
计算机科学, 2022, 49(6): 3-11. https://doi.org/10.11896/jsjkx.220100249
[7] 周天清, 岳亚莉.
超密集物联网络中多任务多步计算卸载算法研究
Multi-Task and Multi-Step Computation Offloading in Ultra-dense IoT Networks
计算机科学, 2022, 49(6): 12-18. https://doi.org/10.11896/jsjkx.211200147
[8] 邱旭, 卞浩卜, 吴铭骁, 朱晓荣.
基于5G毫米波通信的高速公路车联网任务卸载算法研究
Study on Task Offloading Algorithm for Internet of Vehicles on Highway Based on 5G MillimeterWave Communication
计算机科学, 2022, 49(6): 25-31. https://doi.org/10.11896/jsjkx.211100198
[9] 胥昊, 曹桂均, 闫璐, 李科, 王振宏.
面向铁路集装箱的高可靠低时延无线资源分配算法
Wireless Resource Allocation Algorithm with High Reliability and Low Delay for Railway Container
计算机科学, 2022, 49(6): 39-43. https://doi.org/10.11896/jsjkx.211200143
[10] 洪志理, 赖俊, 曹雷, 陈希亮, 徐志雄.
基于遗憾探索的竞争网络强化学习智能推荐方法研究
Study on Intelligent Recommendation Method of Dueling Network Reinforcement Learning Based on Regret Exploration
计算机科学, 2022, 49(6): 149-157. https://doi.org/10.11896/jsjkx.210600226
[11] 沈家芳, 钱丽萍, 杨超.
面向集能型中继窄带物联网的非正交多址接入和多维网络资源优化
Non-orthogonal Multiple Access and Multi-dimension Resource Optimization in EH Relay NB-IoT Networks
计算机科学, 2022, 49(5): 279-286. https://doi.org/10.11896/jsjkx.210400239
[12] 李鹏, 易修文, 齐德康, 段哲文, 李天瑞.
一种基于深度学习的供热策略优化方法
Heating Strategy Optimization Method Based on Deep Learning
计算机科学, 2022, 49(4): 263-268. https://doi.org/10.11896/jsjkx.210300155
[13] 彭冬阳, 王睿, 胡谷雨, 祖家琛, 王田丰.
视频缓存策略中QoE和能量效率的公平联合优化
Fair Joint Optimization of QoE and Energy Efficiency in Caching Strategy for Videos
计算机科学, 2022, 49(4): 312-320. https://doi.org/10.11896/jsjkx.210800027
[14] 欧阳卓, 周思源, 吕勇, 谭国平, 张悦, 项亮亮.
基于深度强化学习的无信号灯交叉路口车辆控制
DRL-based Vehicle Control Strategy for Signal-free Intersections
计算机科学, 2022, 49(3): 46-51. https://doi.org/10.11896/jsjkx.210700010
[15] 张海波, 张益峰, 刘开健.
基于NOMA-MEC的车联网任务卸载、迁移与缓存策略
Task Offloading,Migration and Caching Strategy in Internet of Vehicles Based on NOMA-MEC
计算机科学, 2022, 49(2): 304-311. https://doi.org/10.11896/jsjkx.210100157
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发