Computer Science ›› 2022, Vol. 49 ›› Issue (6): 3-11.doi: 10.11896/jsjkx.220100249

• Smart IoT Technologies and Applications Empowered by 6G • Previous Articles     Next Articles

PPO Based Task Offloading Scheme in Aerial Reconfigurable Intelligent Surface-assisted Edge Computing

XIE Wan-cheng1, LI Bin1,2, DAI Yue-yue3   

  1. 1 School of Computer and Software,Nanjing University of Information Science and Technology,Nanjing 210044,China
    2 Key Lab of Broadband Wireless Communication and Sensor Network Technology (Nanjing University of Posts and Telecommunications), Ministry of Education,Nanjing 210003,China
    3 Research Center of 6G Mobile Communications and School of Cyber Science and Engineering,Huazhong University of Science and Technology,Wuhan 430074,China
  • Received:2022-01-26 Revised:2022-03-10 Online:2022-06-15 Published:2022-06-08
  • About author:XIE Wan-cheng,born in 2001,postgra-duate,is a student member of China Computer Federation.His main research interests include IoT and edge intelligence.
    LI Bin,born in 1987,Ph.D,associate professor,is a member of China Computer Federation.His main research interests include IoT and edge intelligence.
  • Supported by:
    National Natural Science Foundation of China(62101277),National Natural Science Foundation of Jiangsu Pro-vince(BK20200822),Natural Science Foundation of Jiangsu Higher Education Institutions of China(20KJB510036),Open Research Fund of Key Lab of Broadband Wireless Communication and Sensor Network Technology(Nanjing University of Posts and Telecommunications) and Ministry of Education (JZNY202103).

Abstract: In order to compensate the performance loss caused by obstacle blocking in mobile edge computing (MEC) system in 6G-enabled “intelligent Internet of Things”,this paper proposes a partial task offloading scheme supported by aerial reconfigurable intelligent surface (RIS).Firstly,we investigate the joint design of the RIS phase shift vector,the proportion of offloading task,time slot allocation,the transmit power of users and the position of UAV,formulating a non-convex problem for minimization of the total energy consumption of users.Then,the original non-convex problem is decomposed into four subproblems,and the proximal policy optimization (PPO) method in deep reinforcement learning (DRL) is utilized to provide time slot allocation.The alternative optimization (AO) is leveraged to decouple the original problem into four subproblems,including the RIS phase shift design,the convex optimization of transmit power and offloading task amount,and the UAV altitude optimization.Simulation results show that the proposed PPO model can be trained quickly,the total energy consumption of users can be reduced by about 23% and 5.3%,compared with the fully-offload strategy and fixed-UAV-height strategy,respectively.

Key words: Deep reinforcement learning, Mobile edge computing, Reconfigurable intelligent surface, Task offloading, Unmanned aerial vehicle

CLC Number: 

  • TN929.5
[1] WU D P,ZHANG P N,WANG R Y.Smart Internet of things Aided by “Terminal-Edge-Cloud” Cooperation[J].Chinese Journal on Internet of Things,2018,2(3):21-28.
[2] LI Z J,ZHANG X L.Resource Allocation and Offloading Decision of Edge Computing for Reducing Core Network Congestion[J].Computer Science,2021,48(3):281-288.
[3] XU Y,ZHANG T,YANG D,et al.UAV-Assisted Relaying and MEC Networks:Resource Allocation and 3D Deployment[C]//2021 IEEE International Conference on Communications Workshops (ICC Workshops).2021:1-6.
[4] ZHANG T,XU Y,LOO J,et al.Joint Computation and Communication Design for UAV-Assisted Mobile Edge Computing in IoT[J].IEEE Transactions on Industrial Informatics,2020,16(8):5505-5516.
[5] DIAO X B,YANG W D,YANG L X,et al.UAV-Relaying-Assisted Multi-Access Edge Computing With Multi-Antenna Base Station:Offloading and Scheduling Optimization[J].IEEE Transactions on Vehicular Technology,2021,70(9):9495-9509.
[6] WANG J,NA Z,LIU X.Collaborative Design of Multi-UAVTrajectory and Resource Scheduling for 6G-Enabled Internet of Things[J].IEEE Internet of Things Journal,2021,8(20):15096-15106.
[7] TIAN H,NI W L,WANG W,et al.Data-Importance-Aware Resource Allocation in IRS-Aided Edge Intelligent System[J].Journal of Beijing University of Posts and Telecommunications,2020,43(6):51-58.
[8] LI Z Y,CHEN M,YANG Z H,et al.Energy Efficient Reconfi-gurable Intelligent Surface Enabled Mobile Edge Computing Networks With NOMA[J].IEEE Transactions on Cognitive Communications and Networking,2021,7(2):427-440.
[9] LI A C,LIU Y,LI M,et al.Joint Scheduling Design in Wireless Powered MEC IoT Networks Aided by Reconfigurable Intelligent Surface[C]//2021 IEEE/CIC International Conference on Communications in China (ICCC Workshops).Xiamen,China:IEEE,2021:159-164.
[10] HUANG S F,WANG S,WANG R,et al.Reconfigurable Intelligent Surface Assisted Mobile Edge Computing With Heterogeneous Learning Tasks[J].IEEE Transactions on Cognitive Communications and Networking,2021,7(2):369-382.
[11] YANG Z H,HUANG C W,SHI J F,et al.Optimal Control for Full-Duplex Communications with Reconfigurable Intelligent Surface[C]//ICC 2021-IEEE International Conference on Communications.Montreal,QC,Canada:IEEE,2021:1-6.
[12] LIU X,LIU Y W,CHEN Y.Machine Learning EmpoweredTrajectory and Passive Beamforming Design in UAV-RIS Wireless Networks[J].IEEE Journal on Selected Areas in Communications,2021,39(7):2042-2055.
[13] MEI H B,YANG K,SHEN J,et al.Joint Trajectory-Task-Cache Optimization With Phase-Shift Design of RIS-Assisted UAV for MEC[J].IEEE Wireless Communications Letters,2021,10(7):1586-1590.
[14] LONG H,CHEN M,YANG Z H,et al.Joint Trajectory andPassive Beamforming Design for Secure UAV Networks with RIS[C]//2020 IEEE Globecom Workshops.Taipei,Taiwan:IEEE,2020:1-6.
[15] MURSIA P,DEVOTI F,SCIANCALEPORE V,et al.RISe ofFlight:RIS-Empowered UAV Communications for Robust and Reliable Air-to-Ground Networks[J].IEEE Open Journal of the Communications Society,2021,2:1616-1629.
[16] SAMIR M,ELHATTAB M,ASSI C,et al.Optimizing Age of Information Through Aerial Reconfigurable Intelligent Surfaces:A Deep Reinforcement Learning Approach[J].IEEE Transactions on Vehicular Technology,2021,70(4):3978-3983.
[17] ZHAN C,HU H,SUI X F,et al.Completion Time and Energy Optimization in the UAV-Enabled Mobile-Edge Computing System[J].IEEE Internet of Things Journal,2020,7(8):7808-7822.
[18] LI A,DAI L B,YU L S,et al.Resource Allocation for Un-manned Aerial Vehicle-assisted Mobile Edge Computing to Mini-mize Weighted Energy Consumption[J].Journal of Electronics &Information Technology,2021:1-8.
[19] WANG F,XU J,CUI S.Optimal Energy Allocation and TaskOffloading Policy for Wireless Powered Mobile Edge Computing Systems[J].IEEE Transactions on Wireless Communications,2020,19(4):2443-2459.
[20] WANG F,XU J,WANG X,et al.Joint Offloading and Computing Optimization in Wireless Powered Mobile-Edge Computing Systems[J].IEEE Transactions on Wireless Communications,2018,17(3):1784-1797.
[21] LIANG J B,ZHANG H H,JIANG C,et al.Research Progress of Task Offloading Based on Deep Reinforcement Learning in Mobile Edge Computing[J].Computer Science,2021,48(7):316-323.
[22] ENGSTROM L,ILYAS A,SANTURKAR S,et al.Implementa-tion Matters in Deep Policy Gradients:A Case Study on PPO and TRPO[C]//2020 International Conference on Learning Representations.2019:1-14.
[23] LIU C H,DAI Z,ZHAO Y,et al.Distributed and Energy-Efficient Mobile Crowdsensing with Charging Stations by Deep Reinforcement Learning[J].IEEE Transactions on Mobile Computing,2021,20(1):130-146.
[24] BOYD S,VANDENBERGHE L.Convex Optimization[M].Cambridge:Cambridge University Press,2004.
[1] JIAN Qi-rui, CHEN Ze-mao, WU Xiao-kang. Authentication and Key Agreement Protocol for UAV Communication [J]. Computer Science, 2022, 49(8): 306-313.
[2] YU Bin, LI Xue-hua, PAN Chun-yu, LI Na. Edge-Cloud Collaborative Resource Allocation Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2022, 49(7): 248-253.
[3] LI Meng-fei, MAO Ying-chi, TU Zi-jian, WANG Xuan, XU Shu-fang. Server-reliability Task Offloading Strategy Based on Deep Deterministic Policy Gradient [J]. Computer Science, 2022, 49(7): 271-279.
[4] FANG Tao, YANG Yang, CHEN Jia-xin. Optimization of Offloading Decisions in D2D-assisted MEC Networks [J]. Computer Science, 2022, 49(6A): 601-605.
[5] LIU Zhang-hui, ZHENG Hong-qiang, ZHANG Jian-shan, CHEN Zhe-yi. Computation Offloading and Deployment Optimization in Multi-UAV-Enabled Mobile Edge Computing Systems [J]. Computer Science, 2022, 49(6A): 619-627.
[6] DONG Dan-dan, SONG Kang. Performance Analysis on Reconfigurable Intelligent Surface Aided Two-way Internet of Things Communication System [J]. Computer Science, 2022, 49(6): 19-24.
[7] HONG Zhi-li, LAI Jun, CAO Lei, CHEN Xi-liang, XU Zhi-xiong. Study on Intelligent Recommendation Method of Dueling Network Reinforcement Learning Based on Regret Exploration [J]. Computer Science, 2022, 49(6): 149-157.
[8] LI Peng, YI Xiu-wen, QI De-kang, DUAN Zhe-wen, LI Tian-rui. Heating Strategy Optimization Method Based on Deep Learning [J]. Computer Science, 2022, 49(4): 263-268.
[9] SHI Dian-xi, LIU Cong, SHE Fu-jiang, ZHANG Yong-jun. Cooperation Localization Method Based on Location Confidence of Multi-UAV in GPS-deniedEnvironment [J]. Computer Science, 2022, 49(4): 302-311.
[10] OUYANG Zhuo, ZHOU Si-yuan, LYU Yong, TAN Guo-ping, ZHANG Yue, XIANG Liang-liang. DRL-based Vehicle Control Strategy for Signal-free Intersections [J]. Computer Science, 2022, 49(3): 46-51.
[11] ZHANG Hai-bo, ZHANG Yi-feng, LIU Kai-jian. Task Offloading,Migration and Caching Strategy in Internet of Vehicles Based on NOMA-MEC [J]. Computer Science, 2022, 49(2): 304-311.
[12] DAI Shan-shan, LIU Quan. Action Constrained Deep Reinforcement Learning Based Safe Automatic Driving Method [J]. Computer Science, 2021, 48(9): 235-243.
[13] CHENG Zhao-wei, SHEN Hang, WANG Yue, WANG Min, BAI Guang-wei. Deep Reinforcement Learning Based UAV Assisted SVC Video Multicast [J]. Computer Science, 2021, 48(9): 271-277.
[14] XU Hao, LIU Yue-lei. UAV Sound Recognition Algorithm Based on Deep Learning [J]. Computer Science, 2021, 48(7): 225-232.
[15] LIANG Jun-bin, ZHANG Hai-han, JIANG Chan, WANG Tian-shu. Research Progress of Task Offloading Based on Deep Reinforcement Learning in Mobile Edge Computing [J]. Computer Science, 2021, 48(7): 316-323.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!