车联网中基于联邦深度强化学习的任务卸载算法

doi:10.11896/jsjkx.220800243

Abstract

Abstract: With the rapid development of the service system of Internet of Vehicles applications,vehicles with limited computational resources have difficulty in handling these computation-intensive and latency-sensitive applications.As a key technique in mobile edge computing,task offloading can address the challenge.Specially,a task offloading algorithm based on federated deep reinforcement learning(TOFDRL) is proposed for dynamic multi-vehicle multi-road-side-unit(multi-RSU) task offloading environment in Internet of Vehicles.Each vehicle is considered as an agent,and a federated learning framework is used to train each agent.Each agent makes distributed decisions,aiming to minimize the average system response time.Evaluation experiments are set up to compare and analyze the performance of the proposed algorithm under a variety of dynamically changing scenarios.Si-mulation results show that the average response time of system solved by the proposed algorithm is shorter than that of the rule-based algorithm and the multi-agent deep reinforcement learning algorithm,close to the ideal scheme,and its solution time is much shorter than the ideal solution.Experimental results demonstrate that the proposed algorithm is able to solve an average system response time which is close to the ideal solution within an acceptable execution time.

Key words: Mobile edge computing, Task offloading, Internet of Vehicles, Deep reinforcement learning, Federated learning

CLC Number:

TP338

LIN Xinyu, YAO Zewei, HU Shengxi, CHEN Zheyi, CHEN Xing. Task Offloading Algorithm Based on Federated Deep Reinforcement Learning for Internet of Vehicles[J].Computer Science, 2023, 50(9): 347-356.

References

[1]FENG J,LIU Z,WU C,et al.Mobileedge computing for the internet of vehicles:offloading framework and job scheduling[J].IEEE Vehicular Technology Magazine,2019,14(1):28-36.
[2]ZHAN W,LUO C,WANG J,et al.Deep-reinforce ment-lear-ning-based offloading scheduling for vehicular edge computing[J].IEEE Internet of Things Journal,2020,7(6):5449-5465.
[3]MACH P,BECVAR Z.Mobile edge computing:a survey on architecture and computation offloading[J].IEEE Communications Surveys & Tutorials,2017,19(3):1628-1656.
[4]DINH H T,LEE C,NIYATO D,et al.A survey of mobile cloud computing:architecture,applications,and approaches[J].Wireless Communications and Mobile Computing,2013,13(18):1587-1611.
[5]RAZA S,WANG S,AHMED M,et al.Task offloading and resource allocation for iov using 5g nr-v2x communication[J].IEEE Internet of Things Journal,2022,9(13):10397-10410.
[6]LIN Y,ZHANG Y,LI J,et al.Popularity-aware online task offloading for heterogeneous vehicular edge computing using contextual clustering of bandits[J].IEEE Internet of Things Journal,2022,9(7):5422-5433.
[7]DAI P,HU K,WU X,et al.A probabilistic approach for coope-rative computation offloading in mec-assisted vehicular networks[J].IEEE Transactions on Intelligent Transportation Systems,2022,23(2):899-911.
[8]CHEN M,HAO Y.Task offloading for mobile edge computing in software defined ultra-dense network[J].IEEE Journal on Selected Areas in Communications,2018,36(3):587-597.
[9]LIN B,GUO W,CHEN G.Scheduling strategy for scienceworkflow with deadline constraint on multi-cloud[J].Journal on Communications,2018,39(1):56-69.
[10]HOU X,REN Z,WANG J,et al.Reliablecomputation offloading for edge-computing-enabled software-defined iov[J].IEEE Internet of Things Journal,2020,7(8):7097-7111.
[11]LIU Z,ZHENG H,ZHANG J,et al.Computation offloading and deployment optimization in multi-uav-enabled mobile edge computing systems[J].Computer Science,2022,49(S1):619-627.
[12]YAO Z,LIN J,HU J,et al.PSO-GA based approach to multi-edge load balancing[J].Computer Science,2021,48(S2):456-463.
[13]ALASMARI K R,II R C G,ALAM M.Mobile edge offloading using markov decision processes[C]//Edge Computing－EDGE 2018.Seattle,WA,USA,2018.
[14]SUTTON R S,BARTO A G.Reinforcement learning,secondedition:an introduction[M].Cambridge:The MIT Press,2018.
[15]ZHANG K,ZHU Y,LENG S,et al.Deep learning empowered task offloading for mobile edge computing in urban informatics[J].IEEE Internet of Things Journal,2019,6(5):7635-7647.
[16]MIN M,XIAO L,CHEN Y,et al.Learning-based computation offloading for iot devices with energy harvesting[J].IEEE Transactions on Vehicular Technology,2019,68(2):1930-1941.
[17]HUANG L,BI S,ZHANG Y J A.Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks[J].IEEE Transactions on Mobile Computing,2020,19(11):2581-2593.
[18]LUO Q,LI C,LUAN T H,et al.Collaborative data scheduling for vehicular edge computing via deep reinforcement learning[J].IEEE Internet of Things Journal,2020,7(10):9637-9650.
[19]KE H,WANG J,DENG L,et al.Deep reinforcement learning-based adaptive computation offloading for mec in heterogeneous vehicular networks[J].IEEE Transactions on Vehicular Technology,2020,69(7):7916-7929.
[20]YU S,CHEN X,ZHOU Z,et al.When deep reinforcementlearning meets federated learning:intelligent multitimescale resource management for multiaccess edge computing in 5g ultradense network[J].IEEE Internet of Things Journal,2021,8(4):2238-2251.
[21]TANG M,WONG V W.Deep reinforcement learning for task offloading in mobile edge computing systems[J].IEEE Transactions on Mobile Computing,2022,21(6):1985-1997.
[22]HUANG X,LENG S,MAHARJAN S,et al.Multi-agent deep reinforcement learning for computation offloading and interfe-rence coordination in small cell networks[J].IEEE Transactions on Vehicular Technology,2021,70(9):9282-9293.
[23]CHEN X,HU J,CHEN Z,et al.A reinforcement learning-empowered feedback control system for industrial internet of things[J].IEEE Transactions on Industrial Informatics,2022,18(4):2724-2733.
[24]MNIH V,KAVUKCUOGLU K,SILVER D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529-533.
[25]JIA M,LIANG W,XU Z,et al.Qos-aware cloudlet load balancing in wireless metropolitan area networks[J].IEEE Transactions on Cloud Computing,2020,8(2):623-634.
[26]KLEINROCK L.Queueing systems,volume I:theory[M].Hoboken:Wiley,1975:101-103.
[27]LIANG L,YE H,LI G Y.Spectrum sharing in vehicular networks based on multi-agent reinforcement learning[J].IEEE Journal on Selected Areas in Communications,2019,37(10):2282-2292.
[28]WATKINS C J C H.Learning from delayed rewards[D].Cambridge:University of Cambridge,1989.
[29]HINTON G.Overview of mini-batch gradient descent[EB/OL].http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.
[30]WU Y,GUO K,HUANG J,et al.Secrecy-based energy-efficient data offloading via dual connectivity over unlicensed spectrums[J].IEEE Journal on Selected Areas in Communications,2016,34(12):3252-3270.
[31]NAIR V,HINTON G E.Rectified linear units improve restric-ted boltzmann machines[C]//Proceedings of the 27th International Conference on Machine Learning(ICML-10).Haifa,Is-rael,2010:807-814.

Related Articles 15

[1]	LIU Xingguang, ZHOU Li, ZHANG Xiaoying, CHEN Haitao, ZHAO Haitao, WEI Jibo. Edge Intelligent Sensing Based UAV Space Trajectory Planning Method [J]. Computer Science, 2023, 50(9): 311-317.
[2]	ZHAO Yuhao, CHEN Siguang, SU Jian. Privacy-enhanced Federated Learning Algorithm Against Inference Attack [J]. Computer Science, 2023, 50(9): 62-67.
[3]	JIN Tiancheng, DOU Liang, ZHANG Wei, XIAO Chunyun, LIU Feng, ZHOU Aimin. OJ Exercise Recommendation Model Based on Deep Reinforcement Learning and Program Analysis [J]. Computer Science, 2023, 50(8): 58-67.
[4]	XIONG Liqin, CAO Lei, CHEN Xiliang, LAI Jun. Value Factorization Method Based on State Estimation [J]. Computer Science, 2023, 50(8): 202-208.
[5]	ZHANG Naixin, CHEN Xiaorui, LI An, YANG Leyao, WU Huaming. Edge Offloading Framework for D2D-MEC Networks Based on Deep Reinforcement Learningand Wireless Charging Technology [J]. Computer Science, 2023, 50(8): 233-242.
[6]	LI Rongchang, ZHENG Haibin, ZHAO Wenhong, CHEN Jinyin. Data Reconstruction Attack for Vertical Graph Federated Learning [J]. Computer Science, 2023, 50(7): 332-338.
[7]	CHEN Xuzhan, LIN Bing, CHEN Xing. Stackelberg Model Based Distributed Pricing and Computation Offloading in Mobile Edge Computing [J]. Computer Science, 2023, 50(7): 278-285.
[8]	ZHANG Lianfu, TAN Zuowen. Robust Federated Learning Algorithm Based on Adaptive Weighting [J]. Computer Science, 2023, 50(6A): 230200188-9.
[9]	LEI Xuemei, LIU Li, WANG Qian. MEC Offloading Model Based on Linear Programming Relaxation [J]. Computer Science, 2023, 50(6A): 211200229-5.
[10]	CHEN Che, ZHENG Yifeng, YANG Jingmin, YANG Liwei, ZHANG Wenjie. Dynamic Energy Optimization Strategy Based on Relay Selection and Queue Stability [J]. Computer Science, 2023, 50(6A): 220100082-8.
[11]	WANG Hanmo, ZHENG Shijie, XU Ruonan, GUO Bin, WU Lei. Self Reconfiguration Algorithm of Modular Robot Based on Swarm Agent Deep Reinforcement Learning [J]. Computer Science, 2023, 50(6): 266-273.
[12]	ZHANG Qiyang, CHEN Xiliang, CAO Lei, LAI Jun, SHENG Lei. Survey on Knowledge Transfer Method in Deep Reinforcement Learning [J]. Computer Science, 2023, 50(5): 201-216.
[13]	GAO Lixue, CHEN Xin, YIN Bo. Task Offloading Strategy Based on Game Theory in 6G Overlapping Area [J]. Computer Science, 2023, 50(5): 302-312.
[14]	YU Ze, NING Nianwen, ZHENG Yanliu, LYU Yining, LIU Fuqiang, ZHOU Yi. Review of Intelligent Traffic Signal Control Strategies Driven by Deep Reinforcement Learning [J]. Computer Science, 2023, 50(4): 159-171.
[15]	PEI Cui, FAN Guisheng, YU Huiqun, YUE Yiming. Auction-based Edge Cloud Deadline-aware Task Offloading Strategy [J]. Computer Science, 2023, 50(4): 241-248.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Task Offloading Algorithm Based on Federated Deep Reinforcement Learning for Internet of Vehicles

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 15

Metrics

Comments

Recommended 0