Computer Science ›› 2023, Vol. 50 ›› Issue (1): 194-204.doi: 10.11896/jsjkx.220500241

• Artificial Intelligence • Previous Articles     Next Articles

Bi-level Path Planning Method for Unmanned Vehicle Based on Deep Reinforcement Learning

HUANG Yuzhou, WANG Lisong, QIN Xiaolin   

  1. College of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 211106,China
  • Received:2022-05-25 Revised:2022-09-12 Online:2023-01-15 Published:2023-01-09
  • About author:HUANG Yuzhou,born in 1998,postgraduate.His main research interests include robot intelligence,etc.
    QIN Xiaolin,born in 1953,Ph.D,is a senior member of China Computer Fe-deration.His main research interests include data management and security in distributed environment,etc.
  • Supported by:
    National Natural Science Foundation of China(61728204).

Abstract: With the wide application of intelligent unmanned vehicles,intelligent navigation,path planning and obstacle avoidance technology have become important research contents.This paper proposes model-free deep reinforcement learning algorithms DDPG and SAC,which use environmental information to track to the target point,avoid static and dynamic obstacles,and can be generally suitable for different environments.Through the combination of global planning and local obstacle avoidance,it solves the path planning problem with better globality and robustness,solves the obstacle avoidance problem with better dynamicity and generalization,and shortens the iteration time.In the network training stage,PID,A* and other traditional algorithms are combined to improve the convergence speed and stability of the method.Finally,a variety of experimental scenarios such as navigation and obstacle avoidance are designed in the robot operating system ROS and the simulation program gazebo.Simulation results verify the reliability of the proposed approach,which takes the global and dynamic nature of the problem into account and optimizes the generated paths and time efficiency.

Key words: Unmanned vehicle, Obstacle avoidance, Path planning, Deep reinforcement learning

CLC Number: 

  • TP311
[1]CAI K,WANG C,CHENG J,et al.Mobile robot path planning in dynamic environments:a survey[J].arXiv:2006.14195,2020.
[2]ZHANG H,WANG Y,YI J F,et al.Research on intelligent robot systems for emergency prevention and control of major pandemics[J].Scientia Sinica Informationis,2020,50(7):1069-1090.
[3]ZHANG H,LIN W,CHEN A.Path planning for the mobile robot:A review[J].Symmetry,2018,10(10):450-466.
[4]XU X,CAI P,AHMED Z,et al.Path planning and dynamic collision avoidance algorithm under COLREGs via deep reinforcement learning[J/OL].Neurocomputing,2022,468:181-197.https://doi.org/10.1016/j.neucom.2021.09.071.
[5]LI W,CHEN D,LE J.Robot patrol path planning based on combined deep reinforcement learning[C]//2018 IEEE Intl. Conf. on Parallel & Distributed Processing with Applications,Ubiquitous Computing & Communications,Big Data & Cloud Computing,Social Computing & Networking,Sustainable Computing &Communications(ISPA/IUCC/BDCloud/SocialCom/SustainCom).IEEE,2018:659-666.
[6]GAO J,YE W,GUO J,et al.Deep reinforcement learning for indoor mobile robot path planning[J].Sensors,2020,20(19):5493-5507.
[7]PFEIFFER M,SCHAEUBLE M,NIETO J,et al.From perception to decision:A data-driven approach to end-to-end motion planning for autonomous ground robots[C]//2017 IEEE International Conference on Robotics and Automation(ICRA).IEEE,2017:1527-1533.
[8]TAI L,PAOLO G,LIU M,et al.Virtual-to-real deep reinforcement learning:Continuous control of mobile robots for mapless navigation[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE,2017:31-36.
[9]HAN S H,CHOI H J,BENZ P,et al.Sensor-based mobile robot navigation via deep reinforcement learning[C]//2018 IEEE International Conference on Big Data and Smart Computing(BigComp).IEEE,2018:147-154.
[10]LING F,JIMENEZ-RODRIGUEZ A,PRESCOTT T J.Obstacle Avoidance Using Stereo Vision and Deep Reinforcement Lear-ning in an Animal-like Robot[C]//2019 IEEE International Conference on Robotics and Biomimetics(ROBIO).IEEE,2019:71-76.
[11]XIE L,WANG S,MARKHAM A,et al.Towards monocular vision based obstacle avoidance through deep reinforcement lear-ning[J].arXiv:1706.09829,2017.
[12]XIE L,WANG S,ROSA S,et al.Learning with training wheels:speeding up training with a simple controller for deep reinforcement learning[C]//2018 IEEE International Conference on Robotics and Automation(ICRA).IEEE,2018:6276-6283.
[13]KULHÁNEK J,DERNER E,DE BRUIN T,et al.Vision-based navigation using deep reinforcement learning[C]//2019 Euro-pean Conference on Mobile Robots(ECMR).IEEE,2019:1-8.
[14]KÄSTNER L,SHEN Z,MARX C,et al..Autonomous Navigation in Complex Environments using Memory-Aided Deep Reinforcement Learning[C]//2021 IEEE/SICE International Symposium on System Integration(SII).IEEE,2021:170-175.
[15]LUONG M,PHAM C.Incremental learning for autonomousnavigation of mobile robots based on deep reinforcement lear-ning[J].Journal of Intelligent & Robotic Systems,2021,101(1):1-11.
[16]FAN T,CHENG X,PAN J,et al.Crowdmove:Autonomousmapless navigation in crowded scenarios[J].arXiv:1807.07870,2018.
[17]KÄSTNER L,ZHAO X,BUIYAN T,et al.Connecting Deep-Reinforcement-Learning-based Obstacle Avoidance with Conventional Global Planners using Waypoint Generators[C]//2021 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE,2021:1213-1220.
[18]CIMURS R,SUH I H,LEE J H.Goal-driven autonomous mapping through deep reinforcement learning and planning-based navigation[J].arXiv:2103.07119,2021.
[19]GULDENRING R,GÖRNER M,HENDRICH N,et al.Lear-ning local planners for human-aware navigation in indoor environments[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE,2020:6053-6060.
[20]GAO P,LIU Z,WU Z,et al.A global path planning algorithm for robots using reinforcement learning[C]//2019 IEEE International Conference on Robotics and Biomimetics(ROBIO).IEEE,2019:1693-1698.
[21]SICHKAR V N.Reinforcement learning algorithms in globalpath planning for mobile robot[C]//2019 International Confe-rence on Industrial Engineering,Applications and Manufacturing(ICIEAM).IEEE,2019:1-5.
[22]PANOV A I,YAKOVLEV K S,SUVOROV R.Grid path planning with deep reinforcement learning:Preliminary results[J].Procedia Computer Science,2018,123:347-353.
[23]MOERLAND T M,BROEKENS J,JONKER C M.Model-based reinforcement learning:A survey[J].:arXiv:2006.16712,2020.
[24]LIU R,NAGEOTTE F,ZANNE P,et al.Deep reinforcementlearning for the control of robotic manipulation:a focussed mini-review[J].Robotics,2021,10(1):22-34.
[25]NGUYEN H,LA H.Review of deep reinforcement learning for robot manipulation[C]//2019 Third IEEE International Confe-rence on Robotic Computing(IRC).IEEE,2019:590-595.
[26]CHEN W,QIU X,CAI T,et al.Deep reinforcement learning for Internet of Things:A comprehensive survey[J].IEEE Communications Surveys & Tutorials,2021,23(3):1659-1692.
[27]LI D,OKHRIN O.DDPG car-following model with real-worldhuman driving experience in CARLA[J].arXiv:2112.14602,2021.
[28]DE JESUS J C,KICH V A,KOLLING A H,et al.Soft Actor-Critic for Navigation of Mobile Robots[J].Journal of Intelligent &Robotic Systems,2021,102(2):1-11.
[1] ZHANG Qiyang, CHEN Xiliang, ZHANG Qiao. Sparse Reward Exploration Method Based on Trajectory Perception [J]. Computer Science, 2023, 50(1): 262-269.
[2] WEI Nan, WEI Xianglin, FAN Jianhua, XUE Yu, HU Yongyang. Backdoor Attack Against Deep Reinforcement Learning-based Spectrum Access Model [J]. Computer Science, 2023, 50(1): 351-361.
[3] WANG Bing, WU Hong-liang, NIU Xin-zheng. Robot Path Planning Based on Improved Potential Field Method [J]. Computer Science, 2022, 49(7): 196-203.
[4] YU Bin, LI Xue-hua, PAN Chun-yu, LI Na. Edge-Cloud Collaborative Resource Allocation Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2022, 49(7): 248-253.
[5] LI Meng-fei, MAO Ying-chi, TU Zi-jian, WANG Xuan, XU Shu-fang. Server-reliability Task Offloading Strategy Based on Deep Deterministic Policy Gradient [J]. Computer Science, 2022, 49(7): 271-279.
[6] CHEN Bo-chen, TANG Wen-bing, HUANG Hong-yun, DING Zuo-hua. Pop-up Obstacles Avoidance for UAV Formation Based on Improved Artificial Potential Field [J]. Computer Science, 2022, 49(6A): 686-693.
[7] TAN Ren-shen, XU Long-bo, ZHOU Bing, JING Zhao-xia, HUANG Xiang-sheng. Optimization and Simulation of General Operation and Maintenance Path Planning Model for Offshore Wind Farms [J]. Computer Science, 2022, 49(6A): 795-801.
[8] XIE Wan-cheng, LI Bin, DAI Yue-yue. PPO Based Task Offloading Scheme in Aerial Reconfigurable Intelligent Surface-assisted Edge Computing [J]. Computer Science, 2022, 49(6): 3-11.
[9] HONG Zhi-li, LAI Jun, CAO Lei, CHEN Xi-liang, XU Zhi-xiong. Study on Intelligent Recommendation Method of Dueling Network Reinforcement Learning Based on Regret Exploration [J]. Computer Science, 2022, 49(6): 149-157.
[10] LI Peng, YI Xiu-wen, QI De-kang, DUAN Zhe-wen, LI Tian-rui. Heating Strategy Optimization Method Based on Deep Learning [J]. Computer Science, 2022, 49(4): 263-268.
[11] OUYANG Zhuo, ZHOU Si-yuan, LYU Yong, TAN Guo-ping, ZHANG Yue, XIANG Liang-liang. DRL-based Vehicle Control Strategy for Signal-free Intersections [J]. Computer Science, 2022, 49(3): 46-51.
[12] SHI Dian-xi, SU Ya-qian-wen, LI Ning, SUN Yi-xuan, ZHANG Yong-jun. Multi-UAV Cooperative Exploring for Large Unknown Indoor Environment Based on Behavior Tree [J]. Computer Science, 2022, 49(11A): 210900083-11.
[13] CAI Yue, WANG En-liang, SUN Zhe, SUN Zhi-xin. Study on Dual Sequence Decision-making for Trucks and Cargo Matching Based on Dual Pointer Network [J]. Computer Science, 2022, 49(11A): 210800257-9.
[14] DAI Shan-shan, LIU Quan. Action Constrained Deep Reinforcement Learning Based Safe Automatic Driving Method [J]. Computer Science, 2021, 48(9): 235-243.
[15] CHENG Zhao-wei, SHEN Hang, WANG Yue, WANG Min, BAI Guang-wei. Deep Reinforcement Learning Based UAV Assisted SVC Video Multicast [J]. Computer Science, 2021, 48(9): 271-277.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!