Computer Science ›› 2020, Vol. 47 ›› Issue (2): 169-174.doi: 10.11896/jsjkx.190600154

• Artificial Intelligence • Previous Articles     Next Articles

Traffic Signal Control Method Based on Deep Reinforcement Learning

SUN Hao,CHEN Chun-lin,LIU Qiong,ZHAO Jia-bao   

  1. (Department of Control and Systems Engineering,Nanjing University,Nanjing 210093,China)
  • Received:2019-03-25 Online:2020-02-15 Published:2020-03-18
  • About author:SUN Hao,born in 1996,postgraduate.His main research interests include deep learning and reinforcement lear-ning;ZHAO Jia-bao,born in 1972,Ph.D,associate professor.His main research interests include coordination and control methods for CAVs and knowledge automation in AIOps (Artificial Intelligence for IT Operations).
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (71732003) and National Key Research and Development Program of China (2016YFD0702100).

Abstract: The control of traffic signals is always a hotspot in intelligent transportation systems research.In order to adapt and coordinate traffic more timely and effectively,a novel traffic signal control algorithm based on distributional deep reinforcement learning was proposed.The model utilizes a deep neural network framework composed of target network,double Q network and value distribution to improve the performance.After integrating the discretization of the high-dimensional real-time traffic information at intersections with waiting time,queue length,delay time and phase information as states and making appropriate definitions of actions,rewards in the algorithm,it can learn the control strategy of traffic signals online and realize the adaptive control of traffic signals.It was compared with three typical deep reinforcement learning algorithms,and the experiments were performed in SUMO (Simulation of Urban Mobility) with the same setting.The results show that the distributional deep reinforcement learning algorithm is more efficient and robust,and has better performance on average delay,travel time,queue length,and wai-ting time of vehicles.

Key words: Deep reinforcement learning, Distributional reinforcement learning, Intelligent transportation, Traffic signal control

CLC Number: 

  • TP181
[1]SUTTON R S,BARTO A G.Introduction to reinforcement learning[M].Cambridge:MIT Press,1998.
[2]BELLEMARE M G,DABNEY W,MUNOS R.A distributionalperspective on reinforcement learning[C]∥Proceedings of the 34th International Conference on Machine,2017:449-458.
[3]CHIS S.Adaptive traffic signal control using fuzzy logic[C]∥Proceedings of the Intelligent Vehicles92 Symposium.IEEE,1992:98-107.
[4]PANDIT K,GHOSAL D,ZHANG H M,et al.Adaptive traffic signal control with vehicular ad hoc networks[J].IEEE Transactions on Vehicular Technology,2013,62(4):1459-1471.
[5]LIN W H,WANG C.An enhanced 0-1 mixed-integer LP formulation for traffic signal control[J].IEEE Transactions on Intelligent transportation systems,2004,5(4):238-245.
[6]PRASHANTH L A,BHATNAGAR S.Reinforcement learning with function approximation for traffic signal control[J].IEEE Transactions on Intelligent Transportation Systems,2010,12(2):412-421.
[7]GIRIANNA M,BENEKOHAL R F.Using genetic algorithms to design signal coordination for oversaturated networks[J].Journal of Intelligent Transportation Systems,2004,8(2):117-129.
[8]SANCHEZ-MEDINA J J,GALAN-MORENO M J,RUBIO-ROYO E.Traffic signal optimization in “La Almozara” district in Saragossa under congestion conditions,using genetic algorithms,traffic microsimulation,and cluster computing[J].IEEE Transactions on Intelligent Transportation Systems,2009,11(1):132-141.
[9]YU X H,RECKER W.Stochastic adaptive control model for traffic signal systems[J].Transportation Research Part C:Emerging Technologies,2006,14(4):263-282.
[10]GOKULAN B P,SRINIVASAN D.Distributed geometric fuzzy multi agent urban traffic signal control[J].IEEE Transactions on Intelligent Transportation Systems,2010,11(3):714-727.
[11]BOWLING M.Multi agent learning in the presence of agents with limitations[R].Carnegie-Mellon Univ Pittsburgh Pa School of Computer Science,2003.
[12]PRASHANTH L,BHATNAGAR S.Threshold tuning using stochastic optimization for graded signal control[J].IEEE Transactions on Vehicular Technology,2012,61(9):3865-3880.
[13]LIU W,QIN G,HE Y,et al.Distributed cooperative reinforce-ment learning-based traffic signal control that integrates v2x networks’ dynamic clustering[J].IEEE Transactions onVehi-cular Technology,2017,66(10):8667-8681.
[14]GENDERS W,RAZAVI S.Using a deep reinforcement learning agent for traffic signal control[J].arXiv:1611.01142.
[15]El-TANTAWY S,ABDULHAI B,ABDELGAWAD H.Multi agent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC):methodology and large-scale application on downtown Toronto[J].IEEE Transactions on Intelligent Transportation Systems,2013,14(3):1140-1150.
[16]WIERING M A.Multi-agent reinforcement learning for traffic light control[C]∥Machine Learning:Proceedings of the Seventeenth International Conference (ICML’2000).2000:1151-1158.
[17]WIERING M,VREEKEN J,VAN VEENEN J,et al.Simulation and optimization of traffic in a city[C]∥IEEE Intelligent Vehicles Symposium,2004.IEEE,2004:453-458.
[18]MARSETIC R,SEMROV D,ZURA M.Road artery traffic light optimization with use of the reinforcement learning[J].PROMET-Traffic & Transportation,2014,26(2):101-108.
[19]PUTERMAN M L.Markov Decision Processes:Discrete Sto-chastic Dynamic Programming[M].John Wiley & Sons,2014.
[20]MNIH V,KAVUKCUOGLU K,SILVER D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529-533.
[21]VAN HASSELT H,GUEZ A,SILVER D.Deep Reinforcement Learning with Double Q-Learning[C]∥Association for the Advance of Artificial Intelligence.2016:2094-2100.
[22]SCHAUL T,QUAN J,ANTONOGLOU I,et al.Prioritized experience replay[C]∥Proceedings of the 4th International Con-ference on Learning Representations.San Juan,Puerto Rico,2016:322-355.
[1] YU Bin, LI Xue-hua, PAN Chun-yu, LI Na. Edge-Cloud Collaborative Resource Allocation Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2022, 49(7): 248-253.
[2] LI Meng-fei, MAO Ying-chi, TU Zi-jian, WANG Xuan, XU Shu-fang. Server-reliability Task Offloading Strategy Based on Deep Deterministic Policy Gradient [J]. Computer Science, 2022, 49(7): 271-279.
[3] XIE Wan-cheng, LI Bin, DAI Yue-yue. PPO Based Task Offloading Scheme in Aerial Reconfigurable Intelligent Surface-assisted Edge Computing [J]. Computer Science, 2022, 49(6): 3-11.
[4] HONG Zhi-li, LAI Jun, CAO Lei, CHEN Xi-liang, XU Zhi-xiong. Study on Intelligent Recommendation Method of Dueling Network Reinforcement Learning Based on Regret Exploration [J]. Computer Science, 2022, 49(6): 149-157.
[5] LI Peng, YI Xiu-wen, QI De-kang, DUAN Zhe-wen, LI Tian-rui. Heating Strategy Optimization Method Based on Deep Learning [J]. Computer Science, 2022, 49(4): 263-268.
[6] OUYANG Zhuo, ZHOU Si-yuan, LYU Yong, TAN Guo-ping, ZHANG Yue, XIANG Liang-liang. DRL-based Vehicle Control Strategy for Signal-free Intersections [J]. Computer Science, 2022, 49(3): 46-51.
[7] DAI Shan-shan, LIU Quan. Action Constrained Deep Reinforcement Learning Based Safe Automatic Driving Method [J]. Computer Science, 2021, 48(9): 235-243.
[8] CHENG Zhao-wei, SHEN Hang, WANG Yue, WANG Min, BAI Guang-wei. Deep Reinforcement Learning Based UAV Assisted SVC Video Multicast [J]. Computer Science, 2021, 48(9): 271-277.
[9] LIANG Jun-bin, ZHANG Hai-han, JIANG Chan, WANG Tian-shu. Research Progress of Task Offloading Based on Deep Reinforcement Learning in Mobile Edge Computing [J]. Computer Science, 2021, 48(7): 316-323.
[10] WANG Ying-kai, WANG Qing-shan. Reinforcement Learning Based Energy Allocation Strategy for Multi-access Wireless Communications with Energy Harvesting [J]. Computer Science, 2021, 48(7): 333-339.
[11] ZHOU Shi-cheng, LIU Jing-ju, ZHONG Xiao-feng, LU Can-ju. Intelligent Penetration Testing Path Discovery Based on Deep Reinforcement Learning [J]. Computer Science, 2021, 48(7): 40-46.
[12] LI Bei-bei, SONG Jia-rui, DU Qing-yun, HE Jun-jiang. DRL-IDS:Deep Reinforcement Learning Based Intrusion Detection System for Industrial Internet of Things [J]. Computer Science, 2021, 48(7): 47-54.
[13] ZHOU Xin, LIU Shuo-di, PAN Wei, CHEN Yuan-yuan. Vehicle Color Recognition in Natural Traffic Scene [J]. Computer Science, 2021, 48(6A): 15-20.
[14] NIU Kang-li, CHEN Yu-zhang, ZHANG Gong-ping, TAN Qian-cheng, WANG Yi-chong, LUO Mei-qi. Vehicle Flow Measuring of UVA Based on Deep Learning [J]. Computer Science, 2021, 48(6A): 275-280.
[15] FAN Yan-fang, YUAN Shuang, CAI Ying, CHEN Ruo-yu. Deep Reinforcement Learning-based Collaborative Computation Offloading Scheme in VehicularEdge Computing [J]. Computer Science, 2021, 48(5): 270-276.
Full text



No Suggested Reading articles found!