Computer Science ›› 2025, Vol. 52 ›› Issue (7): 271-278.doi: 10.11896/jsjkx.240800133

• Artificial Intelligence • Previous Articles     Next Articles

Research on Multi-machine Conflict Resolution Based on Deep Reinforcement Learning

HUO Dan, YU Fuping, SHEN Di, HAN Xueyan   

  1. College of Air Traffic Control and Navigation, Air Force Engineering University, Xi'an 710000, China
  • Received:2024-08-23 Revised:2024-12-02 Published:2025-07-17
  • About author:HUO Dan,born in 1990, master,lec-turer .Her main research interests include air traffic control and collision prevention safety.
  • Supported by:
    National Social Science Foundation of China(22BGL319).

Abstract: With the increase in military,civilian,and general aviation flight activities,the conflict over airspace use has become prominent,and it has become a normal phenomenon for multiple aircraft to fly simultaneously in the same airspace.Therefore,it is an urgent problem that needs to be solved how to provide assistance in avoiding flight collisions through technical means.To tackle the challenge of resolving conflicts between multiple aircraft in flight,this paper introduces a Graph Convolutional Deep Reinforcement Learning(GDQN) algorithm.This algorithm combins multi-agent deep reinforcement learning with a graph con-volutional neural network framework.Initially,it constructs a message-passing function to develop a multi-agent flight conflict model,which can navigate multiple aircraft through three-dimensional,unstructures airspace while avoiding conflicts and collisions.Subsequently,it employes a deep self-learning method based on graph convolutional networks to offer intelligent conflict avoidance solutions for airport scheduling,creats a multi-agent system(MAS) for managing multi-aircraft conflict scenarios.The effectiveness of the algorithm is validated through simulations using extensive training datasets in a controlled environment.The results indicate that the optimized algorithm is effective,achieving a conflict resolution success rate of over 90%,with resolution decision times of less than 3 seconds.Additionally,it significantly reduces the number of air traffic control(ATC) commands issued and improves overall operational efficiency.

Key words: Deep reinforcement learning, Graph convolutional neural network, Message passing, Multi-agent model, Multi-aircraft flight, Conflict resolution

CLC Number: 

  • TP389.1
[1]WANG Z,LI H,WANG J,et al.Deep reinforcement learningbasedconflict detection and resolution in air traffic control[J].IET Intelligent Transport Systems,2019,13(6):133-142.
[2]LIU X,XIAO G.Flight Conflict Resolution and Trajectory Recovery Through Mixed Integer Nonlinear Programming Based on Speed and Heading Angle Change[J].Transportation Research Record,2024,2678(4):751-775.
[3]WEN H.Research on Flight Conflict Resolution based on Deep reinforcement Learning[D].Chengdu:Sichuan University,2021:12-23.
[4]CAI M,WAN L J,GAO Z Z,et al.Conflict detection method based on K-Means spatial clustering in grid system[C]//China Association for Science and Technology,Ministry of Transport,Chinese Academy of Engineering,Hubei Provincial People's Government.Proceedings of World Transport Congress 2022(WTC2022).Air traffic Control and Navigation College,Air Force Engineering University,2022:666-673.
[5]TONG L,YANG J,GAN X S,et al.More chaotic ant colony algorithm based on improved machine conflict free simulation[J].Journal of system simulation,2025,5(1):155-166.
[6]BRITTAIN M,WEI P.Autonomous separation assurance in anhigh density en-route sector:A deep multi-agent reinforcement learning approach[C]//Proceedingsof Institute of Electrical and Electronics Engineers(IEEE) Intelligent Transportation Systems Conference.Piscataway,NJ:IEEE Press,2019:3256-3262.
[7]MOHAMMAD E,FARIBORZ H HAGHIIGHAT,et al.Transfer learning for occupancy-based HVAC control:A data-driven approach using unsupervised learning of occupancy profiles and deep reinforcement learning[J].Energy & Buildings,2023,300(9):356-362.
[8]VSWANI A,SHAZEER N.Attention is all you need[C]//The 31st Annual Conference on Neural Information Processing Systems(NeurIPS).Long Beach,CA,2017:5999-6009.
[9]LI S,EGOROV M,KOCHENDERFER J.Optimizing CollisionAvoidance in Dense Airspace using Deep ReinforcementLearning[C]//The 13th USA/Europe Air Traffic Management Research and Development Seminar(ATM2019),Vienna,Austria,2019:45-49.
[10]XIE X F,SMITH S,BARLOW G.Coordinated look-aheadscheduling for real-time traffic signal control[C]//International Joint Conference on Autonomous Agents and Multiagent Systems(AAMAS).Valencia,Spain,2012:1271-1272.
[11]LIU Y P,SUI D,LIN Y D.Research on the learning behavior of Controller Agent based on Q Learning[J].Journal of Harbin University of Commerce(Natural Science Edition),2016,32(6):763-768.
[12]SILVER D,HUANG A.mastering the game of go with deep neural networks and tree search[J].Nature,2016,529(7587):484-489.
[13]BROWN N,SANDHOLM T.Superhuman AI for multiplayerpoker[J].Science,2019,365(6456):885-890.
[14]SUI D,XU W,ZHANG K.Study on the resolution of multi-aircraft flight conflicts based on an IDQN[J].Chinese Journal of Aeronautics,2022,35(2):195-213.
[15]BI K X,WU M G,WEN X X,et al.Conflict resolution strategy based on flight conflict network and genetic algorithm[J].Systems Engineering and Electronics,2023,45(5):1429-1440.
[16]RASHID T,SAMVELYAN M.Monotonic value function factorisation for deep multi-agent reinforcement learning[C]//The 35th International Conference on Machine Learning(ICML).Stockholm,Sweden,2018:1335-1345.
[17]SUKHBAATAR S,SZLAM A,FERGUS R.Learning Multiagent Communication with Backpropagation[C]//The 30th Annual Conference on Neural Information Processing Systems(NeurIPS).Barcelona,Spain,2016:2244-2252.
[18]JIANG J,DUN C.Graph convolutional reinforcement learning[C]//The 8th International Conference on Learning Representations(ICLR).Addis Ababa Ethiopia,2020:1265-1273.
[19]JUNPENG Y,YIYU C.A practical reinforcement learningframework for automatic radar detection[J].ZTE Communications,2023,21(3):22-28.
[20]ZHANG Z,CUI P,ZHU W.Deep learning on graphs:A survey[C]//Proceedings of Institute of Electrical and Electronics Engineers(IEEE)Transactions on Knowledge and Data Enginee-ring.Piscataway,NJ:IEEE Press,2020.
[21]LI S,EGOROV M,KOCHENDERFER J.Optimizing collision avoidance in dense airspace using deep reinforcement learning[C]//The 13th USA/Europe Air Traffic Management Research and Development Seminar(ATM2019).Vienna,Austria,2019:45-49.
[1] WU Zongming, CAO Jijun, TANG Qiang. Online Parallel SDN Routing Optimization Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(6A): 240900018-9.
[2] ZHAO Xuejian, YE Hao, LI Hao, SUN Zhixin. Multi-AGV Path Planning Algorithm Based on Improved DDPG [J]. Computer Science, 2025, 52(6): 306-315.
[3] WANG Chenyuan, ZHANG Yanmei, YUAN Guan. Class Integration Test Order Generation Approach Fused with Deep Reinforcement Learning andGraph Convolutional Neural Network [J]. Computer Science, 2025, 52(6): 58-65.
[4] LI Yuanbo, HU Hongchao, YANG Xiaohan, GUO Wei, LIU Wenyan. Intrusion Tolerance Scheduling Algorithm for Microservice Workflow Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(5): 375-383.
[5] ZHENG Longhai, XIAO Bohuai, YAO Zewei, CHEN Xing, MO Yuchang. Graph Reinforcement Learning Based Multi-edge Cooperative Load Balancing Method [J]. Computer Science, 2025, 52(3): 338-348.
[6] DU Likuan, LIU Chen, WANG Junlu, SONG Baoyan. Self-learning Star Chain Space Adaptive Allocation Method [J]. Computer Science, 2025, 52(3): 359-365.
[7] HUO Xingpeng, SHA Letian, LIU Jianwen, WU Shang, SU Ziyue. Windows Domain Penetration Testing Attack Path Generation Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(3): 400-406.
[8] XU Donghong, LI Bin, QI Yong. Task Scheduling Strategy Based on Improved A2C Algorithm for Cloud Data Center [J]. Computer Science, 2025, 52(2): 310-322.
[9] WANG Tianjiu, LIU Quan, WU Lan. Offline Reinforcement Learning Algorithm for Conservative Q-learning Based on Uncertainty Weight [J]. Computer Science, 2024, 51(9): 265-272.
[10] ZHOU Wenhui, PENG Qinghua, XIE Lei. Study on Adaptive Cloud-Edge Collaborative Scheduling Methods for Multi-object State Perception [J]. Computer Science, 2024, 51(9): 319-330.
[11] LI Danyang, WU Liangji, LIU Hui, JIANG Jingqing. Deep Reinforcement Learning Based Thermal Awareness Energy Consumption OptimizationMethod for Data Centers [J]. Computer Science, 2024, 51(6A): 230500109-8.
[12] GAO Yuzhao, NIE Yiming. Survey of Multi-agent Deep Reinforcement Learning Based on Value Function Factorization [J]. Computer Science, 2024, 51(6A): 230300170-9.
[13] WANG Shuanqi, ZHAO Jianxin, LIU Chi, WU Wei, LIU Zhao. Fuzz Testing Method of Binary Code Based on Deep Reinforcement Learning [J]. Computer Science, 2024, 51(6A): 230800078-7.
[14] LIU Hui, JI Ke, CHEN Zhenxiang, SUN Runyuan, MA Kun, WU Jun. Malicious Attack Detection in Recommendation Systems Combining Graph Convolutional Neural Networks and Ensemble Methods [J]. Computer Science, 2024, 51(6A): 230700003-9.
[15] HUANG Rui, XU Ji. Text Classification Based on Invariant Graph Convolutional Neural Networks [J]. Computer Science, 2024, 51(6A): 230900018-5.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!