Computer Science

Computer Science ›› 2025, Vol. 52 ›› Issue (2): 279-290.doi: 10.11896/jsjkx.240100133

• Computer Network • Previous Articles     Next Articles

Fully Distributed Event Driven Bipartite Consensus Algorithm Based on Reinforcement Learning

CAI Yuliang1, LYU Chunhui1, HE Qiang2, YU Bo3, CHEN Dongyue4, WANG Youtong1, WANG Qiang1, LIU Yuxuan1, ZHAO Jingjing1   

  1. 1 School of Mathematics and Statistics,Liaoning University,Shenyang 110036,China
    2 College of Medicine and Biological Information Engineering,Northeastern University,Shenyang 110016,China
    3 Shenyang Institute of Computing Technology,Chinese Academy of Sciences Co.,Ltd.,Shenyang 110168,China
    4 College of Information Science and Engineering,Northeastern University,Shenyang 110819,China
  • Received:2024-01-16 Revised:2024-09-13 Online:2025-02-15 Published:2025-02-17
  • About author:CAI Yuliang,born in 1988,Ph.D,asso-ciate professor.Her main research inte-rests include network control,mul-tiagent systems and machine learning.
    HE Qiang,born in 1991,Ph.D,associate professor,is a member of CCF(No.D3158M).His main research interests include social networks and machine learning.
  • Supported by:
    National Key Research and Development Program of China(2021YFB3300900),Youth Science Foundation of National Natural Science Foundation of China(62303202),75th Batch of General Projects of China Postdoctoral Science Foundation(2024M753407),Natural Science Foundation of Liaoning Province,China(2023-BS-082) and Liaoning Province Social Science Planning Fund Project(23C10140012).

PDF (PC)

116

Abstract: Reinforcement learning(RL) methods and fully distributed event driven control strategies are used to study the bipartite consensus problem of multi-agent systems(MASs) with unknown system model information.Firstly,a hybrid event triggered mechanism based on state threshold and time threshold is proposed to reduce the communication frequency between intelligent agents.Secondly,an adaptive event triggered consensus control protocol is designed using locally sampled state information,resulting in the consensus error of all following agents eventually approaching zero.The effectiveness of the above event triggered mechanism is confirmed by excluding Zeno behavior within a limited time.Then,based on the RL method,a model free algorithm is proposed to obtain the feedback gain matrix,and an adaptive event triggered control strategy is constructed in the presence of unknown model information.Unlike existing related works,the RL-based event triggered adaptive control algorithm only relies on locally sampled state information and is independent of any model information or global network information.In addition,we extend the above results to the switching topology scenario,which is more challenging because the state estimation is updated in the following two situations:1)when the interaction graph switches or 2)when the event triggering mechanism is satisfied.Finally,the effectiveness of the adaptive event triggered control algorithm is verified through examples.

Key words: Reinforcement learning, Event-driven, Fully distributed control, Multi-agent systems, Bipartite consensus

CLC Number: 

  • TP393
[1]SU H S,MIAO S X.Consensus on Directed Matrix-weightedNetworks[J].IEEE Transactions on Automatic Control,2023,68(4):2529-2535.
[2]JIN X Z,CHE W W,WU Z G,et al.Adaptive Consensus and Circuital Implementation of a Class of Faulty Multiagent Systems[J].IEEE Transactions on Systems,Man,and Cybernetics:Systems,2022,52(1):226-237.
[3]TIAN L,DONG X W,ZHAO Q L.Distributed Adaptive Time-varying Output Formation Tracking for Heterogeneous Swarm Systems[J].Acta Automatica Sinica,2021,47(10):2386-2401.
[4]CAI Y L,ZHANG H G,WANG Y C,et al.Adaptive Bipartite Fixed-time Time-varying Output Formation-containment Tracking of Heterogeneous Linear Multiagent Systems[J].IEEE Transactions on Neural Networks and Learning Systems,2022,33(9):4688-4698.
[5]LI T S,BAI W W,LIU Q,et al.Distributed Fault-tolerant Containment Control Protocols for the Discrete-time Multiagent Systems via Reinforcement Learning Method[J].IEEE Transactions on Neural Networks and Learning Systems,2023,34(8):3979-3991.
[6]ZHAO X Y,ZHOU S L,WANG S L.Formation Containment Control of Multi-UAV System under Switching Topology[J].Computer Science,2020,47(S1):577-582.
[7]WANG B H,CHEN W S,ZHANG B.Semi-global RobustTracking Consensus for Multi-agent Uncertain Systems with Input Saturation via Metamorphic Low-Gain Feedback[J].Automatica,2019,103:363-373.
[8]WANG X L,SU H S,CHEN M Z Q,et al.Observer-based Robust Coordinated Control of Multiagent Systems with Input Saturation[J].IEEE Transactions on Neural Networks and Learning Systems,2018,29(5):1933-1946.
[9]XIN Y X,HUA D Y,ZHANG L.Multi-agent ReinforcementLearning Algorithm Based on Planning[J].Computer Science,2024,51(5):179-192.
[10]LI Z J,WU J,ZHAN X S,et al.Distributed Adaptive Prede-fined-time Bipartite Containment Algorithm for Nonlinear Multi-agent Systems ith Actuator Faults[J].IEEE Transactions on Circuits and Systems II:Express Briefs,2023,70(6):2141-2145.
[11]LI Y H,CUI L Z,BU X H,et al.Model-free Adaptive Cluster Consensus Control for Nonlinear Multi-agent Systems[J]. Control and Decision,2024,39(1):345-352.
[12]TIAN Y P,LIU C L.Consensus of Multi-agent Systems with Diverse Input and Communication Delays[J].IEEE Transactions on Automatic Control,2008,53(9):2122-2128.
[13]YE Y Y,SU H S.Consensus of Delayed Fractional-order Multiagent Systems with Intermittent Sampled Data[J].IEEE Transactions on Industrial Informatics,2020,16(6):3828-3837.
[14]ZHAO M Y,YE J.Synchronization of Uncertain Complex Networks with Sapmpled-data and Input Saturation[J].Computer Science,2021,48(S2):481-484.
[15]LIU J,WU Y B,XUE L,et al.A New Intermittent Control Approach to Practical Fixed-time Consensus with Input Delay[J].IEEE Transactions on Circuits and Systems II:Express Briefs,2023,70(6):2186-2190.
[16]ZUO Z Q,JI J W,ZHANG Z C,et al.Consensus of Multi-agent Systems with Asymmetric Input Saturation over Directed Graph[J].IEEE Transactions on Circuits and Systems II:Express Briefs,2023,70(4):1515-1519.
[17]WANG Y J,LI J D,LI D Q.Distributed Subgradient Optimization Algorithm with Communication Delays for Multi-agent Switched Networks[J].Computer Science,2019,46(7):81-85.
[18]DIMAROGONAS D V,FRAZZOLI E,JOHANSSON K H.Distributed Event-triggered Control for Multi-agent Systems[J].IEEE Transactions on Automatic Control,2012,57(5):1291-1297.
[19]NOWZARI C,GARCIA E,CORT′ES J.Event-triggered Communication and Control of Networked Systems for Multi-agent Consensus[J].Automatica,2019,105:1-27.
[20]YI X L,YANG T,WU J F,et al.Distributed Event-triggered Control for Global Consensus of Multi-agent Systems with Input Saturation[J].Automatica,2019,100:1-9.
[21]DU C K,LIU X D,REN W,et al.Finite-time Consensus for Linear Multiagent Systems via Event-triggered Strategy without Continuous Communication[J].IEEE Transactions on Control of Network Systems,2020,7(1):19-29.
[22]WANG L,HU A H,JIANG Z X.Dynamic event-triggered Consensus of Multi-agent Systems under Cyberattacks[J].Control and Decision,2023,38(5):1295-1302.
[23]HU W F,LIU L,FENG G.Output Consensus of Heterogeneous Linear Multi-agent Systems by Distributed Event-triggered/self-triggered Strategy[J].IEEE Transactions on Cybernetics,2017,47(8):1914-1924.
[24]SU H S,WANG X,ZENG Z G.Consensus of Second-order Hybrid Multiagent Systems by Event-triggered Strategy[J].IEEE Transactions on Cybernetics,2020,50(11):4649-4657.
[25]LI Z K,WEN G H,DUAN Z S,et al.Designing Fully Distributed Consensus Protocols for Linear Multi-agent Systems with Directed Graphs[J].IEEE Transactions on Automatic Control,2015,60(4):1152-1157.
[26]JIANG W,WEN G G,PENG Z X,et al.Fully Distributed Formation-containment Control of Heterogeneous Linear Multi-agent Systems[J].IEEE Transactions on Automatic Control,2019,64(9):3889-3896.
[27]WANG X L,SU H S,WANG X F,et al.Fully DistributedEvent-triggered Semiglobal Consensus of Multi-agent Systems with Input Saturation[J].IEEE Transactions on Industrial Electronics,2017,64(6):5055-50064.
[28]YUAN S Z,LIU Z L,ZHENG L H.Ship Berthing Based onEvent-triggered Adaptive Horizon MPC[J].Control and Decision,2024,39(1):336-344.
[29]CHEN S M,JIANG G L,ZHANG Z.Bipartite Practical Consensus Control of Multi-agent Systems with Communication Constraints[J].Acta Automatica Sinica,2022,48(5):1318-1326.
[30]ZHAO L N,MA H J,XU L X,et al.Observer-Based Adaptive Sampled-data Event-triggered Distributed Control for Multi-agent Systems[J].IEEE Transactions on Circuits and Systems II:Express Briefs,2020,67(1):97-101.
[31]MA C Q,XIE L H.Necessary and Sufficient Conditions for Leader-following Bipartite Consensus with Measurement Noise[J].IEEE Transactions on Systems,Man,and Cybernetics:Systems,2020,50(5):1976-1981.
[32]ZHU Y R,LI S L,MA J Y,et al.Bipartite Consensus in Networks of Agents with Antagonistic Interactions and Quantization[J].IEEE Transactions on Circuits and Systems II:Express Briefs,2018,65(12):2012-2016.
[33]WEN G H,WANG H,YU X H,et al.Bipartite Tracking Consensus of Linear Multi-agent Systems with a Dynamic Leader[J].IEEE Transactions on Circuits and Systems II:Express Briefs,2018,65(9):1204-1208.
[34]YAN H Y,LIU X Y,CAO J D.Bipartite Quasi-consensus of Heterogeneous Multi-agent Systems Based on Neural Network Approximation[J].Acta Automatica Sinica,2023,38(5):1312-1318.
[35]YAGHMAIE F A,SU R,LEWIS F L,et al.Bipartite and Cooperative Output Synchronizations of Linear Heterogeneous Agents:A Unified Framework[J].Automatica,2017,80:172-176.
[36]LI J J,CHEN X.Event-triggered Bipartite Consensus for Multi-agent Systems Associated with Signed Graphs[C]//2018 33rd Youth Academic Annual Conference of Chinese Association of Automation.Nanjing,China,2018:988-992.
[37]ZUO Z Q,MA J J,WANG Y J.Layered Event-triggered Control for Group Consensus with Both Competition and Cooperation Interconnections[J].Neurocomputing,2018,275(31):1964-1972.
[38]PAN L L,SHAO H B,LI D W,et al.Event-triggered Consensus Problem of General Multi-agent System on Signed Networks[C]//2018 15th International Conference on Control,Automation,Robotics and Vision(ICARCV).Singapore,2018:1069-1074.
[39]ABOUHEAF M L,LEWIS F L,MAHMOUD M S,et al.Discrete-time Dynamic graphical Games:Model-free Reinforcement Learning Solution[J].Control Theory and Technology,2015,13(1):55-69.
[40]WANG W,CHEN X.Model-free Optimal Containment Control of Multi-agent Systems Based on Actor-critic Framework[J].Neurocomputing,2018,314(7):242-250.
[41]RIZVI S A A,LIN Z L.Output Feedback Reinforcement Learning Based Optimal Output Synchronisation of Heterogeneous Discrete-time Multi-agent Systems[J].IET Control Theory and Applications,2019,13(17):2866-2876.
[42]LONG M K,SU H S,WANG X L,et al.An Iterative Q-learning Based Global Consensus of Discrete-time Saturated Multi-agent Systems[J].Chaos Interdisciplinary Journal of Nonlinear Science,2019,29(10):103127.
[43]LONG M K,SU H S,ZENG Z G.Output-feedback Global Consensus of Discrete-time Multiagent Systems Subject to Input Saturation via Q-learning Method[J].IEEE Transactions on Cybernetics,2022,52(3):1661-1670.
[44]QIN Z H,LI N,LIU X T.Overview of Research on Model-freeReinforcement Learning[J].Computer Science,2021,48(3):180-187.
[45]XU W Y,HO D W C,LI L L,et al.Event-triggered Schemes on Leader-following Consensus of General Linear Multiagent Systems under Different Topologies[J].IEEE Transactions on Cybernetics,2017,47(1):212-223.
[46]CHENG T H,KAN Z,KLOTZ J R,et al.Event-triggered Control of Multiagent Systems for Fixed and Time-varying Network Topologies[J].IEEE Transactions on Automatic Control,2017,62(10):5365-5371.
[47]ALTAFINI C.Consensus Problems on Networks with Antagonistic Interactions[J].IEEE Transactions on Automatic Control,2013,58(4):935-946.
[48]CAI Y L,ZHANG H G,LIU Y.et al.Distributed Bipartite Finite-time Event-triggered Output Consensus for Heterogeneous Linear Multi-agent Systems under Directed Signed Communication Topology[J].Applied Mathematics and Computation,2020,378:125162.
[49]AHMED I,REHAN M,IQBAL N,et al.A Novel Event-Triggered Consensus Approach for Generic Linear Multi-Agents Under Heterogeneous Sector-Restricted Input Nonlinearities[J].IEEE Transactions on Network Science and Engineering,2023,10(3):1648-1658.
[50]WANG X L,SU H S.Completely model-free RL-based consensus of continuous-time multi-agent systems[J].Applied Mathematics and Computation,2020,382:125312.
[51]KHALIL H K,GRIZZLE J.Nonlinear Systems Third Edition[M].Upper Saddle River Nj Prentice Hall Inc,2002.
[52]WANG X L,SU H S.Completely Model-free RL-based Consensus of Continuous-time Multi-agent Systems[J].Applied Mathematics and Computation,2020,382:125312.
[53]BIAN T,JIANG Z P.Value Iteration and Adaptive DynamicProgramming for Data-driven Adaptive Optimal Control Design[J].Automatica,2016,71:348-360.
[54]LI Z K,REN W,LIU X D,et al.Distributed Consensus of Linear Multiagent Systems with Adaptive Dynamic Protocols[J].Automatica,2013,49(7):1986-1995.
[55]LIU X D,DU C K,LIU H K.Decentralized Event-triggeredOutput Consensus for Heterogeneous Multi-agent Systems with General Linear Dynamics[C]//2017 36th Chinese Control Conference(CCC).IEEE,2017:8282-8287.
[56]HU W,LIU L.Cooperative Output Regulation of Heterogeneous Linear Multi-agent Systems by Event-triggered Control[J].IEEE Transactions on Cybernetics,2017,47(1):105-116.
[57]BIAN T,JIANG Z P.Value Iteration and Adaptive DynamicProgramming for Data-driven Adaptive Optimal Control Design[J].Automatica,2016,71:348-360.
[58]BERNSTEIN D S,Matrix Mathematics:Theory,Facts,and Formulas[M].Princeton,NJ:Princeton University Press,2009.
[1] PIAO Mingjie, ZHANG Dongdong, LU Hu, LI Rupeng, GE Xiaoli. Study on Multi-agent Supply Chain Inventory Management Method Based on Improved Transformer [J]. Computer Science, 2025, 52(6A): 240500054-10.
[2] WU Zongming, CAO Jijun, TANG Qiang. Online Parallel SDN Routing Optimization Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(6A): 240900018-9.
[3] ZHAO Chanchan, YANG Xingchen, SHI Bao, LYU Fei, LIU Libin. Optimization Strategy of Task Offloading Based on Meta Reinforcement Learning [J]. Computer Science, 2025, 52(6A): 240800050-8.
[4] XU Dan, WANG Jiangtao. Design of Autonomous Decision for Trajectory Optimization of Intelligent Morphing Aircraft [J]. Computer Science, 2025, 52(6A): 240600068-7.
[5] WANG Chenyuan, ZHANG Yanmei, YUAN Guan. Class Integration Test Order Generation Approach Fused with Deep Reinforcement Learning andGraph Convolutional Neural Network [J]. Computer Science, 2025, 52(6): 58-65.
[6] ZHAO Xuejian, YE Hao, LI Hao, SUN Zhixin. Multi-AGV Path Planning Algorithm Based on Improved DDPG [J]. Computer Science, 2025, 52(6): 306-315.
[7] LI Yuanbo, HU Hongchao, YANG Xiaohan, GUO Wei, LIU Wenyan. Intrusion Tolerance Scheduling Algorithm for Microservice Workflow Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(5): 375-383.
[8] HAN Lin, WANG Yifan, LI Jianan, GAO Wei. Automatic Scheduling Search Optimization Method Based on TVM [J]. Computer Science, 2025, 52(3): 268-276.
[9] ZHENG Longhai, XIAO Bohuai, YAO Zewei, CHEN Xing, MO Yuchang. Graph Reinforcement Learning Based Multi-edge Cooperative Load Balancing Method [J]. Computer Science, 2025, 52(3): 338-348.
[10] DU Likuan, LIU Chen, WANG Junlu, SONG Baoyan. Self-learning Star Chain Space Adaptive Allocation Method [J]. Computer Science, 2025, 52(3): 359-365.
[11] HUO Xingpeng, SHA Letian, LIU Jianwen, WU Shang, SU Ziyue. Windows Domain Penetration Testing Attack Path Generation Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(3): 400-406.
[12] XU Donghong, LI Bin, QI Yong. Task Scheduling Strategy Based on Improved A2C Algorithm for Cloud Data Center [J]. Computer Science, 2025, 52(2): 310-322.
[13] WANG Qidi, SHEN Liwei, WU Tianyi. Option Discovery Method Based on Symbolic Knowledge [J]. Computer Science, 2025, 52(1): 277-288.
[14] WANG Yanning, ZHANG Fengdi, XIAO Dengmin, SUN Zhongqi. Multi-agent Pursuit Decision-making Method Based on Hybrid Imitation Learning [J]. Computer Science, 2025, 52(1): 323-330.
[15] BAO Zepeng, QIAN Tieyun. Survey on Large Model Red Teaming [J]. Computer Science, 2025, 52(1): 34-41.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.