计算机科学

计算机科学 ›› 2025, Vol. 52 ›› Issue (9): 337-345.doi: 10.11896/jsjkx.240700197

• 人工智能 • 上一篇    下一篇

基于多智能体深度强化学习的光储充电站动态定价及能源调度策略

陈锦韬1,3, 林兵2,3, 林崧1, 陈静3, 陈星3   

  1. 1 福建师范大学计算机与网络空间安全学院 福州 350117
    2 福建师范大学物理与能源学院 福州 350117
    3 福建省网络计算与智能信息处理重点实验室 福州 350116
  • 收稿日期:2024-07-30 修回日期:2024-11-29 出版日期:2025-09-15 发布日期:2025-09-11
  • 通讯作者: 林兵(WheelLX@163.com)
  • 作者简介:(963200594@qq.com)
  • 基金资助:
    国家自然科学基金(62072108);福建省高校产学合作项目(2022H6024,2021H6026);福建省高校物理学学科联盟教学改革项目(FJPHYS-2022-B02);福建省促进海洋与渔业产业高质量发展专项资金(FJHYF-ZH-2023-02);福建省技术创新重点攻关及产业化项目(2024XQ004)

Dynamic Pricing and Energy Scheduling Strategy for Photovoltaic Storage Charging Stations Based on Multi-agent Deep Reinforcement Learning

CHEN Jintao1,3, LIN Bing2,3, LIN Song1, CHEN Jing3, CHEN Xing3   

  1. 1 College of Computer and Cyber Security,Fujian Normal University,Fuzhou 350117,China
    2 College of Physics and Energy,Fujian Normal University,Fuzhou 350117,China
    3 Fujian Key Laboratory of Network Computing and Intelligent Information Processing,Fuzhou 350116,China
  • Received:2024-07-30 Revised:2024-11-29 Online:2025-09-15 Published:2025-09-11
  • About author:CHEN Jintao,born in 2000,postgra-duate.His main research interests include resource scheduling and reinforcement learning.
    LIN Bing,born in 1986,Ph.D,associate professor,postgraduate supervisor,is a member of CCF(No.83773M).His main research interests include cloud computing technology and computationalintelligence.
  • Supported by:
    Natural Science Foundation of China(62072108), University-Industry Cooperation of Fujian Province(2022H6024,2021H6026),Fujian University Physics Union(FJPHYS-2022-B02),Special Funds for Promoting High-quality Development of Marine and Fishery Industries in Fujian Province(FJHYF-ZH-2023-02) and Fujian Key Technological Innovation and Industrialization Projects(2024XQ004).

PDF (PC)

12

摘要： 光储充电站运营收益的提升,能够使充电站运营商加大对光储充电站基础设施的投资和部署,从而缓解日益增长的电动汽车渗透到电网时带来的负荷压力。针对光储充电站的运营收益提升问题,提出了一种基于多智能体深度强化学习的动态定价及能源调度策略,旨在提高完全合作关系下光储充电站的整体运营收益。首先,以最大化所有光储充电站的总运营收益为目标,将在单个光储充电站运营商下的多个光储充电站和电动汽车建模成马尔可夫博弈模型;其次,采用多智能体双延迟确定性策略梯度算法进行模型求解,通过制定充电服务价格和储能系统的充放电策略,以达到总运营收益最大化的目标,并通过余弦退火方法对算法学习率进行调整,提升该算法的收敛速率和收敛阈值;最后,为防止完全合作关系下多站可能出现的价格垄断问题,引入反需求函数对充电服务价格进行约束。实验结果表明,所提策略和对比方法相比,提高了4.17%~66.67%的充电站运营收益,且所用的反需求函数能够有效预防多站的价格垄断问题。

关键词: 多智能体深度强化学习, 光储充电站, 能源调度, 动态定价, 反需求函数

Abstract: The improvement in the operational profits of photovoltaic storage charging stations(PSCSs) can enable charging station operators to increase their investment and deployment of PSCSs infrastructure,thereby alleviating the load pressure on the grid caused by the growing penetration of electric vehicles(EVs).To address the issue of improving PSCSs operational profits,a dynamic pricing and energy scheduling strategy based on multi-agent deep reinforcement learning(MADRL) is proposed to enhance the overall operational profits of PSCSs under a fully cooperative relationship.Firstly,aiming to maximize the total operational profits of all PSCSs,multiple PSCSs and EVs under a single PSCS operator are modeled as a Markov game.Secondly,the multi-agent twin delayed deep deterministic policy gradient(MATD3) algorithm is used to solve the model,setting the selling price of charging services and the charging and discharging strategies of the energy storage system(ESS) to achieve profit maximization.The cosine annealing method is employed to adjust the learning rate of the algorithm,improving its convergence rate and threshold.Finally,to prevent potential price monopolies under a fully cooperative relationship among multiple stations,an inverse demand function is introduced to constrain the selling price of charging services.Experimental results show that the proposed strategy improves the operational profits of charging stations by 4.17% to 66.67% compared to benchmark methods,and using the inverse demand function effectively prevents price monopolies among multiple stations.

Key words: Multi-agent deep reinforcement learning, Photovoltaic storage charging station, Energy scheduling, Dynamic pricing, Inverse demand function

中图分类号: 

  • TP391
[1]FANG Q.Energy storage operation strategy of optical storage charging station based on PPO algorithm[J].Electric Engineering,2024(2):97-100.
[2]LIU Y X,ZHANG S,GUO L,et al.A coordinated optimalscheduling method of distribution grid and photovoltaic storage charging station taking into account electric energy-standby coupling[J].Power System Technology,2024,48(8):3175-3185.
[3]LEE S,CHOI D H.Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations:A privacy-preserving deep reinforcement learning approach[J].Applied Energy,2021,304:117754.
[4]AFFOLABI L,SHAHIDEHPOUR M,GAN W,et al.Optimal transactive energy trading of electric vehicle charging stations with on-site PV generation in constrained power distribution networks[J].IEEE Transactions on Smart Grid,2021,13(2):1427-1440.
[5]WANG R S,CHEN Z,XING Q,et al.A Modified Rainbow-Based Deep Reinforcement Learning Method for Optimal Scheduling of Charging Station[J].Sustainability,2022,14(3):1884.
[6]ZHANG S,LIU J J,SU Y T.Research on real-time controlstrategy of optical storage charging station based on PSO-DDPG algorithm[J].Mechanical and Electrical Information,2023(17):5-9.
[7]YANG X Y,CUI T X,WANG H R,et al.Multiagent Deep Reinforcement Learning for Electric Vehicle Fast Charging Station Pricing Game in Electricity-Transportation Nexus[J].IEEE Transactions on Industrial Informatics,2024 20(4):6345-6355.
[8]QIAN T,SHAO C,LI X,et al.Multi-agent deep reinforcement learning method for EV charging station game[J].IEEE Transactions on Power Systems,2021,37(3):1682-1694.
[9]ACKERMANN J,GABLER V,OSA T,et al.Reducing overestimation bias in multi-agent domains using double centralized critics[J].arXiv:1910.01465,2019.
[10]HUANG Z Q,LIN B,LU Y,et al.A charging station siting andcapacity determination method for multi-objective optimization[J].Journal of Fujian Normal University(Natural Science Edition),2024,40(2):23-35.
[11]QUE H K,FENG X F,GUO W C,et al.Charging station layout model based on fuzzy bi-objective planning[J].Computer Science,2022,39(3):751-757.
[12]WANG S,BI S,ZHANG Y A.Reinforcement learning for real-time pricing and scheduling control in EV charging stations[J].IEEE Transactions on Industrial Informatics,2019,17(2):849-859.
[13]XU H,WU Q,WEN J,et al.Joint bidding and pricing for electricity retailers based on multi-task deep reinforcement learning[J].International Journal of Electrical Power & Energy Systems,2022,138:107897.
[14]SHIN M J,CHOI D H,KIM J.Cooperative management forPV/ESS-enabled electric vehicle charging stations:A multiagent deep reinforcement learning approach[J].IEEE Transactions on Industrial Informatics,2019,16(5):3493-3503.
[15]KABIR M E,ASSIC C,TUSHAR M H K,et al.Optimal scheduling of EV charging at a solar power-based charging station[J].IEEE Systems Journal,2020,14(3):4221-4231.
[16]ZHOU X Y,QIAN L P,HUANG Y P,et al.An Optimization Method for Electric Vehicle Charging Scheduling Based on Ant Colony Algorithm[J].Computer Science.2020,47(11):280-285.
[17]WU W T,LIN Y,LIU R H,et al.Online EV charge scheduling based on time-of-use pricing and peak load minimization:Properties and efficient algorithms[J].IEEE Transactions on Intelligent Transportation Systems,2020,23(1):572-586.
[18]YU W W,LIU S L,CHEN Q G,et al.Multi-objective optimal scheduling of photovoltaic microgrids considering electric vehicle charging and demand-side response[J].Proceedings of the CSU-EPSA,2018,30(1):88-97.
[19]HAO Y.Energy optimization management of new energy charging station based on genetic algorithm[J].Telecom Power Technologies,2019,36(11):27-28,31.
[20]SU L,JIANG X C,WANG W,et al.Optimized energy management of microgrids taking into account electric vehicles and photovoltaic energy storage[J].Automation of Electric Power Systems,2015,39(9):164-171.
[21]LYU C,ZHAN S,ZHANG Y,et al.Synergistic two-stage optimization for multi-objective energy management strategy of integrated photovoltaic-storage charging stations[J].Journal of Energy Storage,2024,89:111665.
[22]MURIITHI G,CHOWDHURY S.Optimal energy management of a grid-tied solar pv-battery microgrid:A reinforcement learning approach[J].Energies,2021,14(9):2700.
[23]CUI L,WANG Q,QU H,et al.Dynamic pricing for fast charging stations with deep reinforcement learning[J].Applied Energy,2023,346:121334.
[24]LEE S,CHOI D H.Three-Stage Deep Reinforcement Learning for Privacy-and Safety-Aware Smart Electric Vehicle Charging Station Scheduling and Volt/VAR Control[J].IEEE Internet of Things Journal,2023,11(5):8578-8589.
[25]QIAN T,SHAO C,LI X,et al.Multi-agent deep reinforcement learning method for EV charging station game[J].IEEE Transactions on Power Systems,2021,37(3):1682-1694.
[26]VARIAN H R.Intermediate microeconomics with calculus:amodern approach[M].New York:W.W.NORTON & Company,2014:114-115.
[27]LOWE R,WU Y I,TAMAR A,et al.Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Advances in Neural Information Processing Systems.2017.
[28]ZHANG Y,YANG Q,AN D,et al.Multistep multiagent reinforcement learning for optimal energy schedule strategy of charging stations in smart grid[J].IEEE Transactions on Cyber-netics,2022,53(7):4292-4305.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发