计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (11A): 231200147-7.doi: 10.11896/jsjkx.231200147

• 交叉&应用 • 上一篇    下一篇

基于强化学习考虑电池损耗的电动汽车充放电控制算法

卢钺1, 王琼2, 刘顺1, 李清涛1, 刘洋1, 王洪彪2   

  1. 1 国网北京海淀供电公司 北京 100080
    2 国网北京市电力公司 北京 100032
  • 出版日期:2024-11-16 发布日期:2024-11-13
  • 通讯作者: 王琼(coycathy@163.com)
  • 作者简介:(339545765@qq.com)
  • 基金资助:
    国网北京市电力公司科技项目:电动汽车充放电站 V2G/S2G 车网互动及智慧集群调控技术研究及示范 (520204220008)

Reinforcement Learning Algorithm for Charging/Discharging Control of Electric Vehicles Considering Battery Loss

LU Yue1, WANG Qiong2, LIU Shun 1, LI Qingtao1, LIU Yang1, WANG Hongbiao2   

  1. 1 State Grid Beijing Haidian Power Supply Company,Beijing 100080,China
    2 State Grid Beijing Electric Power Company,Beijing 100032,China
  • Online:2024-11-16 Published:2024-11-13
  • About author:LU Yue,born in 1989,postgraduate,engineer.His main research interests include power supply marketing and comprehensive energy.
    WANG Qiong,born in1983,doctoral candidate,engineer.Her main research interests include the development of software and hardware for charging stations and power-side infrastructure,alongside the integration of advanced security initiatives in vehicle networking platforms.
  • Supported by:
    State Grid Beijing Electric Power Company Technology Project:Research and Demonstration of V2G/S2G Vehicle Network Interaction and Intelligent Cluster Control Technology for Electric Vehicle Charging and Discharging Stations(520204220008).

PDF (PC)

194

摘要： 随着电动汽车数量的逐步增加,其接入对电网的负荷带来显著影响。在这一背景下,V2G/G2V技术被广泛认为能在电网管理方面发挥重要作用。以电动汽车的充放电控制算法为研究对象,引入了一种基于软演员评论家(SAC)的深度强化学习算法,从而实现对电动汽车连续充放电行为的精细控制。研究着眼于解决电网中负荷时序动态变化的难题,通过调整不同车辆在不同电价条件下的充放电功率,最大程度地提升用户的经济效益。此外,为应对充放电过程中可能导致电池损耗加剧的问题,引入了基于物理混合神经网络(PHNN)的电池损耗预测模型。同时,通过将充放电过程建模为马尔可夫决策问题,并将PHNN模型融入电动汽车的充放电控制中,构建了一个全新的奖励函数,以精确量化电池损耗所带来的成本。基于SAC算法,该奖励函数用于学习最优的充放电策略。实验结果显示,该算法能够有效地调控车辆的充放电行为,发挥电力网络调控作用,同时在充放电过程中降低对电池寿命造成的损耗,进一步保障用户经济利益。

关键词: 电动汽车, 电动汽车充放电控制, 深度强化学习, 电网调控, 电池建模

Abstract: With the gradual increase in the number of electric vehicles,their integration has a significant impact on the load of the power grid.In this context,V2G/G2V technology is widely believed to play an important role in power grid management.Taking the charging and discharging control algorithm of electric vehicles as the research object,a deep reinforcement learning algorithm based on Soft Actor-Critic(SAC) is introduced.In terms of the dynamic change of load sequence in the power grid,the charging/discharging rate of different vehicles is controlled to maximize the benefits for users under different electricity prices.In addition,in order to address the issue of increased battery loss during the charging and discharging process,a battery loss prediction model based on physical hybrid neural network(PHNN) is introduced in the research.Meanwhile,the charging/discharging process is modeled as a Markov decision process.By integrating the PHNN model into the charging and discharging control of electric vehicles,a new reward function is constructed to accurately quantify the cost of battery loss.Based on the SAC algorithm,this reward function is used to learn the optimal charging and discharging strategy.Experimental results show that this algorithm can effectively regulate the charging and discharging behavior of vehicles,play a regulatory role in the power network,and reduce the loss of battery life during the charging and discharging process,further ensuring the economic interests of users.

Key words: Electric vehicle, Electric vehicle charging/discharging control, Deep reinforcement learning, Power grid regulation, Battery modeling

中图分类号: 

  • TP311
[1]ZHANG X,GAO F,GONG X,et al.Comparison of climatechange impact between power system of electric vehicles and internal combustion engine vehicles[C]//Advances in Energy and Environmental Materials:Proceedings of Chinese Materials Conference 2017 18th.2018:739-747.
[2]LOPES J A P,SOARES F J,ALMEIDA P M R.Integration of electric vehicles in the electric power system[C]//Proceedings of the IEEE.2010:168-183.
[3]KEMPTON W,TOMIĆ J.Vehicle-to-grid power implementa-tion:From stabilizing the grid to supporting large-scale renewable energy[J].Journal of Power Sources,2005,144(1):280-294.
[4]SOVACOOL B K,KESTER J,NOEL L,et al.Actors,business models,and innovation activity systems for vehicle-to-grid(V2G) technology:A comprehensive review[J].Renewable and Sustainable Energy Reviews,2020,131:109963.
[5]BALTHAZAR P,HARRI P,PRATER A,et al.Protecting yourpatients' interests in the era of big data,artificial intelligence,and predictive analytics[J].Journal of the American College of Radiology,2018,15(3):580-586.
[6]CAO Y,WANG H,LI D,et al.Smart online charging algorithm for electric vehicles via customized actor-critic learning[J].IEEE Internet of Things Journal,2021,9(1):684-694.
[7]WU D,ZENG H,LU C,et al.Two-stage energy management for office buildings with workplace EV charging and renewable energy[J].IEEE Transactions on Transportation Electrification,2017,3(1):225-237.
[8]SORTOMME E,EL-SHARKAWI M A.Optimal combined bidding of vehicle-to-grid ancillary services[J].IEEE Transactions on Smart Grid,2011,3(1):70-79.
[9]HAN S,HAN S,SEZAKI K.Development of an optimal vehicle-to-grid aggregator for frequency regulation[J].IEEE Transactions on Smart Grid,2010,1(1):65-72.
[10]HUANG Q,JIA Q S,QIU Z,et al.Matching EV charging load with uncertain wind:A simulation-based policy improvement approach[J].IEEE Transactions on Smart Grid,2015,6(3):1425-1433.
[11]CHUNG H M,MAHARJAN S,ZHANG Y,et al.Intelligentcharging management of electric vehicles considering dynamic user behavior and renewable energy:A stochastic game approach[J].IEEE Transactions on Intelligent Transportation Systems,2020,22(12):7760-7771.
[12]ALSABBAGH A,WU B,MA C.Distributed electric vehiclescharging management considering time anxiety and customer behaviors[J/OL].IEEE Transactions on Industrial Informatics,2020,17(4):2422-2431.https://www.eia.gov/conference/2008/conf_pdfs/Tuesday/Hogan.pdf.
[13]WAN Z,LI H,HE H,et al.Model-free real-time EV charging scheduling based on deep reinforcement learning[J].IEEE Transactions on Smart Grid,2018,10(5):5246-5257.
[14]ZHANG F,YANG Q,AN D.CDDPG:A deep-reinforcement-learning-based approach for electric vehicle charging control[J].IEEE Internet of Things Journal,2020,8(5):3075-3087.
[15]LI H,WAN Z,HE H.Constrained EV charging schedulingbased on safe deep reinforcement learning[J].IEEE Transactions on Smart Grid,2019,11(3):2427-2439.
[16]YAN L,CHEN X,ZHOU J,et al.Deep reinforcement learning for continuous electric vehicles charging control with dynamic user behaviors[J].IEEE Transactions on Smart Grid,2021,12(6):5124-5134.
[17]ARORA P,WHITE R E,DOYLE M.Capacity fade mechanisms and side reactions in lithium-ion batteries[J].Journal of the Electrochemical Society,1998,145(10):3647.
[18]SONG Y,PENG Y,LIU D.Model-based health diagnosis forlithium-ion battery pack in space applications[J].IEEE Transactions on Industrial Electronics,2020,68(12):12375-12384.
[19]HU X,YUAN H,ZOU C,et al.Co-estimation of state of charge and state of health for lithium-ion batteries based on fractional-order calculus[J].IEEE Transactions on Vehicular Technology,2018,67(11):10319-10329.
[20]LI Y,WEI Z,XIONG B,et al.Adaptive ensemble-based electrochemical-thermal degradation state estimation of lithium-ion batteries[J].IEEE Transactions on Industrial Electronics,2021,69(7):6984-6996.
[21]FU Y,XU J,SHI M,et al.A fast impedance calculation-based battery state-of-health estimation method[J].IEEE Transactions on Industrial Electronics,2021,69(7):7019-7028.
[22]ZHANG Y,LIU Y,WANG J,et al.State-of-health estimation for lithium-ion batteries by combining model-based incremental capacity analysis with support vector regression[J].Energy,2022,239:121986.
[23]REN L,DONG J,WANG X,et al.A data-driven auto-CNN-LSTM prediction model for lithium-ion battery remaining useful life[J].IEEE Transactions on Industrial Informatics,2020,17(5):3478-3487.
[24]QIN Y,YUEN C,YIN X,et al.A Transferable Multistage ModelWith Cycling Discrepancy Learning for Lithium-Ion Battery State of Health Estimation[J].IEEE Transactions on Industrial Informatics,2022,19(2):1933-1946.
[25]XU P,WANG C,YE J,et al.State-of-Charge Estimation and Health Prognosis for Lithium-Ion Batteries Based on Temperature-Compensated Bi-LSTM Network and Integrated Attention Mechanism[J].IEEE Transactions on Industrial Electronics,2023,66(10):8773-8783.
[26]FAN Y,XIAO F,LI C,et al.A novel deep learning frameworkfor state of health estimation of lithium-ion battery[J].Journal of Energy Storage,2020,32:101741.
[27]DUBARRY M,QIN N,BROOKER P.Calendar aging of commercial Li-ion cells of different chemistries-A review[J].Current Opinion in Electrochemistry,2018,9:106-113.
[28]LI K,ZHOU P,LU Y,et al.Battery life estimation based oncloud data for electric vehicles[J].Journal of Power Sources,2020,468:228192.
[29]WANG J,LIU P,HICKS-GARNER J,et al.Cycle-life model for graphite-LiFePO4 cells[J].Journal of Power Sources,2011,196(8):3942-3948.
[30]NING G,HARAN B,POPOV B N.Capacity fade study of lithium-ion batteries cycled at high discharge rates[J].Journal of Power Sources,2003,117(1/2):160-169.
[31]ISO New England.Real-time maps and charts[EB/OL].ht-tps://www.iso-ne.com/isoexpress/.
[32]SAHA,B.AND GOEBEL,K.Moffett Field,CA,USA:NASA AmesPrognostics Data Repository[EB/OL].http://ti.arc.nasa.gov/project/prognostic-data-repository.
[33]SCHULMAN J,WOLSKI F,DHARIWAL P,et al.Proximalpolicy optimization algorithms[J].arXiv:1707.06347,2017.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发