Computer Science

Computer Science ›› 2024, Vol. 51 ›› Issue (11A): 231200147-7.doi: 10.11896/jsjkx.231200147

• Interdiscipline & Application • Previous Articles     Next Articles

Reinforcement Learning Algorithm for Charging/Discharging Control of Electric Vehicles Considering Battery Loss

LU Yue1, WANG Qiong2, LIU Shun 1, LI Qingtao1, LIU Yang1, WANG Hongbiao2   

  1. 1 State Grid Beijing Haidian Power Supply Company,Beijing 100080,China
    2 State Grid Beijing Electric Power Company,Beijing 100032,China
  • Online:2024-11-16 Published:2024-11-13
  • About author:LU Yue,born in 1989,postgraduate,engineer.His main research interests include power supply marketing and comprehensive energy.
    WANG Qiong,born in1983,doctoral candidate,engineer.Her main research interests include the development of software and hardware for charging stations and power-side infrastructure,alongside the integration of advanced security initiatives in vehicle networking platforms.
  • Supported by:
    State Grid Beijing Electric Power Company Technology Project:Research and Demonstration of V2G/S2G Vehicle Network Interaction and Intelligent Cluster Control Technology for Electric Vehicle Charging and Discharging Stations(520204220008).

PDF (PC)

194

Abstract: With the gradual increase in the number of electric vehicles,their integration has a significant impact on the load of the power grid.In this context,V2G/G2V technology is widely believed to play an important role in power grid management.Taking the charging and discharging control algorithm of electric vehicles as the research object,a deep reinforcement learning algorithm based on Soft Actor-Critic(SAC) is introduced.In terms of the dynamic change of load sequence in the power grid,the charging/discharging rate of different vehicles is controlled to maximize the benefits for users under different electricity prices.In addition,in order to address the issue of increased battery loss during the charging and discharging process,a battery loss prediction model based on physical hybrid neural network(PHNN) is introduced in the research.Meanwhile,the charging/discharging process is modeled as a Markov decision process.By integrating the PHNN model into the charging and discharging control of electric vehicles,a new reward function is constructed to accurately quantify the cost of battery loss.Based on the SAC algorithm,this reward function is used to learn the optimal charging and discharging strategy.Experimental results show that this algorithm can effectively regulate the charging and discharging behavior of vehicles,play a regulatory role in the power network,and reduce the loss of battery life during the charging and discharging process,further ensuring the economic interests of users.

Key words: Electric vehicle, Electric vehicle charging/discharging control, Deep reinforcement learning, Power grid regulation, Battery modeling

CLC Number: 

  • TP311
[1]ZHANG X,GAO F,GONG X,et al.Comparison of climatechange impact between power system of electric vehicles and internal combustion engine vehicles[C]//Advances in Energy and Environmental Materials:Proceedings of Chinese Materials Conference 2017 18th.2018:739-747.
[2]LOPES J A P,SOARES F J,ALMEIDA P M R.Integration of electric vehicles in the electric power system[C]//Proceedings of the IEEE.2010:168-183.
[3]KEMPTON W,TOMIĆ J.Vehicle-to-grid power implementa-tion:From stabilizing the grid to supporting large-scale renewable energy[J].Journal of Power Sources,2005,144(1):280-294.
[4]SOVACOOL B K,KESTER J,NOEL L,et al.Actors,business models,and innovation activity systems for vehicle-to-grid(V2G) technology:A comprehensive review[J].Renewable and Sustainable Energy Reviews,2020,131:109963.
[5]BALTHAZAR P,HARRI P,PRATER A,et al.Protecting yourpatients' interests in the era of big data,artificial intelligence,and predictive analytics[J].Journal of the American College of Radiology,2018,15(3):580-586.
[6]CAO Y,WANG H,LI D,et al.Smart online charging algorithm for electric vehicles via customized actor-critic learning[J].IEEE Internet of Things Journal,2021,9(1):684-694.
[7]WU D,ZENG H,LU C,et al.Two-stage energy management for office buildings with workplace EV charging and renewable energy[J].IEEE Transactions on Transportation Electrification,2017,3(1):225-237.
[8]SORTOMME E,EL-SHARKAWI M A.Optimal combined bidding of vehicle-to-grid ancillary services[J].IEEE Transactions on Smart Grid,2011,3(1):70-79.
[9]HAN S,HAN S,SEZAKI K.Development of an optimal vehicle-to-grid aggregator for frequency regulation[J].IEEE Transactions on Smart Grid,2010,1(1):65-72.
[10]HUANG Q,JIA Q S,QIU Z,et al.Matching EV charging load with uncertain wind:A simulation-based policy improvement approach[J].IEEE Transactions on Smart Grid,2015,6(3):1425-1433.
[11]CHUNG H M,MAHARJAN S,ZHANG Y,et al.Intelligentcharging management of electric vehicles considering dynamic user behavior and renewable energy:A stochastic game approach[J].IEEE Transactions on Intelligent Transportation Systems,2020,22(12):7760-7771.
[12]ALSABBAGH A,WU B,MA C.Distributed electric vehiclescharging management considering time anxiety and customer behaviors[J/OL].IEEE Transactions on Industrial Informatics,2020,17(4):2422-2431.https://www.eia.gov/conference/2008/conf_pdfs/Tuesday/Hogan.pdf.
[13]WAN Z,LI H,HE H,et al.Model-free real-time EV charging scheduling based on deep reinforcement learning[J].IEEE Transactions on Smart Grid,2018,10(5):5246-5257.
[14]ZHANG F,YANG Q,AN D.CDDPG:A deep-reinforcement-learning-based approach for electric vehicle charging control[J].IEEE Internet of Things Journal,2020,8(5):3075-3087.
[15]LI H,WAN Z,HE H.Constrained EV charging schedulingbased on safe deep reinforcement learning[J].IEEE Transactions on Smart Grid,2019,11(3):2427-2439.
[16]YAN L,CHEN X,ZHOU J,et al.Deep reinforcement learning for continuous electric vehicles charging control with dynamic user behaviors[J].IEEE Transactions on Smart Grid,2021,12(6):5124-5134.
[17]ARORA P,WHITE R E,DOYLE M.Capacity fade mechanisms and side reactions in lithium-ion batteries[J].Journal of the Electrochemical Society,1998,145(10):3647.
[18]SONG Y,PENG Y,LIU D.Model-based health diagnosis forlithium-ion battery pack in space applications[J].IEEE Transactions on Industrial Electronics,2020,68(12):12375-12384.
[19]HU X,YUAN H,ZOU C,et al.Co-estimation of state of charge and state of health for lithium-ion batteries based on fractional-order calculus[J].IEEE Transactions on Vehicular Technology,2018,67(11):10319-10329.
[20]LI Y,WEI Z,XIONG B,et al.Adaptive ensemble-based electrochemical-thermal degradation state estimation of lithium-ion batteries[J].IEEE Transactions on Industrial Electronics,2021,69(7):6984-6996.
[21]FU Y,XU J,SHI M,et al.A fast impedance calculation-based battery state-of-health estimation method[J].IEEE Transactions on Industrial Electronics,2021,69(7):7019-7028.
[22]ZHANG Y,LIU Y,WANG J,et al.State-of-health estimation for lithium-ion batteries by combining model-based incremental capacity analysis with support vector regression[J].Energy,2022,239:121986.
[23]REN L,DONG J,WANG X,et al.A data-driven auto-CNN-LSTM prediction model for lithium-ion battery remaining useful life[J].IEEE Transactions on Industrial Informatics,2020,17(5):3478-3487.
[24]QIN Y,YUEN C,YIN X,et al.A Transferable Multistage ModelWith Cycling Discrepancy Learning for Lithium-Ion Battery State of Health Estimation[J].IEEE Transactions on Industrial Informatics,2022,19(2):1933-1946.
[25]XU P,WANG C,YE J,et al.State-of-Charge Estimation and Health Prognosis for Lithium-Ion Batteries Based on Temperature-Compensated Bi-LSTM Network and Integrated Attention Mechanism[J].IEEE Transactions on Industrial Electronics,2023,66(10):8773-8783.
[26]FAN Y,XIAO F,LI C,et al.A novel deep learning frameworkfor state of health estimation of lithium-ion battery[J].Journal of Energy Storage,2020,32:101741.
[27]DUBARRY M,QIN N,BROOKER P.Calendar aging of commercial Li-ion cells of different chemistries-A review[J].Current Opinion in Electrochemistry,2018,9:106-113.
[28]LI K,ZHOU P,LU Y,et al.Battery life estimation based oncloud data for electric vehicles[J].Journal of Power Sources,2020,468:228192.
[29]WANG J,LIU P,HICKS-GARNER J,et al.Cycle-life model for graphite-LiFePO4 cells[J].Journal of Power Sources,2011,196(8):3942-3948.
[30]NING G,HARAN B,POPOV B N.Capacity fade study of lithium-ion batteries cycled at high discharge rates[J].Journal of Power Sources,2003,117(1/2):160-169.
[31]ISO New England.Real-time maps and charts[EB/OL].ht-tps://www.iso-ne.com/isoexpress/.
[32]SAHA,B.AND GOEBEL,K.Moffett Field,CA,USA:NASA AmesPrognostics Data Repository[EB/OL].http://ti.arc.nasa.gov/project/prognostic-data-repository.
[33]SCHULMAN J,WOLSKI F,DHARIWAL P,et al.Proximalpolicy optimization algorithms[J].arXiv:1707.06347,2017.
[1] WANG Tianjiu, LIU Quan, WU Lan. Offline Reinforcement Learning Algorithm for Conservative Q-learning Based on Uncertainty Weight [J]. Computer Science, 2024, 51(9): 265-272.
[2] ZHOU Wenhui, PENG Qinghua, XIE Lei. Study on Adaptive Cloud-Edge Collaborative Scheduling Methods for Multi-object State Perception [J]. Computer Science, 2024, 51(9): 319-330.
[3] GAO Yuzhao, NIE Yiming. Survey of Multi-agent Deep Reinforcement Learning Based on Value Function Factorization [J]. Computer Science, 2024, 51(6A): 230300170-9.
[4] WANG Shuanqi, ZHAO Jianxin, LIU Chi, WU Wei, LIU Zhao. Fuzz Testing Method of Binary Code Based on Deep Reinforcement Learning [J]. Computer Science, 2024, 51(6A): 230800078-7.
[5] LI Danyang, WU Liangji, LIU Hui, JIANG Jingqing. Deep Reinforcement Learning Based Thermal Awareness Energy Consumption OptimizationMethod for Data Centers [J]. Computer Science, 2024, 51(6A): 230500109-8.
[6] YANG Xiuwen, CUI Yunhe, QIAN Qing, GUO Chun, SHEN Guowei. COURIER:Edge Computing Task Scheduling and Offloading Method Based on Non-preemptivePriorities Queuing and Prioritized Experience Replay DRL [J]. Computer Science, 2024, 51(5): 293-305.
[7] CAO Yongsheng, LIU Yang, WANG Yongquan, XIA Tian. Online Electric Vehicle Charging Algorithm Based on Carbon Peak Constraint [J]. Computer Science, 2024, 51(3): 265-270.
[8] LI Junwei, LIU Quan, XU Yapeng. Option-Critic Algorithm Based on Mutual Information Optimization [J]. Computer Science, 2024, 51(2): 252-258.
[9] SHI Dianxi, PENG Yingxuan, YANG Huanhuan, OUYANG Qianying, ZHANG Yuhui, HAO Feng. DQN-based Multi-agent Motion Planning Method with Deep Reinforcement Learning [J]. Computer Science, 2024, 51(2): 268-277.
[10] ZHAO Xiaoyan, ZHAO Bin, ZHANG Junna, YUAN Peiyan. Study on Cache-oriented Dynamic Collaborative Task Migration Technology [J]. Computer Science, 2024, 51(2): 300-310.
[11] AN Yang, WANG Xiuqing, ZHAO Minghua. Mobile Robots' Path Planning Method Based on Policy Fusion and Spiking Deep ReinforcementLearning [J]. Computer Science, 2024, 51(11A): 240100211-11.
[12] TANG Jianing, LI Chengyang, ZHOU Sida, MA Mengxing, SHI Yang. Autonomous Exploration Methods for Unmanned Aerial Vehicles Based on Deep ReinforcementLearning [J]. Computer Science, 2024, 51(11A): 231100139-6.
[13] GU Wei, DUAN Jing, ZHANG Dong, HAO Xiaowei, XUE Honglin, AN Yi , DUAN Jie. Prediction of Spatial and Temporal Distribution of Electric Vehicle Charging Loads Based on Joint Data and Modeling Drive [J]. Computer Science, 2024, 51(11A): 231100110-6.
[14] CHEN Juan, WANG Yang, WU Zongling, CHEN Peng, ZHANG Fengchun , HAO Junfeng. Cloud-Edge Collaborative Task Transfer and Resource Reallocation Optimization Based on Deep Reinforcement Learning [J]. Computer Science, 2024, 51(11A): 231100170-10.
[15] YANG Haolin, LIU Quan. Advantage Weighted Double Actors-Critics Algorithm Based on Key-Minor Architecture for Policy Distillation [J]. Computer Science, 2024, 51(11): 81-94.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.