大语言模型赋能强化学习智能体研究综述

doi:10.11896/jsjkx.250800110

Abstract

Abstract: With the rapid development of the capabilities of large language models(LLMs),the technology of LLM-empowered reinforcement learning(RL),which leverages the powerful abilities of LLMs in sample processing,reasoning generation,and task planning,has emerged as a new hotspot in the field of machine learning.This technology provides new ideas for addressing the challenges faced by RL and enhancing its performance.Firstly,this paper systematically elaborates on the optimization mechanisms of LLMs on the basic elements of RL from four perspectives:task decomposition and planning,state representation and enhancement,reward design and shaping,and policy generation and optimization.It reveals how to integrate LLMs into RL to improve its performance and analyzes the principles and architectures of the above optimization mechanisms and integration methods based on the limitations of RL and the capabilities of LLMs.Subsequently,the applications of LLM-assisted RL in the fields of game AI,robotics,and autonomous driving are introduced.Finally,the main challenges currently faced are summarized,and future research directions are prospected.

Key words: Artificial intelligence, Reinforcement learning agent, Large language model, Large language model-empowered reinforcement learning

CLC Number:

TP181

ZHOU Xin, CAO Lei, CHEN Xiliang, CAO Yuqing. Review of Research on Large Language Model-empowered Reinforcement Learning Agents[J].Computer Science, 2026, 53(10): 254-267.

References

[1] SUTTON R S,BARTO A G.Reinforcement Learning:An In-troduction[M].Cambridge,USA:MIT Press,1998.
[2] PÉREZ-DATTARI R,CELEMIN C,RUIZ-DEL-SOLAR J,et al.Continuous control for high-dimensional state spaces:An interactive learning approach[C]//2019 International Conference on Robotics and Automation(ICRA).IEEE,2019:7611-7617.
[3] MNIH V,KAVUKCUOGLU K,SILVER D,et al.Playing atari with deep reinforcement learning[J].arXiv:1312.5602,2013.
[4] MNIH V,KAVUKCUOGLU K,SILVER D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529-533.
[5] MNIH V,BADIA A P,MIRZA M,et al.Asynchronous methods for deep reinforcement learning[C]//International Conference on Machine Learning.PMLR,2016:1928-1937.
[6] SCHULMAN J,WOLSKI F,DHARIWAL P,et al.Proximalpolicy optimization algorithms[J].arXiv:1707.06347,2017.
[7] TASRIN T,AL NAHIAN M S,PERERA H,et al.Influencing reinforcement learning through natural language guidance [C]//Proceedings of the 34th International Florida Artificial Intelligence Research Society Conference(FLAIRS-34).AAAI Press,2021.
[8] YAO S,ZHAO J,YU D,et al.React:Synergizing reasoning and acting in language models[C]//International Conference on Learning Representations(ICLR).2023.
[9] VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[C]//Proceedings of the 31st International Confe-rence on Neural Information Processing Systems.2017:6000-6010.
[10] LU J,BATRA D,PARIKH D,et al.Vilbert:Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[C]//NeurIPS.2019.
[11] GUO D Y,YANG D,ZHANG H R,et al.DeepSeek-R1:In-centivizing Reasoning Capability in LLMs via Reinforcement Learning[J].Nature,2025,638:1-9.
[12] RODRIGUEZ-SANCHEZ R,SPIEGEL B A,WANG J,et al.RLang:a declarative language for describing partial world knowledge to reinforcement learning agents[C]//International Conference on Machine Learning.PMLR,2023:29161-29178.
[13] FENG X,LIU B,SONG Y,et al.Natural language reinforce-ment learning[J].arXiv:2411.14251,2024.
[14] BROWN T,MANN B,RYDER N,et al.Language models are few-shot learners[C]//Advances in Neural Information Processing Systems 33.2020:1877-1901.
[15] WEI J,WANG X,SCHUURMANS D,et al.Chain-of-thoughtprompting elicits reasoning in large language models[J].Advances inNeural Information Processing Systems,2022,35:24824-24837.
[16] YAO S,YU D,ZHAO J,et al.Tree of thoughts:Deliberateproblem solving with large language models[J].Advances inNeural Information Processing Systems,2023,36:11809-11822.
[17] BESTA M,BLACH N,KUBICEK A,et al.Graph of thoughts:Solving elaborate problems with large language models[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2024:17682-17690.
[18] XU F,LIN Q,HAN J,et al.Are large language models really good logical reasoners? a comprehensive evaluation and beyond[J].IEEE Transactions on Knowledge and Data Engineering,2025,37(4):1620-1634.
[19] LI J,LI D,SAVARESE S,et al.BLIP-2:Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models[C]//International Conference on Machine Learning(ICML).PMLR,2023:19730-19742.
[20] RADFORD A,KIM J W,HALLACY C,et al.Learning Transferable Visual Models from Natural Language Supervision[C]//International Conference on Machine Learning(ICML).PMLR,2021:8748-8763.
[21] YAN X,SONG Y,FENG X,et al.Efficient reinforcement lear-ning with large language model priors[J].arXiv:2410.07927,2024.
[22] DALAL M,CHIRUVOLU T,CHAPLOT D,et al.Plan-seq-learn:Language model guided rl for solving long horizon robo-tics tasks[J].arXiv:2405.01534,2024.
[23] DU Y,WATKINS O,WANG Z,et al.Guiding pretraining in reinforcement learning with large language models[C]//International Conference on Machine Learning.PMLR,2023:8657-8677.
[24] QU Y,WANG B,JIANG Y,et al.Choices are more important than efforts:Llm enables efficient multi-agent exploration[J].arXiv:2410.02511,2024.
[25] HASLUM P,LIPOVETZKY N,MAGAZZENI D,et al.An introduction to the planning domain definition language[M].Morgan & Claypool Publishers,2019.
[26] SHINN N,CASSANO F,GOPINATH A,et al.Reflexion:Language agents with verbal reinforcement learning[C]//Advances in Neural Information Processing Systems(NeurIPS).2023:8634-8652.
[27] CHENTANEZ N,BARTO A,SINGH S.Intrinsically motivated reinforcement learning[C]//NIPS.2004.
[28] BHAMBRI S,BHATTACHARJEE A,KALWAR D,et al.Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learning[J].arXiv:2405.15194,2024.
[29] HUANG W,ABBEEL P,PATHAK D,et al.Inner Monologue:Embodied Reasoning through Planning with Language Models[C]//Proceedings of the Conference on Robot Learning(CoRL).2022.
[30] DASGUPTA I,KAESER-CHEN C,MARINO K,et al.Collaborating with language models for embodied reasoning[J].arXiv:2302.00763,2023.
[31] YUAN H,ZHANG C,WANG H,et al.Plan4mc:Skill rein-forcement learning and planning for open-world minecraft tasks [J].arXiv:2303.16563,2023.
[32] SHEK C L,TOKEKAR P.Option Discovery Using LLM-guidedSemantic Hierarchical Reinforcement Learning[J].arXiv:2503.19007,2025.
[33] BROHAN A,CHEBOTAR Y,FINN C,et al.Do as i can,not as i say:Grounding language in robotic affordances[C]//Confe-rence on Robot Learning.PMLR,2023:287-318.
[34] NOTTINGHAM K,AMMANABROLU P,SUHR A,et al.Do embodied agents dream of pixelated sheep:Embodied decision making using language guided world modelling[C]//International Conference on Machine Learning.PMLR,2023:26311-26325.
[35] BERTSEKAS D P.Feature-based aggregation and deep rein-forcement learning:A survey and some new implementations[J].IEEE/CAA Journal of Automatica Sinica,2018,6(1):1-31.
[36] FALZARI M.Autoencoder-based Deep Reinforcement Learning for ground-level walking of Human Musculoskeletal models[D].Groningen Provincie:University of Groningen,2022.
[37] LUKETINA J,NARDELLI N,FARQUHAR G,et al.A survey of reinforcement learning informed by natural language [C]//Proceedings of the Twenty-Eighth International Joint Confe-rence on Artificial Intelligence(IJCAI).2019:6309-6317.
[38] BASAVATIA S,MURUGESAN K,RATNAKAR S.Starling:Self-supervised training of text-based reinforcement learning agent with large language models [C]//Findings of the Association for Computational Linguistics:ACL 2024.Bangkok:Association for Computational Linguistics,2024:15804-15819.
[39] REN A Z,GOVIL B,YANG T Y,et al.Leveraging language for accelerated learning of tool manipulation[C]//Conference on Robot Learning.PMLR,2023:1531-1541.
[40] ROCAMONDE J,MONTESINOS V,NAVA E,et al.Vision-language models are zero-shot reward models for reinforcement learning [C]//Proceedings of the 37th Conference on Neural Information Processing Systems(NeurIPS 2023).New Orleans,USA:Curran Associates,Inc.,2023.
[41] ZHU J,WU F,ZHAO J.An overview of the action space fordeep reinforcement learning[C]//Proceedings of the 2021 4th International Conference on Algorithms,Computing and Artificial Intelligence.2021:1-10.
[42] JEONG D P,LIPTON Z C,RAVIKUMAR P.Llm-select:Feature selection with large language models[J].arXiv:2407.02694,2024.
[43] ASADI K,MISRA D,LITTMAN M.Lipschitz continuity inmodel-based reinforcement learning[C]//International Confe-rence on Machine Learning.PMLR,2018:264-273.
[44] WANG B,QU Y,JIANG Y,et al.LLM-Empowered State Representation for Reinforcement Learning [C]//Proceedings of the 41st International Conference on Machine Learning.Vienna,Austria:PMLR,2024:51348-51375.
[45] WEI Y,SHAN X,MIAO R,et al.LERO:LLM-driven Evolutionary framework with Hybrid Rewards and Enhanced Observation for Multi-Agent Reinforcement Learning[C]//International Conference on Intelligent Computing.Singapore:Springer Nature Singapore,2025:15-26.
[46] SHEK C L,WU X,SUTTLE W A,et al.Lancar:Leveraging language for context-aware robot locomotion in unstructured environments[C]//2024 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE,2024:9612-9619.
[47] IBRAHIM S,MOSTAFA M,JNADI A,et al.Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications[J].IEEE Access,2024,12:175473-17550.
[48] ESCHMANN J.Reward function design in reinforcement lear-ning[J].Reinforcement learning algorithms:Analysis and Applications,2021,883:25-33.
[49] WU Z,HU Y,SHI W,et al.Fine-grained human feedback gives better rewards for language model training[J].Advances in Neural Information Processing Systems,2023,36:59008-59033.
[50] TURCATO N,IOVINO M,SYNODINOS A,et al.Towards autonomous reinforcement learning for real-world robotic manipulation with large language models[J].arXiv:2503.04280,2025.
[51] XU P,HUANG K Q.An efficient reinforcement learning method based on large language model[J].Journal of Graphics,2024,45(6):1165-1177.
[52] PARK S G,KIM H B,LEE Y J,et al.TARG:Tree of Action-reward Generation With Large Language Model for Cabinet Opening Using Manipulator[J].International Journal of Control,Automation and Systems,2025,23(2):449-458.
[53] LI H,YANG X,WANG Z,et al.Auto mc-reward:Automated dense reward design with large language models for minecraft[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2024:16426-16435.
[54] SUN S,LIU R,LYU J,et al.A large language model-driven reward design framework via dynamic feedback for reinforcement learning[J].Knowledge-Based Systems,2025,326:114065.
[55] MA Y J,LIANG W,WANG G,et al.Eureka:Human-level reward design via coding large language models [C]//The Twelfth International Conference on Learning Representations(ICLR 2024).Vienna,Austria:OpenReview.net,2024.
[56] MASADOME S,HARADA T.Reward design using large language models for natural language explanation of reinforcement learning agent actions[J].IEEJ Transactions on Electrical and Electronic Engineering,2025,20(8):1203-1211.
[57] NAZIR M S,BANERJEE C.Zero-Shot LLMs in Human-in-the-Loop RL:Replacing Human Feedback for Reward Shaping[J].arXiv:2503.22723,2025.
[58] SYGKOUNAS A,HAZRA R,PERSSON A,et al.REvolve:Reward Evolution with Large Language Models using Human Feedback[C]//Greeks in AI Symposium 2025.2025.
[59] VERMA S,BOEHMER N,KONG L,TAMBE M.Balancingact:Prioritization strategies for LLM-designed restless bandit rewards [C]//Proceedings of the 40th Conference on Uncertainty in Artificial Intelligence(UAI 2024).Rotterdam,Netherlands:AUAI Press,2024.
[60] YANG R,YAN J P,LI X.Survey of sparse reward algorithms in reinforcement learning_theory and experiment[J].CAAI Transactions on Intelligent Systems,2020,15(5):888-899.
[61] MIKOLOV T,SUTSKEVER I,CHEN K,et al.Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems.Lake Tahoe,Nevada,USA:Curran Associates Inc.,2013:3111-3119.
[62] CHU K,ZHAO X,WEBER C,et al.Accelerating reinforcement learning of robotic manipulations via feedback from large language models[J].arXiv:2311.02379,2023.
[63] PLACE A.Adaptive reinforcement learning with llm-augmented reward functions[J].Authorea Preprints,2023.
[64] CAO M,SHU L,YU L,et al.Beyond sparse rewards:Enhancing reinforcement learning with language model critique in text generation [C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing(EMNLP 2024).Miami,Florida,USA:Association for Computational Linguistics,2024:9119-9138.
[65] SUTTON R S,MODAYIL J,DELP M,et al.Horde:A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction[C]//Proceedings of the International Conference on Autonomous Agents and Multiagent Systems(AAMAS).2011.
[66] QU Y,JIANG Y,WANG B,et al.Latent reward:Llm-empowered credit assignment in episodic reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2025:20095-20103.
[67] NAGPAL K,DONG D,MEHR N.Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignment[C]//Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems(AAMAS '25).Richland,SC:International Foundation for Autonomous Agents and Multiagent Systems,2025:1501-1510.
[68] KADAMALA K,CHAMBERS D,BARRETT E.EnhancingHVAC control efficiency:a hybrid approach using Imitation and reinforcement learning[C]//Joint European Conference on Machine Learning and Knowledge Discovery in Databases.Cham:Springer Nature Switzerland,2024:256-270.
[69] ZHAO A,HUANG D,XU Q,et al.Expel:Llm agents are experiential learners[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2024:19632-19642.
[70] KHEIRANDISH A,XU D,FEKRI F.LLM-Augmented Sym-bolic Reinforcement Learning with Landmark-Based Task Decomposition [C]//ICASSP 2025－2025 IEEE International Conference on Acoustics,Speech and Signal Processing.New Delhi,India:IEEE,2025.
[71] MA R,LUIJKX J,AJANOVIĆZ,et al.Explorllm:Guiding exploration in reinforcement learning with large language models[C]//2025 IEEE International Conference on Robotics and Automation(ICRA).IEEE,2025:9011-9017.
[72] PENG H J,ZHANG W Y,CHEN R H,et al.Large-language-model-driven training optimization for air-combat self-defense jamming strategies[J].Journal of Detection & Control,2025,47(2):71-81.
[73] KARINE K,MARLIN B M.Combining LLM decision and RL action selection to improve RL policy for adaptive interventions[J].arXiv:2501.06980,2025.
[74] ZHANG S,ZHENG S,KE S,et al.How can llm guide rl? a va-lue-based approach[J].arXiv:2402.16181,2024.
[75] WANG B,MCKEOWN K,YING R.DYSTIL:Dynamic Strategy Induction with Large Language Models for Reinforcement Learning[J].arXiv:2505.03209,2025.
[76] WANG X S,WANG R R,CHENG Y H. Safe reinforcement learning:A survey[J].Acta Automatica Sinica,2023,49(9):1813-1835.
[77] BRIJEN T ,ASHWIN B,SURAJ N,et al.Recovery rl:Safe reinforcement learning with learned recovery zones[J].IEEE Robotics and Automation Letters,2021,6(3):4915-4922.
[78] YANG T Y,HU M Y,CHOW Y,et al.Safe reinforcementlearning with natural language constraints[J].Advances in Neural Information Processing Systems,2021,34:13794-13808.
[79] GROSS D,SPIEKER H.Enhancing rl safety with counterfactual llm reasoning[C]//IFIP International Conference on Testing Software and Systems.Cham:Springer Nature Switzerland,2024:23-29.
[80] MA J,LIANG W,WANG H J,et al.Dreureka:Language model guided sim-to-real transfer[J].arXiv:2406.01967,2024.
[81] PETRONI F,ROCKTÄSCHEL T,RIEDEL S,et al.Language models as knowledge bases?[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Proces-sing.EMNLP,2019.
[82] PANG J C,YANG S H,LI K,et al.Kalm:Knowledgeable agents by offline reinforcement learning from large language model rollouts[J].Advances in Neural Information Processing Systems,2024,37:126620-126652.
[83] BHAMBR I,SIDDHAN T,AMRITA BHATTACHARJE E,et al.Efficient reinforcement learning via large language model-based search[C]//NeurIPS 2024 Workshop on Open-World Agents.2024.
[84] FRANÇOIS-LAVET V,HENDERSON P,ISLAM R,et al.An introduction to deep reinforcement learning[J].Foundations and Trends® in Machine Learning,2018,11(3/4):219-354.
[85] WANG J,WANG T,CAI W,et al.Boosting Efficient Reinforcement Learning for Vision-and-Language Navigation With Open-Sourced LLM[J].IEEE Robotics and Automation Letters,2025,10(1):612-619.
[86] CHEN Z,LENG B,LI Z,et al.HCRMP:A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving[J].arXiv:2505.15793,2025.
[87] WANG X,REN H,TAN G Z,et al.Autonomous Driving Decision-making Method Based on Cooperative Reinforcement Learning of Large Language Model[J].Journal of Transportation Systems Engineering and Information Technology,2025,25(4):137-146.
[88] XU C,LIU J,GUO Y,et al.Towards human-centric autono-mous driving:a fast-slow architecture integrating large language model guidance with reinforcement learning [J].arXiv:2505.06875,2025.
[89] XU Y,HONG W,ZHA J,et al.Scalable UAV Multi-Hop Networking via Multi-Agent Reinforcement Learning with Large Language Models[J].arXiv:2505.08448.2025.
[90] HUANG L,YU W,MA W,et al.A Survey on Hallucination in Large Language Models:Principles,Taxonomy,Challenges,and Open Questions[J].ACM Transactions on Information Systems,2023,43:1-55.
[91] LEWIS P,PEREZ E,PIKTUS A,et al.Retrieval-augmentedgeneration for knowledge-intensive nlp tasks[J].Advances inNeural Information Processing Systems,2020,33:9459-9474.
[92] HUANG W,LI H,QIN F,et al.MFA-NRM:A novel frame-work for multimodal fusion and semantic alignment in visual neural decoding[J].Information Fusion,2025,127(Part A):103717.
[93] KAPLAN J,MCCANDLISH S,HENIGHAN T,et al.Scaling laws for neural language models[J].arXiv:2001.08361,2020.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Review of Research on Large Language Model-empowered Reinforcement Learning Agents

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 15

Metrics

Comments

Recommended 0

[1]	HE Jiaojun, LI Xin. Review of Graph Learning Based on Large Language Models:Methods,Benchmarks and Advances [J]. Computer Science, 2026, 53(9): 1-15.
[2]	LI Luozheng, LI Lingbo, YUAN Quan. Survey of Chinese Datasets for LLM Safety Alignment:Landscape and Prospects [J]. Computer Science, 2026, 53(9): 16-23.
[3]	AN Ran, CHENG Kai, SHAO Tianhao, LI Changyuan, CHEN Yan. Framework,Techniques and Challenges of Task Planning Agent Based on Large Language Model [J]. Computer Science, 2026, 53(9): 24-37.
[4]	XIA Yi, ZHOU Gang, ZHANG Kaixiang, LAN Mingjing, LI Zhufeng, SU Benrong, HE Haofeng, FENG Shizhong. Survey on Collaborative-driven Knowledge Reasoning Technologies Integrating Large Language Models and Knowledge Graphs [J]. Computer Science, 2026, 53(9): 38-54.
[5]	ZHANG Rongjie, PANG Xiongwen, WANG Fengling. Review of Large Language Model-based Time Series Modeling via Fine-tuning and AgentArchitecture [J]. Computer Science, 2026, 53(9): 55-70.
[6]	CHEN Hai, FAN Changjun, SUN Boliang, CHENG Jie, KONG Junfeng, YUAN Yuchen. Order-based,LLM-augmented Kill-chain for Time-sensitive Target Engagement [J]. Computer Science, 2026, 53(9): 71-81.
[7]	GUO Yuyang, SHI Lei, LIU Huan, DONG Yixiang, LI Rui. Domain-adapted and Dynamically Retrieval-augmented Approach for Large-scale History Discipline Model Construction [J]. Computer Science, 2026, 53(9): 92-100.
[8]	LEI Kunhan, TAO Jiuyang, LI Chenyu, ZHONG Qi, YE Linfa. Method for Generating Operational Action Plans Based on Large Language Model Guided byChain-of-Thought [J]. Computer Science, 2026, 53(9): 101-109.
[9]	LI Zhennan, QIAN Jiayan, WANG Xinzhi, ZHANG Hui. Technology Risk Structure Recognition Based on Multi-granularity Semantic Dual Reflection [J]. Computer Science, 2026, 53(9): 395-404.
[10]	JIA Zishuo, ZHANG Jian’ge, HE Haofeng, FENG Shizhong, LIU Yilin. Survey on Mutually Augmenting Technologies and Applications of Large Models and KnowledgeGraphs [J]. Computer Science, 2026, 53(8): 219-228.
[11]	LIU Jing. Review of Music Artificial Intelligence Driven by Large Language Models [J]. Computer Science, 2026, 53(8): 229-244.
[12]	ZHANG Haoran, HAO Wenning, JIN Dawei, CHENG Kai, LIU Junyang. Agentic Retrieval Augmented Generation Framework Based on Retrieval Task Planning and Reflection Mechanism [J]. Computer Science, 2026, 53(8): 285-297.
[13]	ZHANG Zhonglin, XIA Hang. LSQ-RAG:Retrieval-enhanced Generation Framework Based on LLM-enhanced Ranker [J]. Computer Science, 2026, 53(8): 298-306.
[14]	FAN Ruxin, SUN Baicai, GONG Lina, YAO Xiangjuan, GONG Dunwei. Test Case Generation and Prioritization for Program Fault Diagnosis Based on Large LanguageModel [J]. Computer Science, 2026, 53(8): 365-374.
[15]	LI Xiaochao, YUAN Zisu, LI Qianmu, LIU Fan, CHE Xun. Survey on Code Representation Learning for Vulnerability Detection [J]. Computer Science, 2026, 53(8): 388-402.