大语言模型赋能强化学习智能体研究综述

doi:10.11896/jsjkx.250800110

摘要/Abstract

摘要： 随着大语言模型能力的快速发展,利用大语言模型在样本处理、推理生成和任务规划等方面展现的强大能力赋能强化学习技术,正成为机器学习领域的新兴热点,为应对强化学习面临的挑战与提升其性能提供了新思路。首先,从任务分解与规划、状态表征及增强、奖励设计和塑形、策略生成与优化这4个角度系统阐述了大语言模型对强化学习基本要素的优化机制,揭示了如何将大语言模型整合到强化学习中以提高其性能,并通过分析强化学习的局限及大语言模型的能力,对上述优化机制及整合方法的原理和架构进行了解析;然后,介绍了其在游戏AI、机器人技术和自动驾驶等领域的应用情况;最后,总结了当前面临的主要挑战,并对未来探索领域进行了展望。

关键词: 人工智能, 强化学习智能体, 大语言模型, 大语言模型赋能强化学习

Abstract: With the rapid development of the capabilities of large language models(LLMs),the technology of LLM-empowered reinforcement learning(RL),which leverages the powerful abilities of LLMs in sample processing,reasoning generation,and task planning,has emerged as a new hotspot in the field of machine learning.This technology provides new ideas for addressing the challenges faced by RL and enhancing its performance.Firstly,this paper systematically elaborates on the optimization mechanisms of LLMs on the basic elements of RL from four perspectives:task decomposition and planning,state representation and enhancement,reward design and shaping,and policy generation and optimization.It reveals how to integrate LLMs into RL to improve its performance and analyzes the principles and architectures of the above optimization mechanisms and integration methods based on the limitations of RL and the capabilities of LLMs.Subsequently,the applications of LLM-assisted RL in the fields of game AI,robotics,and autonomous driving are introduced.Finally,the main challenges currently faced are summarized,and future research directions are prospected.

Key words: Artificial intelligence, Reinforcement learning agent, Large language model, Large language model-empowered reinforcement learning

中图分类号:

TP181

周鑫, 曹雷, 陈希亮, 曹育箐. 大语言模型赋能强化学习智能体研究综述[J]. 计算机科学, 2026, 53(10): 254-267. https://doi.org/10.11896/jsjkx.250800110

ZHOU Xin, CAO Lei, CHEN Xiliang, CAO Yuqing. Review of Research on Large Language Model-empowered Reinforcement Learning Agents[J]. Computer Science, 2026, 53(10): 254-267. https://doi.org/10.11896/jsjkx.250800110

参考文献

[1] SUTTON R S,BARTO A G.Reinforcement Learning:An In-troduction[M].Cambridge,USA:MIT Press,1998.
[2] PÉREZ-DATTARI R,CELEMIN C,RUIZ-DEL-SOLAR J,et al.Continuous control for high-dimensional state spaces:An interactive learning approach[C]//2019 International Conference on Robotics and Automation(ICRA).IEEE,2019:7611-7617.
[3] MNIH V,KAVUKCUOGLU K,SILVER D,et al.Playing atari with deep reinforcement learning[J].arXiv:1312.5602,2013.
[4] MNIH V,KAVUKCUOGLU K,SILVER D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529-533.
[5] MNIH V,BADIA A P,MIRZA M,et al.Asynchronous methods for deep reinforcement learning[C]//International Conference on Machine Learning.PMLR,2016:1928-1937.
[6] SCHULMAN J,WOLSKI F,DHARIWAL P,et al.Proximalpolicy optimization algorithms[J].arXiv:1707.06347,2017.
[7] TASRIN T,AL NAHIAN M S,PERERA H,et al.Influencing reinforcement learning through natural language guidance [C]//Proceedings of the 34th International Florida Artificial Intelligence Research Society Conference(FLAIRS-34).AAAI Press,2021.
[8] YAO S,ZHAO J,YU D,et al.React:Synergizing reasoning and acting in language models[C]//International Conference on Learning Representations(ICLR).2023.
[9] VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[C]//Proceedings of the 31st International Confe-rence on Neural Information Processing Systems.2017:6000-6010.
[10] LU J,BATRA D,PARIKH D,et al.Vilbert:Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[C]//NeurIPS.2019.
[11] GUO D Y,YANG D,ZHANG H R,et al.DeepSeek-R1:In-centivizing Reasoning Capability in LLMs via Reinforcement Learning[J].Nature,2025,638:1-9.
[12] RODRIGUEZ-SANCHEZ R,SPIEGEL B A,WANG J,et al.RLang:a declarative language for describing partial world knowledge to reinforcement learning agents[C]//International Conference on Machine Learning.PMLR,2023:29161-29178.
[13] FENG X,LIU B,SONG Y,et al.Natural language reinforce-ment learning[J].arXiv:2411.14251,2024.
[14] BROWN T,MANN B,RYDER N,et al.Language models are few-shot learners[C]//Advances in Neural Information Processing Systems 33.2020:1877-1901.
[15] WEI J,WANG X,SCHUURMANS D,et al.Chain-of-thoughtprompting elicits reasoning in large language models[J].Advances inNeural Information Processing Systems,2022,35:24824-24837.
[16] YAO S,YU D,ZHAO J,et al.Tree of thoughts:Deliberateproblem solving with large language models[J].Advances inNeural Information Processing Systems,2023,36:11809-11822.
[17] BESTA M,BLACH N,KUBICEK A,et al.Graph of thoughts:Solving elaborate problems with large language models[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2024:17682-17690.
[18] XU F,LIN Q,HAN J,et al.Are large language models really good logical reasoners? a comprehensive evaluation and beyond[J].IEEE Transactions on Knowledge and Data Engineering,2025,37(4):1620-1634.
[19] LI J,LI D,SAVARESE S,et al.BLIP-2:Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models[C]//International Conference on Machine Learning(ICML).PMLR,2023:19730-19742.
[20] RADFORD A,KIM J W,HALLACY C,et al.Learning Transferable Visual Models from Natural Language Supervision[C]//International Conference on Machine Learning(ICML).PMLR,2021:8748-8763.
[21] YAN X,SONG Y,FENG X,et al.Efficient reinforcement lear-ning with large language model priors[J].arXiv:2410.07927,2024.
[22] DALAL M,CHIRUVOLU T,CHAPLOT D,et al.Plan-seq-learn:Language model guided rl for solving long horizon robo-tics tasks[J].arXiv:2405.01534,2024.
[23] DU Y,WATKINS O,WANG Z,et al.Guiding pretraining in reinforcement learning with large language models[C]//International Conference on Machine Learning.PMLR,2023:8657-8677.
[24] QU Y,WANG B,JIANG Y,et al.Choices are more important than efforts:Llm enables efficient multi-agent exploration[J].arXiv:2410.02511,2024.
[25] HASLUM P,LIPOVETZKY N,MAGAZZENI D,et al.An introduction to the planning domain definition language[M].Morgan & Claypool Publishers,2019.
[26] SHINN N,CASSANO F,GOPINATH A,et al.Reflexion:Language agents with verbal reinforcement learning[C]//Advances in Neural Information Processing Systems(NeurIPS).2023:8634-8652.
[27] CHENTANEZ N,BARTO A,SINGH S.Intrinsically motivated reinforcement learning[C]//NIPS.2004.
[28] BHAMBRI S,BHATTACHARJEE A,KALWAR D,et al.Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learning[J].arXiv:2405.15194,2024.
[29] HUANG W,ABBEEL P,PATHAK D,et al.Inner Monologue:Embodied Reasoning through Planning with Language Models[C]//Proceedings of the Conference on Robot Learning(CoRL).2022.
[30] DASGUPTA I,KAESER-CHEN C,MARINO K,et al.Collaborating with language models for embodied reasoning[J].arXiv:2302.00763,2023.
[31] YUAN H,ZHANG C,WANG H,et al.Plan4mc:Skill rein-forcement learning and planning for open-world minecraft tasks [J].arXiv:2303.16563,2023.
[32] SHEK C L,TOKEKAR P.Option Discovery Using LLM-guidedSemantic Hierarchical Reinforcement Learning[J].arXiv:2503.19007,2025.
[33] BROHAN A,CHEBOTAR Y,FINN C,et al.Do as i can,not as i say:Grounding language in robotic affordances[C]//Confe-rence on Robot Learning.PMLR,2023:287-318.
[34] NOTTINGHAM K,AMMANABROLU P,SUHR A,et al.Do embodied agents dream of pixelated sheep:Embodied decision making using language guided world modelling[C]//International Conference on Machine Learning.PMLR,2023:26311-26325.
[35] BERTSEKAS D P.Feature-based aggregation and deep rein-forcement learning:A survey and some new implementations[J].IEEE/CAA Journal of Automatica Sinica,2018,6(1):1-31.
[36] FALZARI M.Autoencoder-based Deep Reinforcement Learning for ground-level walking of Human Musculoskeletal models[D].Groningen Provincie:University of Groningen,2022.
[37] LUKETINA J,NARDELLI N,FARQUHAR G,et al.A survey of reinforcement learning informed by natural language [C]//Proceedings of the Twenty-Eighth International Joint Confe-rence on Artificial Intelligence(IJCAI).2019:6309-6317.
[38] BASAVATIA S,MURUGESAN K,RATNAKAR S.Starling:Self-supervised training of text-based reinforcement learning agent with large language models [C]//Findings of the Association for Computational Linguistics:ACL 2024.Bangkok:Association for Computational Linguistics,2024:15804-15819.
[39] REN A Z,GOVIL B,YANG T Y,et al.Leveraging language for accelerated learning of tool manipulation[C]//Conference on Robot Learning.PMLR,2023:1531-1541.
[40] ROCAMONDE J,MONTESINOS V,NAVA E,et al.Vision-language models are zero-shot reward models for reinforcement learning [C]//Proceedings of the 37th Conference on Neural Information Processing Systems(NeurIPS 2023).New Orleans,USA:Curran Associates,Inc.,2023.
[41] ZHU J,WU F,ZHAO J.An overview of the action space fordeep reinforcement learning[C]//Proceedings of the 2021 4th International Conference on Algorithms,Computing and Artificial Intelligence.2021:1-10.
[42] JEONG D P,LIPTON Z C,RAVIKUMAR P.Llm-select:Feature selection with large language models[J].arXiv:2407.02694,2024.
[43] ASADI K,MISRA D,LITTMAN M.Lipschitz continuity inmodel-based reinforcement learning[C]//International Confe-rence on Machine Learning.PMLR,2018:264-273.
[44] WANG B,QU Y,JIANG Y,et al.LLM-Empowered State Representation for Reinforcement Learning [C]//Proceedings of the 41st International Conference on Machine Learning.Vienna,Austria:PMLR,2024:51348-51375.
[45] WEI Y,SHAN X,MIAO R,et al.LERO:LLM-driven Evolutionary framework with Hybrid Rewards and Enhanced Observation for Multi-Agent Reinforcement Learning[C]//International Conference on Intelligent Computing.Singapore:Springer Nature Singapore,2025:15-26.
[46] SHEK C L,WU X,SUTTLE W A,et al.Lancar:Leveraging language for context-aware robot locomotion in unstructured environments[C]//2024 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE,2024:9612-9619.
[47] IBRAHIM S,MOSTAFA M,JNADI A,et al.Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications[J].IEEE Access,2024,12:175473-17550.
[48] ESCHMANN J.Reward function design in reinforcement lear-ning[J].Reinforcement learning algorithms:Analysis and Applications,2021,883:25-33.
[49] WU Z,HU Y,SHI W,et al.Fine-grained human feedback gives better rewards for language model training[J].Advances in Neural Information Processing Systems,2023,36:59008-59033.
[50] TURCATO N,IOVINO M,SYNODINOS A,et al.Towards autonomous reinforcement learning for real-world robotic manipulation with large language models[J].arXiv:2503.04280,2025.
[51] XU P,HUANG K Q.An efficient reinforcement learning method based on large language model[J].Journal of Graphics,2024,45(6):1165-1177.
[52] PARK S G,KIM H B,LEE Y J,et al.TARG:Tree of Action-reward Generation With Large Language Model for Cabinet Opening Using Manipulator[J].International Journal of Control,Automation and Systems,2025,23(2):449-458.
[53] LI H,YANG X,WANG Z,et al.Auto mc-reward:Automated dense reward design with large language models for minecraft[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2024:16426-16435.
[54] SUN S,LIU R,LYU J,et al.A large language model-driven reward design framework via dynamic feedback for reinforcement learning[J].Knowledge-Based Systems,2025,326:114065.
[55] MA Y J,LIANG W,WANG G,et al.Eureka:Human-level reward design via coding large language models [C]//The Twelfth International Conference on Learning Representations(ICLR 2024).Vienna,Austria:OpenReview.net,2024.
[56] MASADOME S,HARADA T.Reward design using large language models for natural language explanation of reinforcement learning agent actions[J].IEEJ Transactions on Electrical and Electronic Engineering,2025,20(8):1203-1211.
[57] NAZIR M S,BANERJEE C.Zero-Shot LLMs in Human-in-the-Loop RL:Replacing Human Feedback for Reward Shaping[J].arXiv:2503.22723,2025.
[58] SYGKOUNAS A,HAZRA R,PERSSON A,et al.REvolve:Reward Evolution with Large Language Models using Human Feedback[C]//Greeks in AI Symposium 2025.2025.
[59] VERMA S,BOEHMER N,KONG L,TAMBE M.Balancingact:Prioritization strategies for LLM-designed restless bandit rewards [C]//Proceedings of the 40th Conference on Uncertainty in Artificial Intelligence(UAI 2024).Rotterdam,Netherlands:AUAI Press,2024.
[60] YANG R,YAN J P,LI X.Survey of sparse reward algorithms in reinforcement learning_theory and experiment[J].CAAI Transactions on Intelligent Systems,2020,15(5):888-899.
[61] MIKOLOV T,SUTSKEVER I,CHEN K,et al.Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems.Lake Tahoe,Nevada,USA:Curran Associates Inc.,2013:3111-3119.
[62] CHU K,ZHAO X,WEBER C,et al.Accelerating reinforcement learning of robotic manipulations via feedback from large language models[J].arXiv:2311.02379,2023.
[63] PLACE A.Adaptive reinforcement learning with llm-augmented reward functions[J].Authorea Preprints,2023.
[64] CAO M,SHU L,YU L,et al.Beyond sparse rewards:Enhancing reinforcement learning with language model critique in text generation [C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing(EMNLP 2024).Miami,Florida,USA:Association for Computational Linguistics,2024:9119-9138.
[65] SUTTON R S,MODAYIL J,DELP M,et al.Horde:A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction[C]//Proceedings of the International Conference on Autonomous Agents and Multiagent Systems(AAMAS).2011.
[66] QU Y,JIANG Y,WANG B,et al.Latent reward:Llm-empowered credit assignment in episodic reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2025:20095-20103.
[67] NAGPAL K,DONG D,MEHR N.Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignment[C]//Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems(AAMAS '25).Richland,SC:International Foundation for Autonomous Agents and Multiagent Systems,2025:1501-1510.
[68] KADAMALA K,CHAMBERS D,BARRETT E.EnhancingHVAC control efficiency:a hybrid approach using Imitation and reinforcement learning[C]//Joint European Conference on Machine Learning and Knowledge Discovery in Databases.Cham:Springer Nature Switzerland,2024:256-270.
[69] ZHAO A,HUANG D,XU Q,et al.Expel:Llm agents are experiential learners[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2024:19632-19642.
[70] KHEIRANDISH A,XU D,FEKRI F.LLM-Augmented Sym-bolic Reinforcement Learning with Landmark-Based Task Decomposition [C]//ICASSP 2025－2025 IEEE International Conference on Acoustics,Speech and Signal Processing.New Delhi,India:IEEE,2025.
[71] MA R,LUIJKX J,AJANOVIĆZ,et al.Explorllm:Guiding exploration in reinforcement learning with large language models[C]//2025 IEEE International Conference on Robotics and Automation(ICRA).IEEE,2025:9011-9017.
[72] PENG H J,ZHANG W Y,CHEN R H,et al.Large-language-model-driven training optimization for air-combat self-defense jamming strategies[J].Journal of Detection & Control,2025,47(2):71-81.
[73] KARINE K,MARLIN B M.Combining LLM decision and RL action selection to improve RL policy for adaptive interventions[J].arXiv:2501.06980,2025.
[74] ZHANG S,ZHENG S,KE S,et al.How can llm guide rl? a va-lue-based approach[J].arXiv:2402.16181,2024.
[75] WANG B,MCKEOWN K,YING R.DYSTIL:Dynamic Strategy Induction with Large Language Models for Reinforcement Learning[J].arXiv:2505.03209,2025.
[76] WANG X S,WANG R R,CHENG Y H. Safe reinforcement learning:A survey[J].Acta Automatica Sinica,2023,49(9):1813-1835.
[77] BRIJEN T ,ASHWIN B,SURAJ N,et al.Recovery rl:Safe reinforcement learning with learned recovery zones[J].IEEE Robotics and Automation Letters,2021,6(3):4915-4922.
[78] YANG T Y,HU M Y,CHOW Y,et al.Safe reinforcementlearning with natural language constraints[J].Advances in Neural Information Processing Systems,2021,34:13794-13808.
[79] GROSS D,SPIEKER H.Enhancing rl safety with counterfactual llm reasoning[C]//IFIP International Conference on Testing Software and Systems.Cham:Springer Nature Switzerland,2024:23-29.
[80] MA J,LIANG W,WANG H J,et al.Dreureka:Language model guided sim-to-real transfer[J].arXiv:2406.01967,2024.
[81] PETRONI F,ROCKTÄSCHEL T,RIEDEL S,et al.Language models as knowledge bases?[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Proces-sing.EMNLP,2019.
[82] PANG J C,YANG S H,LI K,et al.Kalm:Knowledgeable agents by offline reinforcement learning from large language model rollouts[J].Advances in Neural Information Processing Systems,2024,37:126620-126652.
[83] BHAMBR I,SIDDHAN T,AMRITA BHATTACHARJE E,et al.Efficient reinforcement learning via large language model-based search[C]//NeurIPS 2024 Workshop on Open-World Agents.2024.
[84] FRANÇOIS-LAVET V,HENDERSON P,ISLAM R,et al.An introduction to deep reinforcement learning[J].Foundations and Trends® in Machine Learning,2018,11(3/4):219-354.
[85] WANG J,WANG T,CAI W,et al.Boosting Efficient Reinforcement Learning for Vision-and-Language Navigation With Open-Sourced LLM[J].IEEE Robotics and Automation Letters,2025,10(1):612-619.
[86] CHEN Z,LENG B,LI Z,et al.HCRMP:A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving[J].arXiv:2505.15793,2025.
[87] WANG X,REN H,TAN G Z,et al.Autonomous Driving Decision-making Method Based on Cooperative Reinforcement Learning of Large Language Model[J].Journal of Transportation Systems Engineering and Information Technology,2025,25(4):137-146.
[88] XU C,LIU J,GUO Y,et al.Towards human-centric autono-mous driving:a fast-slow architecture integrating large language model guidance with reinforcement learning [J].arXiv:2505.06875,2025.
[89] XU Y,HONG W,ZHA J,et al.Scalable UAV Multi-Hop Networking via Multi-Agent Reinforcement Learning with Large Language Models[J].arXiv:2505.08448.2025.
[90] HUANG L,YU W,MA W,et al.A Survey on Hallucination in Large Language Models:Principles,Taxonomy,Challenges,and Open Questions[J].ACM Transactions on Information Systems,2023,43:1-55.
[91] LEWIS P,PEREZ E,PIKTUS A,et al.Retrieval-augmentedgeneration for knowledge-intensive nlp tasks[J].Advances inNeural Information Processing Systems,2020,33:9459-9474.
[92] HUANG W,LI H,QIN F,et al.MFA-NRM:A novel frame-work for multimodal fusion and semantic alignment in visual neural decoding[J].Information Fusion,2025,127(Part A):103717.
[93] KAPLAN J,MCCANDLISH S,HENIGHAN T,et al.Scaling laws for neural language models[J].arXiv:2001.08361,2020.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed