面向大语言模型的强化学习技术研究综述

doi:10.11896/jsjkx.251200192

摘要/Abstract

摘要： 强化学习作为机器学习的一个重要分支,已广泛应用于各类复杂决策任务中。近年来,随着大语言模型的迅速发展,强化学习在大语言模型中的应用逐渐成为研究热点。为此,回顾了强化学习的基本原理,并系统梳理了常见的强化学习算法,包括基于值的强化学习算法、基于策略的强化学习算法以及基于模型的强化学习算法。在此基础上,从全新的视角对现有大语言模型中的强化学习算法进行分类,并讨论了其面临的挑战。最后,指出优化强化学习算法,提高其计算效率和训练稳定性,依然是未来研究的重要方向。

关键词: 强化学习, 大语言模型, 人工智能, 技术研究

Abstract: As a crucial branch of machine learning,reinforcement learning has been widely applied in various complex decision-making tasks.In recent years,with the rapid development of large language models,the application of reinforcement learning in large language models has gradually become a research hotspot.This paper reviews the fundamental principles of reinforcement learning and systematically organizes common reinforcement learning algorithms,including value-based,policy-based,and model-based reinforcement learning algorithms.Building upon this foundation,it categorizes existing reinforcement learning algorithms in large language models from a novel perspective,discusses challenges faced by reinforcement learning in the field of large language models.Finally,this paper highlights that optimizing reinforcement learning algorithms to enhance computational efficiency and training stability remains a key direction for future research.

Key words: Reinforcement learning, Large language model, Artificial intelligence, Technical research

中图分类号:

TP181

商炳楠. 面向大语言模型的强化学习技术研究综述[J]. 计算机科学, 2026, 53(10): 268-276. https://doi.org/10.11896/jsjkx.251200192

SHANG Bingnan. Survey on Reinforcement Learning Techniques for Large Language Models[J]. Computer Science, 2026, 53(10): 268-276. https://doi.org/10.11896/jsjkx.251200192

参考文献

[1] PATIL R,GUDIVADA V.A review of current trends,tech-niques,and challenges in large language models(llms)[J].Applied Sciences,2024,14(5):2074.
[2] ZHAO Y,HE J W,ZHU S C,et al.Security of Large Language Models:Current Status and Challenges[J].Computer Science,2024,51(1):68-71.
[3] KALYAN K S.A survey of GPT-3 family large language models including ChatGPT and GPT-4[J].Natural Language Proces-sing Journal,2024,6:100048.
[4] GRATTAFIORI A,DUBEY A,JAUHRI A,et al.The llama 3 herd of models[J].arXiv:2407.21783,2024.
[5] LIU A,FENG B,XUE B,et al.Deepseek-v3 technical report[J].arXiv:2412.19437,2024.
[6] ERNST D,LOUETTE A,FONTENEAU R.Introduction to reinforcement learning[EB/OL].https://damien-ernst.be/wp-content/uploads/2025/04/introduction_to_reinforcement_learning_v250407.pdf.
[7] HAN G J,ZHU S C,LIN C,et al.Multi-Agent Reinforcement Learning:From Foundational Theory to Cutting-Edge Algorithms[J].Acta Electronica Sinica,2025,53(12):4756-4786.
[8] AGGARWAL P,WELLECK S.L1:Controlling how long a reasoning model thinks with reinforcement learning[J].arXiv:2503.04697,2025.
[9] WANG Z,BI B,PENTYALA S K,et al.A comprehensive survey of llm alignment techniques:Rlhf,rlaif,ppo,dpo and more[J].arXiv:2407.16216,2024.
[10] JI J,QIU T,CHEN B,et al.AI alignment:A comprehensive survey[J].arXiv:2310.19852,2023.
[11] SHEN T,JIN R,HUANG Y,et al.Large language model alignment:A survey[J]. arXiv:2309.15025,2023.
[12] HARMON M E,HARMON S S.Reinforcement Learning:ATutorial[R].Wright Laboratory,Avionics Directorate,1997.
[13] MOUSAVI S S,SCHUKAT M,HOWLEY E.Deep reinforcement learning:an overview[C]//Proceedings of SAI Intelligent Systems Conference.Cham:Springer,2016:426-440.
[14] ZAMBALDI V,RAPOSO D,SANTORO A,et al.Relationaldeep reinforcement learning[J].arXiv:1806.01830,2018.
[15] VINYALS O,BABUSCHKIN I,CZARNECKI W M,et al.Grandmaster level in StarCraft II using multi-agent reinforcement learning[J].Nature,2019,575(7782):350-354.
[16] BERNER C,BROCKMAN G,CHAN B,et al.Dota 2 with large scale deep reinforcement learning[J].arXiv:1912.06680,2019.
[17] SILVER D,HUANG A.MADDISON C,et al.Mastering thegame of Go with deep neural networks and tree search[J].Nature,2016,529:484-489.
[18] BYEON H.Advances in value-based,policy-based,and deeplearning-based reinforcement learning[J].International Journal of Advanced Computer Science and Applications,2023,14(8):348-354.
[19] RIS-ALA R.Q-Learning Algorithm[M]//Fundamentals of Reinforcement Learning.Cham:Springer,2023:31-55.
[20] JIA Y,ZHOU X Y.q-Learning in continuous time[J].Journal of Machine Learning Research,2023,24(161):1-61.
[21] MNIH V,KAVUKCUOGLU K,SILVER D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529-533.
[22] OSBAND I,BLUNDELL C,PRITZEL A,et al.Deep explora-tion via bootstrapped DQN [J].Advances in Neural Information Processing Systems,2016,29:4033-4041.
[23] ZHAO H,TANG W,YAO D.Policy optimization for continuous reinforcement learning[J].Advances in Neural Information Processing Systems,2023,36:13637-13663.
[24] STÅHLBERG S,BONET B,GEFFNER H.Learning generalpolicies with policy gradient methods[C]//Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning.2023:647-657.
[25] WILLIAMS R J.Simple statistical gradient-following algorithms for connectionist reinforcement learning[J].Machine Learning,1992,8(3):229-256.
[26] MNIH V,BADIA A P,MIRZA M,et al.Asynchronous methods for deep reinforcement learning[C]//International Conference on Machine Learning.PMLR,2016:1928-1937.
[27] HAARNOJA T,ZHOU A,ABBEEL P,et al.Soft actor-critic:Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//International Conference on Machine Learning.PMLR,2018:1861-1870.
[28] SEWAK M.Actor-critic models and the A3C:The asynchronous advantage actor-critic model[M]//Deep reinforcement learning:frontiers of artificial intelligence.Singapore:Springer,2019:141-152.
[29] ANDRYCHOWICZ M,RAICHUK A,STAŃCZYK P,et al.What matters in on-policy reinforcement learning? a large-scale empirical study[J].arXiv:2006.05990,2020.
[30] MUNOS R,STEPLETON T,HARUTYUNYAN A,et al.Safe and efficient off-policy reinforcement learning[C]//Advances in Neural Information Processing Systems.2016.
[31] MOERLAND T M,BROEKENS J,PLAAT A,et al.Model-based reinforcement learning:A survey[J].Foundations and Trends® in Machine Learning,2023,16(1):1-118.
[32] WANG D,GAO N,LIU D,et al.Recent progress in reinforcement learning and adaptive dynamic programming for advanced control applications[J].IEEE/CAA Journal of Automatica Sinica,2023,11(1):18-36.
[33] ŚWIECHOWSKI M,GODLEWSKI K,SAWICKI B,et al.Monte Carlo tree search:A review of recent modifications and applications[J].Artificial Intelligence Review,2023,56(3):2497-2562.
[34] JAMES S,KONIDARIS G,ROSMAN B.An analysis of monte carlo tree search[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2017.
[35] CHASLOT G M J,WINANDS M H M,HERIK H J,et al.Progressive strategies for Monte-Carlo tree search[J].New Mathematics and Natural Computation,2008,4(3):343-357.
[36] WINANDS M H M,BJÖRNSSON Y,SAITO J T.Monte-Carlo tree search solver[C]//International Conference on Computers and Games.Berlin:Springer,2008:25-36.
[37] ZHANG D,HUANG X,ZHOU D,et al.Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B:A Technical Report[J].arXiv:2406.07394,2024.
[38] WU J,FENG M,ZHANG S,et al.Beyond examples:High-level automated reasoning paradigm in in-context learning via mcts[J].arXiv:2411.18478,2024.
[39] KARTAL B,NUNES E,GODOY J,et al.Monte Carlo treesearch for multi-robot task allocation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2016.
[40] LI W,LIU Y,MA Y,et al.A self-learning Monte Carlo treesearch algorithm for robot path planning[J].Frontiers in Neurorobotics,2023,17:1039644.
[41] MO S,PEI X,WU C.Safe reinforcement learning for autonomous vehicle using monte carlo tree search[J].IEEE Transactions on Intelligent Transportation Systems,2021,23(7):6766-6773.
[42] MCKINNEY L,ZHANG Y.Training language models to follow instructions with human feedback[C]//Proceedings of the 36th International Conference on Neural Information Processing Systems(NIPS’22).2023:27730-27744.
[43] WU X.A Comprehensive Survey on Learning from Rewards for Large Language Models:Reward Models and Learning Strategies[C]//Findings of the Association for Computational Linguistics(EMNLP 2025).2025:17847-17875.
[44] RAFAILOV R,SHARMA A,MITCHELL E,et al.Direct preference optimization:Your language model is secretly a reward model[J].Advances in Neural Information Processing Systems,2023,36:53728-53741.
[45] ETHAYARAJH K,XU W,MUENNIGHOFF N,et al.Kto:Model alignment as prospect theoretic optimization[J].arXiv:2402.01306,2024.
[46] CHAUDHARI S,AGGARWAL P,MURAHARI V,et al.Rlhf deciphered:A critical analysis of reinforcement learning from human feedback for llms[J].ACM Computing Surveys,2025,58(2):1-37.
[47] DONG H,XIONG W,PANG B,et al.Rlhf workflow:From reward modeling to online rlhf[J].arXiv:2405.07863,2024.
[48] OUYANG L,WU J,JIANG X,et al.Training language models to follow instructions with human feedback[J].Advances in Neural Information Processing Systems,2022,35:27730-27744.
[49] WANG Y,KORDI Y,MISHRA S,et al.Self-instruct:Aligning language models with self-generated instructions[C]//Procee-dings of the 61st Annual Meeting of the Association for Computational Linguistics.2023:13484-13508.
[50] XU C,SUN Q,ZHENG K,et al.Wizardlm:Empowering large language models to follow complex instructions[J].arXiv:2304.12244,2023.
[51] ZHENG R,DOU S,GAO S,et al.Secrets of rlhf in large language models part i:Ppo[J].arXiv:2307.04964,2023.
[52] SCHULMAN J,WOLSKI F,DHARIWAL P,et al.Proximalpolicy optimization algorithms[J].arXiv:1707.06347,2017.
[53] HERSHEY J R,OLSEN P A.Approximating the KullbackLeibler divergence between Gaussian mixture models[C]//2007 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP’07).IEEE,2007.
[54] PANG X,TANG S,YE R,et al.Self-alignment of large lan-guage models via multi-agent social simulation[C]//ICLR 2024 Workshop on Large Language Model(LLM) Agents.2024.
[55] LEVY J S.An introduction to prospect theory[J].Political Psychology,1992,13(2):171-186.
[56] SRIVASTAVA S S,AGGARWAL V.A technical survey of reinforcement learning techniques for large language models[J].arXiv:2507.04136,2025.
[57] UESATO J,KUSHMAN N,KUMAR R,et al.Solving mathword problems with process-and outcome-based feedback[J].arXiv:2211.14275,2022.
[58] LIU S,CAO J,WANG Y,et al.Self-play reinforcement learning with comprehensive critic in computer games[J].Neurocompu-ting,2021,449:207-213.
[59] ARNESEN S,REIN D,MICHAEL J.Training language models to win debates with self-play improves judge accuracy[J].ar-Xiv:2409.16636,2024.
[60] WANG Y,LI Z,ZANG Y,et al.Unified multimodal chain-of-thought reward model through reinforcement fine-tuning[J].arXiv:2505.03318,2025.
[61] SONG Y,KEMPE J,MUNOS R.Outcome-based exploration for llm reasoning[J].arXiv:2509.06941,2025.
[62] ZHANG W,LI X,DONG K,et al.Process vs. Outcome Re-ward:Which is Better for Agentic RAG Reinforcement Learning[J].arXiv:2505.14069,2025.
[63] DONG Y,JIANG X,TAO Y,et al.Countering capability boun-dary collapse of llms in reinforcement learning with hybrid-policy optimization[J].arXiv:2508.00222,2025.
[64] SHINN N,CASSANO F,GOPINATH A,et al.Reflexion:Language agents with verbal reinforcement learning[J].Advances in Neural Information Processing Systems,2023,36:8634-8652.
[65] CAO Y,ZHAO H,CHENG Y,et al.Survey on large language model-enhanced reinforcement learning:Concept,taxonomy,and methods[J].IEEE Transactions on Neural Networks and Lear-ning Systems,2025,36(6):9737-9757.
[66] LIU K,YANG D,QIAN Z,et al.Reinforcement learning meets large language models:A survey of advancements and applications across the llm lifecycle[J].arXiv:2509.16679,2025.
[67] ZHOU X,CAO L,CHEN X l,et al.A Review of Research on Large Language Model-Empowered Reinforcement Learning Agents[J/OL].Computer Science,1-28.[2026-03-04].https://link.cnki.net/urlid/50.1075.tp.20251222.0819.002.
[68] CAO Y Q,CHEN X L,DONG H Y,et al.A Survey on Large Model-Assisted Reward Design Methods for Reinforcement Learning[J].Computer Engineering and Applications,2026,62(9):83-107.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed