Abstract: Reinforcement Learning is one of the most challenging and difficult concerns in the field of artificial intelligence.Least-squares method is one of the advanced function approximate methods that can be used to solve the problem of reinforcement learning.It has advantages of fast convergence rate and sufficient utilization of sample data.After the study and analysis of least squares temporal diffe-rence algorithm (LSTD)this paper proposes a double weights with least-squares Sarsa algorithm (DWLS-Sarsa) based on the LSTD algorithm.DWLS-Sarsa combines two weights in a certain way and takes control of temporal diffe-rence error with Sarsa methods.During the training processtwo weights will produce different values because of the difference in the updated samples and will gradually narrow the gap between the two weights until they converge to the same optimal value duo to the distribution of the sample data.So that the exploration performance and convergence of the algorithm will be ensured.FinallyDWLS-Sarsa algorithm is applied to the experiment and compared with other reinforcement learning algorithms.The experimental results show that DWLS-Sarsa algorithm can deal with local optimum problems effectively to achieve more precise convergence value and has better learning performance and robustness.

Key words: Function approximation, Least-squares, Reinforcement learning, Sarsa, Temporal difference