Abstract: Policy iteration is a reinforcement learning method which evaluates and improves the control policy iteratively．Policy evaluation with the least-square method can extract more useful information from the empirical data and improve the data validity．For the low empirical utilization rate of online least-squares policy iteration method which uses each sample only once,a batch least-squares policy iteration (BLSPI) method was proposed and its convergence was proved in theory．BLSPI method combines online least-squares policy iteration method and batch updating method,stores the generated samplesonline and reuses these samples with least-squares methods to update the control policy．We applied the BLSPI method to the inverted pendulum system,and the experiment results show that the method can effectively utilize the previous experience and knowledge,improve the empirical utilization rate,and accelerate the convergence speed.

Key words: Reinforcement learning,Batch updating,Least-squares,Policy iteration