Vanilla Policy Gradient¶
Table of Contents
Background¶
(Previously:必威绝地大逃杀Introduction to RL,必威电竞Part 3)
The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return,and push down the probabilities of actions that lead to lower return,until you arrive at the optimal policy.
Quick Facts¶
 VPG is an onpolicy algorithm.
 VPG can be used for environments with either discrete or continuous action spaces.
 The Spinning Up implementation of VPG supports parallelization with MPI.
Key Equations¶
Letdenote a policy with parameters,anddenote the expected finitehorizon undiscounted return of the policy.The gradient ofis
whereis a trajectory andis the advantage function for the current policy.
The policy gradient algorithm works by updating policy parameters via stochastic gradient ascent on policy performance:
Policy gradient implementations typically compute advantage function estimates based on the infinitehorizon discounted return,despite otherwise using the finitehorizon undiscounted policy gradient formula.
Exploration vs.Exploitation¶
VPG trains a stochastic policy in an onpolicy way.This means that it explores by sampling actions according to the latest version of its stochastic policy.The amount of randomness in action selection depends on both initial conditions and the training procedure.Over the course of training,the policy typically becomes progressively less random,as the update rule encourages it to exploit rewards that it has already found.This may cause the policy to get trapped in local optima.
Documentation¶

spinup.
vpg
( env_fn, actor_critic=, ac_kwargs={}, seed=0, steps_per_epoch=4000, 时代= 50, gamma=0.99, pi_lr=0.0003, vf_lr=0.001, train_v_iters=80, lam=0.97, max_ep_len=1000, logger_kwargs={}, save_freq=10 ) [source] ¶ 
Parameters:  env_fn– A function which creates a copy of the environment.The environment must satisfy the OpenAI Gym API.
 actor_critic–
A function which takes in placeholder symbolsfor state,
x_ph
,and action,a_ph
,and returns the mainoutputs from the agent's Tensorflow computation graph:Symbol Shape Description pi
(batch,act_dim) Samples actions from policy givenstates.logp
(batch,) Gives log probability,according tothe policy,of taking actionsa_ph
in statesx_ph
.logp_pi
(batch,) Gives log probability,according tothe policy,of the action sampled bypi
.v
(batch,) Gives the value estimate for statesinx_ph
.(Critical: make sureto flatten this!)  ac_kwargs(dict) – Any kwargs appropriate for the actor_criticfunction you provided to VPG.
 seed(int) – Seed for random number generators.
 steps_per_epoch(int) – Number of steps of interaction (stateaction pairs)for the agent and the environment in each epoch.
 epochs(int) – Number of epochs of interaction (equivalent tonumber of policy updates) to perform.
 gamma(float) – Discount factor.(Always between 0 and 1.)
 pi_lr(float) – Learning rate for policy optimizer.
 vf_lr(float) – Learning rate for value function optimizer.
 train_v_iters(int) – Number of gradient descent steps to take onvalue function per epoch.
 lam(float) – Lambda for GAELambda.(Always between 0 and 1,close to 1.)
 max_ep_len(int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs(dict) – Keyword args for EpochLogger.
 save_freq(int) – How often (in terms of gap between epochs) to savethe current policy and value function.
Saved Model Contents¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
pi 
Samples an action from the agent,conditioned on states inx . 
v 
Gives value estimate for states inx . 
This saved model can be accessed either by
 running the trained policy with thetest_policy.pytool,
 or loading the whole saved graph into a program withrestore_tf_graph.
References¶
Relevant Papers¶
 Policy Gradient Methods for Reinforcement Learning with Function Approximation,Sutton et al.2000
 Optimizing Expectations: From Deep Reinforcement Learning to Stochastic Computation Graphs,Schulman 2016(a)
 Benchmarking Deep Reinforcement Learning for Continuous Control,Duan et al.2016
 High Dimensional Continuous Control Using Generalized Advantage Estimation,Schulman et al.2016(b)
Why These Papers?¶
Sutton 2000 is included because it is a timeless classic of reinforcement learning theory,and contains references to the earlier work which led to modern policy gradients.Schulman 2016(a) is included because Chapter 2 contains a lucid 必威绝地大逃杀introduction to the theory of policy gradient algorithms,including pseudocode.Duan 2016 is a clear,recent benchmark paper that shows how vanilla policy gradient in the betway电竞deep RL setting (eg with neural network policies and Adam as the optimizer) compares with other deep RL algorithms.Schulman 2016(b) is included because our implementation of VPG makes use of Generalized Advantage Estimation for computing the policy gradient.