Soft ActorCritic¶
Table of Contents
Background¶
(Previously:Background for TD3)
Soft Actor Critic (SAC) is an algorithm which optimizes a stochastic policy in an offpolicy way,forming a bridge between stochastic policy optimization and DDPGstyle approaches.It isn't a direct successor to TD3 (having been published roughly concurrently),but it incorporates the clipped doubleQ trick,and due to the inherent stochasticity of the policy in SAC,it also winds up benefiting from something like target policy smoothing.
A central feature of SAC isentropy regularization.The policy is trained to maximize a tradeoff between expected return andentropy,a measure of randomness in the policy.This has a close connection to the explorationexploitation tradeoff: increasing entropy results in more exploration,which can accelerate learning later on.It can also prevent the policy from prematurely converging to a bad local optimum.
Quick Facts¶
 SAC is an offpolicy algorithm.
 The version of SAC implemented here can only be used for environments with continuous action spaces.
 An alternate version of SAC,which slightly changes the policy update rule,can be implemented to handle discrete action spaces.
 The Spinning Up implementation of SAC does not support parallelization.
Key Equations¶
To explain Soft Actor Critic,we first have to introduce the entropyregularized reinforcement learning setting.In entropyregularized RL,there are slightlydifferent equations for value functions.
EntropyRegularized Reinforcement Learning¶
Entropy is a quantity which,roughly speaking,says how random a random variable is.If a coin is weighted so that it almost always comes up heads,it has low entropy;if it's evenly weighted and has a half chance of either outcome,it has high entropy.
Letbe a random variable with probability mass or density function.The entropyofis computed from its distributionaccording to
In entropyregularized reinforcement learning,the agent gets a bonus reward at each time step proportional to the entropy of the policy at that timestep.This changesthe RL problemto:
whereis the tradeoff coefficient.(Note: we're assuming an infinitehorizon discounted setting here,and we'll do the same for the rest of this page.) We can now define the slightlydifferent value functions in this setting.is changed to include the entropy bonuses from every timestep:
is changed to include the entropy bonuses from every timestepexcept the first:
With these definitions,andare connected by:
and the Bellman equation foris
You Should Know
The way we've set up the value functions in the entropyregularized setting is a little bit arbitrary,and actually we could have done it differently (eg makeinclude the entropy bonus at the first timestep).The choice of definition may vary slightly across papers on the subject.
Soft ActorCritic¶
SAC concurrently learns a policy,two Qfunctions,and a value function.
Learning Q.The Qfunctions are learned by MSBE minimization,using atarget value networkto form the Bellman backups.They both use the same target,like in TD3,and have loss functions:
The target value network,like the target networks in DDPG and TD3,is obtained by polyak averaging the value network parameters over the course of training.
Learning V.The value function is learned by exploiting (a samplebased approximation of) the connection betweenand.Before we go into the learning rule,let's first rewrite the connection equation by using the definition of entropy to obtain:
The RHS is an expectation over actions,so we can approximate it by sampling from the policy:
SAC sets up a meansquarederror loss forbased on this approximation.But what Qvalue do we use?SAC usesclipped doubleQlike TD3 for learning the value function,and takes the minimum Qvalue between the two approximators.So the SAC loss for value function parameters is:
Importantly,we do不use actions from the replay buffer here: these actions are sampled fresh from the current version of the policy.
Learning the Policy.The policy should,in each state,act to maximize the expected future return plus expected future entropy.That is,it should maximize,which we expand out (as before) into
The way we optimize the policy makes use of thereparameterization trick,in which a sample fromis drawn by computing a deterministic function of state,policy parameters,and independent noise.To illustrate: following the authors of the SAC paper,we use a squashed Gaussian policy,which means that samples are obtained according to
You Should Know
This policy has two key differences from the policies we use in the other policy optimization algorithms:
1.The squashing function.Thein the SAC policy ensures that actions are bounded to a finite range.This is absent in the VPG,TRPO,and PPO policies.It also changes the distribution: before thethe SAC policy is a factored Gaussian like the other algorithms' policies,but after theit is not.(You can still compute the logprobabilities of actions in closed form,though: see the paper appendix for details.)
2.The way standard deviations are parameterized.In VPG,TRPO,and PPO,we represent the log std devs with stateindependent parameter vectors.In SAC,we represent the log std devs as outputs from the neural network,meaning that they depend on state in a complex way.SAC with stateindependent log std devs,in our experience,did not work.(Can you think of why?Or better yet: run an experiment to verify?)
The reparameterization trick allows us to rewrite the expectation over actions (which contains a pain point: the distribution depends on the policy parameters) into an expectation over noise (which removes the pain point: the distribution now has no dependence on parameters):
To get the policy loss,the final step is that we need to substitutewith one of our function approximators.The same as in TD3,we use.The policy is thus optimized according to
which is almost the same as the DDPG and TD3 policy optimization,except for the stochasticity and entropy term.
Exploration vs.Exploitation¶
SAC trains a stochastic policy with entropy regularization,and explores in an onpolicy way.The entropy regularization coefficientexplicitly controls the exploreexploit tradeoff,with highercorresponding to more exploration,and lowercorresponding to more exploitation.The right coefficient (the one which leads to the stablest / highestreward learning) may vary from environment to environment,and could require careful tuning.
At test time,to see how well the policy exploits what it has learned,we remove stochasticity and use the mean action instead of a sample from the distribution.This tends to improve performance over the original stochastic policy.
You Should Know
Our SAC implementation uses a trick to improve exploration at the start of training.For a fixed number of steps at the beginning (set with thestart_steps
keyword argument),the agent takes actions which are sampled from a uniform random distribution over valid actions.After that,it returns to normal SAC exploration.
Documentation¶

spinup.
sac
( env_fn, actor_critic=, ac_kwargs={}, seed=0, steps_per_epoch = 5000, epochs=100, replay_size=1000000, gamma=0.99, polyak=0.995, lr=0.001, alpha=0.2, batch_size=100, start_steps=10000, max_ep_len=1000, logger_kwargs={}, save_freq=1 ) [source] ¶ 
Parameters:  env_fn– A function which creates a copy of the environment.The environment must satisfy the OpenAI Gym API.
 actor_critic–
A function which takes in placeholder symbolsfor state,
x_ph
,and action,a_ph
,and returns the mainoutputs from the agent's Tensorflow computation graph:Symbol Shape Description mu
(batch,act_dim) Computes mean actions from policygiven states.pi
(batch,act_dim) Samples actions from policy givenstates.logp_pi
(batch,) Gives log probability,according tothe policy,of the action sampled bypi
.Critical: must be differentiablewith respect to policy parameters allthe way through action sampling.q1
(batch,) Gives one estimate of Q* forstates inx_ph
and actions ina_ph
.q2
(batch,) Gives another estimate of Q* forstates inx_ph
and actions ina_ph
.q1_pi
(batch,) Gives the composition ofq1
andpi
for states inx_ph
:q1(x,pi(x)).q2_pi
(batch,) Gives the composition ofq2
andpi
for states inx_ph
:q2(x,pi(x)).v
(batch,) Gives the value estimate for statesinx_ph
.  ac_kwargs(dict) – Any kwargs appropriate for the actor_criticfunction you provided to SAC.
 seed(int) – Seed for random number generators.
 steps_per_epoch(int) – Number of steps of interaction (stateaction pairs)for the agent and the environment in each epoch.
 epochs(int) – Number of epochs to run and train agent.
 replay_size(int) – Maximum length of replay buffer.
 gamma(float) – Discount factor.(Always between 0 and 1.)
 polyak(float) –
Interpolation factor in polyak averaging for targetnetworks.Target networks are updated towards main networksaccording to:
whereis polyak.(Always between 0 and 1,usuallyclose to 1.)
 lr(float) – Learning rate (used for both policy and value learning).
 alpha(float) – Entropy regularization coefficient.(Equivalent toinverse of reward scale in the original SAC paper.)
 batch_size(int) – Minibatch size for SGD.
 start_steps(int) – Number of steps for uniformrandom action selection,before running real policy.Helps exploration.
 max_ep_len(int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs(dict) – Keyword args for EpochLogger.
 save_freq(int) – How often (in terms of gap between epochs) to savethe current policy and value function.
Saved Model Contents¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
a 
Tensorflow placeholder for action input. 
mu 
Deterministically computes mean action from the agent,given states inx . 
pi 
Samples an action from the agent,conditioned on states inx . 
q1 
Gives one actionvalue estimate for states inx and actions ina . 
q2 
Gives the other actionvalue estimate for states inx and actions ina . 
v 
Gives the value estimate for states inx . 
This saved model can be accessed either by
 running the trained policy with thetest_policy.pytool,
 or loading the whole saved graph into a program withrestore_tf_graph.
Note: for SAC,the correct evaluation policy is given bymu
and not bypi
.The policypi
may be thought of as the exploration policy,whilemu
is the exploitation policy.