AC QAC

Simplest Q Actor Critic(QAC)

Revisit

Revisit the idea of policy gradient introduced in the last lecture.

1. A scalar metric $J(\theta )$, which can be ${\bar{v}}_{\pi }$ or ${\bar{r}}_{\pi }$.
2. The gradient-ascent algorithm maximizing $J(\theta )$ is $$\begin{array}{rl}{\theta }_{t+1}& ={\theta }_t+\alpha {\nabla }_{\theta }J({\theta }_t)\\ & ={\theta }_t+\alpha {\nabla }_{\theta }{\mathbb{E}}_{S\sim \mu ,A\sim \pi }\left[q_{\pi }(S,A){\nabla }_{\theta }\ln \pi (A|S,\theta )\right].\end{array}$$
3. The stochastic gradient-ascent algorithm is $$\begin{array}{rl}{\theta }_{t+1}& ={\theta }_t+\alpha q_t(s_t,a_t){\nabla }_{\theta }\ln \pi (a_t|s_t,{\theta }_t).\end{array}$$ This expression is very important! We can directly see actor and critic from it:
   • This expression corresponds to actor!
   • The algorithm estimating $q_t(s_t,a_t)$ corresponds to critic!

How to get $q_t(s_t,a_t)$?

So far, we have studied two ways to estimate action values:

1. Monte Carlo learning: If MC is used, the corresponding algorithm is called REINFORCE or Monte Carlo policy gradient.
2. Temporal difference learning: If TD is used, such kind of algorithms are usually called actor-critic.

Core Idea

用one-step difference(TD)代替G

Implementation

$$\boxed{\begin{array}{rl}& \text{Algorithm}\text{Q Actor-Critic (QAC) with On-Policy TD}\\ & \text{Initialization: }\\ & \text{Policy parameters }{\theta }_0,\text{ value function parameters }{w}_0.{\alpha }_{\theta },{\alpha }_w>0.\\ & \text{For each episode, do:}\\ & \text{Initialize state }{s}_0.\\ & \text{Select action }{a}_0\sim \pi (a|s_0,{\theta }_0).\\ & \text{While }{s}_t\text{ is not terminal, do:}\\ & \text{Execute }{a}_t\text{, observe }{r}_{t+1},s_{t+1}.\\ & \text{Select }{a}_{t+1}\sim \pi (a|s_{t+1},{\theta }_t).\\ & \text{Actor (Policy Update):}\\ & {\theta }_{t+1}\leftarrow {\theta }_t+{\alpha }_{\theta }\cdot q(s_t,a_t,w_t)\cdot {\nabla }_{\theta }\ln \pi (a_t|s_t,{\theta }_t)\\ & \text{Critic (Value Update):}\\ & {\delta }_t=r_{t+1}+\gamma q(s_{t+1},a_{t+1},w_t)-q(s_t,a_t,w_t)\\ & w_{t+1}\leftarrow w_t+{\alpha }_w{\delta }_t{\nabla }_{w}q(s_t,a_t,w_t)\\ & s_t\leftarrow s_{t+1},a_t\leftarrow a_{t+1}\end{array}}$$

• critic: SARSA + value function approximation
• actor: policy gradient
• Q Actor-Critic is the simplest actor-critic, reveals the core idea of AC

Later

这里存在的问题

因为是自己一直在评估自己，会导致q越来越高越来越高(bootstrap，自举)，所以这里需要引入一个基准(baseline)，使Q变成Q-baseline，也即advantage

Q Actor-Critic(QAC) ⇒ Advantage Actor-Critic(A2C)