AC DPG

Deterministic Policy Gradient(DPG)

Introduction

注意

本节出现的所有$\mu$都是指的是一个映射，而不是之前章节所提及的on-policy distribution。

Up to now, the policies used in the policy gradient methods are all stochastic since $\pi (a|s,\theta )>0$ for every $(s,a)$.

Can we use deterministic policies in the policy gradient methods? The benefit is that it can handle continuous action spaces.

The ways to represent a policy:

• Up to now, a general policy is denoted as $\pi (a|s,\theta )\in [0,1]$, which can be either stochastic or deterministic.
• Now, the deterministic policy is specifically denoted as $$a=\mu (s,\theta )\doteq \mu (s)$$
  • $\mu$ is a mapping from $\mathcal{S}$ to $\mathcal{A}$.
  • $\mu$ can be represented by, for example, a neural network with the input as $s$, the output as $a$, and the parameter as $\theta$.
  • We may write $\mu (s,\theta )$ in short as $\mu (s)$.

一些理解

之前都是可以直接输入一个state，然后输出一个action的概率分布，因为我的action是离散的（比如输出机器人应该是上下左右），而正因为是离散的，所以每个动作都有一定概率会被选中；但是如果当需要输出的action是连续的（比如输出机器人的速度），此时更像一个回归问题了（我们输出的只能是一个速度，而不是几个速度再概率采样这样子，因为速度是连续的，有无数个速度在定义域中），那也就是确定性策略。

确定性策略的情况下，我们如果要获得对应的动作概率的话，可以用高斯分布来实现（Sutton书中的13.7节说了证怎么求得概率）

Theorem of deterministic policy gradient

• The policy gradient theorems introduced before are merely valid for stochastic policies.
• If the policy must be deterministic, we must derive a new policy gradient theorem.
• The ideas and procedures are similar.

Consider the metric of average state value in the discounted case:

$$J(\theta )=\mathbb{E}[v_{\mu }(s)]=\sum _{s\in \mathcal{S}}{d}_0(s)v_{\mu }(s)$$

where $d_0(s)$ is a probability distribution satisfying ${\sum }_{s\in \mathcal{S}}{d}_0(s)=1$.

How to select $d_0$?

• same as the last lecture.
• normal case: uniform; special case: one-hot, stationary distribution

In the discounted case where $\gamma \in (0,1)$, the gradient of $J(\theta )$ is

$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =\sum _{s\in \mathcal{S}}{\rho }_{\mu }(s){\nabla }_{\theta }\mu (s)({\nabla }_a{q}_{\mu }(s,a)){|}_{a=\mu (s)}\\ & ={\mathbb{E}}_{S\sim {\rho }_{\mu }}\left[{\nabla }_{\theta }\mu (S)({\nabla }_a{q}_{\mu }(S,a)){|}_{a=\mu (S)}\right]\end{array}$$

Here ${\rho }_{\mu }$ is the state distribution under policy $\mu$.

One important difference from the stochastic case:

• The gradient does not involve the distribution of the action $A$.
• As a result, the deterministic policy gradient method is off-policy.

For detail, see Proof.

Algorithm of deterministic policy gradient

Based on the policy gradient, the gradient-ascent algorithm for maximizing $J(\theta )$ is:

$$\begin{array}{rl}{\theta }_{t+1}& ={\theta }_t+{\alpha }_{\theta }{\nabla }_{\theta }J(\theta )\\ & ={\theta }_t+{\alpha }_{\theta }{\mathbb{E}}_{S\sim {\rho }_{\mu }}\left[{\nabla }_{\theta }\mu (S)({\nabla }_a{q}_{\mu }(S,a)){|}_{a=\mu (S)}\right]\end{array}$$

The corresponding stochastic gradient-ascent algorithm is

$${\theta }_{t+1}={\theta }_t+{\alpha }_{\theta }{\nabla }_{\theta }\mu (s_t)({\nabla }_a{q}_{\mu }(s_t,a)){|}_{a=\mu (s_t)}$$

Implementation

Compared with Implementation

$$\boxed{\begin{array}{rl}& \text{Algorithm}\text{Deterministic Actor-Critic}\\ & \text{Initialization: }\\ & \text{Policy μ(s,θ) parameters }{\theta }_0,\text{ value function q(s,a,w) parameters }{w}_0,\\ & \text{given behavior policy }\beta (a|s).{\alpha }_{\theta },{\alpha }_w>0.\\ & \text{For each episode, do:}\\ & \text{Initialize state }{s}_0.\\ & \text{Select action }{a}_0\sim \beta (a|s_0).\\ & \text{While }{s}_t\text{ is not terminal, do:}\\ & \text{Execute }{a}_t\text{, observe }{r}_{t+1},s_{t+1}.\\ & \text{Select }{a}_{t+1}\sim \beta (a|s_{t+1}).\\ & \text{TD Error:}\\ & {\delta }_t=r_{t+1}+\gamma q(s_{t+1},\mu (s_{t+1},{\theta }_t),w_t)-q(s_t,a_t,w_t)\\ & \text{Actor (Policy Update):}\\ & {\theta }_{t+1}\leftarrow {\theta }_t+{\alpha }_{\theta }{\nabla }_{\theta }\mu (s_t,{\theta }_t)({\nabla }_a{q}_{\mu }(s_t,a,w_t)){|}_{a=\mu (s_t)}\\ & \text{Critic (Value Update):}\\ & w_{t+1}\leftarrow w_t+{\alpha }_w{\delta }_t{\nabla }_{w}q(s_t,w_t)\\ & s_t\leftarrow s_{t+1},a_t\leftarrow a_{t+1}\end{array}}$$

• $\mu (s,\theta )$ is deterministic target policy.
• $\pi (a|s,\theta )$ is stochastic behavior policy.
• This is an off-policy implementation where the behavior policy $\beta$ may be different from $\mu$.
• $\beta$ can also be replaced by $\mu +\text{noise}$.
• How to select the function to represent $q(s,a,w)$?
  • Linear function: $q(s,a,w)={\varphi }^T(s,a)w$ where $\varphi (s,a)$ is the feature vector.
  • Neural networks: deep deterministic policy gradient (DDPG) method.