BE state value

State value

Notations

Consider the following single-step process:

$$S_t\stackrel{A_t}{\to }{R}_{t+1},S_{t+1}$$

• $t,t+1$: discrete time instances
• $S_t$: state at time t
• $A_t$: the action taken at state $S_t$
• $R_{t+1}$: the reward obtained after taking $A_t$
• $S_{t+1}$: the state transited to after taking $A_t$

为什么这里的SAR都是大写

代表他们都是我们概率论中所学的随机变量（random variables），也就意味着我们可以去求expectation。

Step	Governed Probability Distribution
$S_t\to A_t$	$\pi (A_t=a\mid S_t=s)$
$S_t,A_t\to R_{t+1}$	$p(R_{t+1}=r\mid S_t=s,A_t=a)$
$S_t,A_t\to S_{t+1}$	$p(S_{t+1}=s^{\prime }\mid S_t=s,A_t=a)$

上述例子我们之所以可以拿得到p都是以为还没有脱离model，我们假设自己是知道enviroment model的（model ⇐> probability distributions）

Consider the following multi-step process:

$$S_t\stackrel{A_t}{\to }{R}_{t+1},S_{t+1}\stackrel{A_{t+1}}{\to }{R}_{t+2},S_{t+2}\stackrel{A_{t+2}}{\to }{R}_{t+3},S_{t+3}\dots$$

the discounted return:

$$G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots$$

• $\gamma \in [0,1)$.
• $G_t$ is also random variable since Rs are random variables.

$\text{State value}$: the expectation (or expected value or mean value) of $G_t$ is defined as the state-value function or simply state value:

$$v_{\pi }(s)=\mathbb{E}[G_t\mid S_t=s]$$

remarks:

• It is a function of $s$. It is a conditional expectation with the condition that ==the state starts from $s$==.
• It is based on the policy $\pi$. For a different policy, the state value may be different.
  • $v(s,\pi )$ is the same as $v_{\pi }(s)$

return & state value

我们在实际中通常$\pi (a\mid s),p(r\mid s,a),p(s^{\prime }\mid s,a)$都是不确定的，所以从一个state出发往往可以有多条trajectory，我们对all possible trajectory的return求平均就得到了state value；当然，如果从一个state出发怎么走都只有一条trajectory的话那么state value=return，这就需要上面提及的三个分布都是determinisitic才行。

Example

$$\begin{array}{rl}{v}_{{\pi }_1}(s_1)& =0+\gamma 1+{\gamma }^{2}1+\cdots ,\\ & =0+\gamma (1+\gamma +{\gamma }^2+\cdots ),\\ & =\frac{\gamma }{1-\gamma }.\end{array}$$ $$\begin{array}{rl}{v}_{{\pi }_2}(s_1)& =-1+\gamma 1+{\gamma }^{2}1+\cdots ,\\ & =-1+\gamma (1+\gamma +{\gamma }^2+\cdots ),\\ & =-1+\frac{\gamma }{1-\gamma }.\end{array}$$ $$\begin{array}{rl}{v}_{{\pi }_3}(s_1)& =0.5(0+\gamma 1+{\gamma }^{2}1+\cdots )+0.5(-1+\gamma 1+{\gamma }^{2}1+\cdots ),\\ & =0.5(\frac{\gamma }{1-\gamma })+0.5(-1+\frac{\gamma }{1-\gamma }),\\ & =-0.5+\frac{\gamma }{1-\gamma }.\end{array}$$

小小说明

上面三幅图中$p(r\mid s,a),p(s^{\prime }\mid s,a)$都是determinisitc的，policy1和policy2也都是确定性的，只有policy3是stochastic的，我们用颜色区分出每条trajectory以及对应的discounted return求解过程，最后可以看到$v_{{\pi }_1}(s_1)>v_{{\pi }_3}(s_1)>v_{{\pi }_2}(s_1)$，这样同样可以说明policy1是最优的，policy3次之、policy2最次，这与我们从图中的直观感觉是一致的。