Concept return

Return

A $\text{trajectory}$ is state-action-reward chain:

$$s_1\underset{r=0}{\overset{a_2}{\to }}{s}_2\underset{r=0}{\overset{a_3}{\to }}{s}_5\underset{r=0}{\overset{a_3}{\to }}{s}_8\underset{r=1}{\overset{a_2}{\to }}{s}_9$$

The $\text{return}$ of this trajectory is the sum of all the rewards collected along the trajectory:

$$\text{return}=0+0+0+1=1$$

A different policy gives a different policy:

$$s_1\underset{r=0}{\overset{a_3}{\to }}{s}_4\underset{r=-1}{\overset{a_3}{\to }}{s}_7\underset{r=0}{\overset{a_2}{\to }}{s}_8\underset{r=1}{\overset{a_2}{\to }}{s}_9$$ $$\text{return}=0-1+0+1=0$$

return有什么用?

从上面这个例子我们可以看到第一个例子的return>第二个例子的return，那么我们可以初步认为policy1是由于policy2的，所以我们用return来衡量一个policy的好坏。

Discounted return

A trajectory maybe infinite:

$$s_1\underset{r=0}{\overset{a_2}{\to }}{s}_2\underset{r=0}{\overset{a_3}{\to }}{s}_5\underset{r=0}{\overset{a_3}{\to }}{s}_8\underset{r=1}{\overset{a_2}{\to }}{s}_9\underset{r=1}{\overset{a_5}{\to }}{s}_9\underset{r=1}{\overset{a_5}{\to }}{s}_9\dots$$ $$\text{return}=0+0+0+1+1+1+\dots =\infty$$

return diverge! so we need a discount rate $\gamma$ to calculate the discounted return:

$$\begin{array}{rl}\text{discounted return}& =0+\gamma 0+{\gamma }^{2}0+{\gamma }^{3}1+{\gamma }^{4}1+{\gamma }^{5}1+\dots \\ & ={\gamma }^3\ast (1+\gamma +{\gamma }^2+\dots )\\ & ={\gamma }^3/(1-\gamma )\end{array}$$

补充证明： $\frac{1}{1-\gamma }$数列求和求极限（高等数学知识）

引入gamma的作用

1. 由于reward只是一个常数，我们轻而易举的将return(sum of trajectory rewards)收敛起来了
2. 我们可以balance the far and near future rewards，直接从$\text{discounted return}=r_0+\gamma r_1+{\gamma }^2{r}_2+\dots$的式子可以看出:
   1. $\gamma$ close to 0, return更加关注于near future
   2. $\gamma$ 越接近，说明更多future的reward被考虑进来，far future