BE action value

Action value

• State value: the average return the agent can get starting from a state
• Action value: the average return the agent can get starting from a state and taking an action

Definition:

$$q_{\pi }(s,a)=\mathbb{E}[G_t\mid S_t=s,A_t=a]$$

• $q_{\pi }$ is a function of state-action pair $(s,a)$
• the function depends on $\pi$

Recalls that:

$$\underset{v_{\pi }(s)}{\underbrace{\mathbb{E}[G_t\mid S_t=s]}}=\sum _a\underset{q_{\pi }(s,a)}{\underbrace{\mathbb{E}[G_t\mid S_t=s,A_t=a]}}\pi (a\mid s)$$

Hence,

$$v_{\pi }(s)=\sum _a\pi (a\mid s)q_{\pi }(s,a)$$

Also we can expand it, refer to the derivation of state value

$$q_{\pi }(s,a)=\sum _{r}p(r\mid s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })$$

Matrix form

Less is more, then we will have six matrix formulla here

All in one:

$$v_{\pi }=\pi @(P_{r}R+\gamma P_s{v}_{\pi })$$

Two sides of the coin:

$$\begin{array}{rl}{q}_{\pi }& =P_{r}R+\gamma P_s{v}_{\pi }\\ v_{\pi }& =\pi @q_{\pi }\end{array}$$

Three basic formula we got during the derivation of state value:

$$\begin{array}{rl}{v}_{\pi }& =r_{\pi }+\gamma P_{\pi }{v}_{\pi }\\ r_{\pi }& =\pi @P_{r}R\\ P_{\pi }& =\pi @P_s\end{array}$$

一些有意思的地方

最后其实可以发现一个有意思的点就是$q_{\pi }\in {\mathbb{R}}^{n\times m}$，也就是说action value和policy的shape是一样的，这也就说明了我们其实说白了是在从action value中通过policy概率去统合每个动作的价值，才得到当前state的价值，这样显得非常直观，但是又非常巧妙的将action probability(policy)和action value一一对应起来了； 这也基本解决当时初学强化学习时一直没弄弄懂Q表和policy的关系，当然也可以直接根据action value的大小选取价值最大的action，从而得到当前最正确的policy，这样policy和Q相乘当然也可以得到最大的state value，刚好两全其美。