VF Sarsa

Sarsa with Value Function

Introduction

So far, we merely considered state value estimation. That is

$$\hat{v}(s)\approx v_{\pi }(s),s\in \mathcal{S}$$

To search for optimal policies, we need to estimate action values.

The Sarsa algorithm with value function approximation is

$$w_{t+1}=w_t+{\alpha }_t\left[r_{t+1}+\gamma \hat{q}(s_{t+1},a_{t+1},w_t)-\hat{q}(s_t,a_t,w_t)\right]{\nabla }_w\hat{q}(s_t,a_t,w_t)$$

This is the same as the algorithm we introduced previously in this lecture except that $v$ is replaced by $q$.

Core Idea

Replace $\hat{v}(s,w)$ with $\hat{q}(s,a,w)$ in VF state value.

Implementation

可以对比Implementation的实现来看这里的改进

$$\boxed{\begin{array}{rl}& \text{Algorithm}\text{SARSA with Function Approximation and }\epsilon \text{-Greedy}\\ & \text{Initialization: }\text{Initial parameters }{\mathbf{w}}_0,\text{ policy }{\pi }_0(a|s)\text{, step size }\alpha \in (0,1],\text{ discount factor }\gamma .\\ & \text{For each episode, do:}\\ & \text{Initialize state }{s}_0.\\ & \text{Select action }{a}_0\text{ using }{\pi }_0(s_0).\\ & \text{While }{s}_t\text{ is not the target state, do:}\\ & \text{Execute }{a}_t\text{, observe }{r}_{t+1},s_{t+1}.\\ & \text{Select }{a}_{t+1}\text{ using }{\pi }_t(s_{t+1}).\\ & \text{Update parameters (update q-value):}\\ & {\mathbf{w}}_{t+1}\leftarrow {\mathbf{w}}_t+\alpha \left[r_{t+1}+\gamma \hat{q}(s_{t+1},a_{t+1},{\mathbf{w}}_t)-\hat{q}(s_t,a_t,{\mathbf{w}}_t)\right]{\nabla }_{\mathbf{w}}\hat{q}(s_t,a_t,{\mathbf{w}}_t)\\ & \text{Update policy for }{s}_t:\\ & {\pi }_{t+1}(a|s_t)\leftarrow \{\begin{array}{ll}1-\epsilon +\frac{\epsilon }{|\mathcal{A}(s_t)|},& \text{if }a=\arg {\max }_{a^{\prime }}\hat{q}(s_t,a^{\prime },{\mathbf{w}}_{t+1})\\ \frac{\epsilon }{|\mathcal{A}(s_t)|},& \text{otherwise}\end{array}\\ & s_t\leftarrow s_{t+1},a_t\leftarrow a_{t+1}\end{array}}$$