VF Qlearning

Q-learning with Value Function

Introduction

The Q-learning algorithm with value function approximation is

$$w_{t+1}=w_t+{\alpha }_t\left[r_{t+1}+\gamma \underset{a^{\prime }\in \mathcal{A}(s_{t+1})}{\max }\hat{q}(s_{t+1},a^{\prime },w_t)-\hat{q}(s_t,a_t,w_t)\right]{\nabla }_w\hat{q}(s_t,a_t,w_t)$$

Core Idea

Replace $\hat{q}(s_{t+1},a_{t+1},w_t)$ with ${\max }_{a^{\prime }\in \mathcal{A}(s_{t+1})}\hat{q}(s_{t+1},a^{\prime },w_t)$ in VF Sarsa.

Implementation

Compared with Version 1 and Version 3

$$\boxed{\begin{array}{rl}& \text{Algorithm}\text{Q-learning with Function Approximation and }\epsilon \text{-Greedy}\\ & \text{Initialization: }\text{Initial parameters }{\mathbf{w}}_0,\text{ policy }{\pi }_0(a|s)\text{, step size }\alpha \in (0,1],\text{ discount factor }\gamma .\\ & \text{For each episode, do:}\\ & \text{Initialize state }{s}_0.\\ & \text{Select action }{a}_0\text{ using }{\pi }_0(s_0).\\ & \text{While }{s}_t\text{ is not the target state, do:}\\ & \text{Execute }{a}_t\text{, observe }{r}_{t+1},s_{t+1}.\\ & \text{Select }{a}_{t+1}\text{ using }{\pi }_t(s_{t+1}).\\ & \text{Update parameters (update q-value):}\\ & {\mathbf{w}}_{t+1}\leftarrow {\mathbf{w}}_t+\alpha \left[r_{t+1}+\gamma \underset{a^{\prime }\in \mathcal{A}(s_{t+1})}{\max }\hat{q}(s_{t+1},a^{\prime },w_t)-\hat{q}(s_t,a_t,{\mathbf{w}}_t)\right]{\nabla }_{\mathbf{w}}\hat{q}(s_t,a_t,{\mathbf{w}}_t)\\ & \text{Update policy for }{s}_t:\\ & {\pi }_{t+1}(a|s_t)\leftarrow \{\begin{array}{ll}1-\epsilon +\frac{\epsilon }{|\mathcal{A}(s_t)|},& \text{if }a=\arg {\max }_{a^{\prime }}\hat{q}(s_t,a^{\prime },{\mathbf{w}}_{t+1})\\ \frac{\epsilon }{|\mathcal{A}(s_t)|},& \text{otherwise}\end{array}\\ & s_t\leftarrow s_{t+1},a_t\leftarrow a_{t+1}\end{array}}$$

一些观点

赵老师的PPT中说上面这个算法是on-policy的，但自己觉得不对，我觉得是off-policy的，理由如下：

• behavior policy: $\epsilon$-greedy $\arg \max \hat{q}$
• target policy: $\arg \max \hat{q}$

两个policy不一样，所以是off-policy的