PG idea

Idea of Policy Gradient

Introduction

启发

受之前tabular的启发，我们下定决心抛弃掉表格化的方法

• 于是就有了本来使用表格存储的state value(index is state) ⇒ 使用函数映射的state value(input)
• 那我们的想法是能不能也同时 将表格存储的action probabilities(index is state) ⇒ 使用函数映射的action probabilities(input is state)
• action probabilities是一个m维的向量，m是action的数量
• action probabilities = policy

• Previously, the action probabilities of all states are stored in a table ($\pi$).

State/Action	$a_1$	$a_2$	$a_3$	$\cdots$	$a_m$
$s_1$	$\pi (s_1\mid a_1)$	$\pi (s_1\mid a_2)$	$\pi (s_1\mid a_3)$	$\cdots$	$\pi (s_1\mid a_m)$
$⋮$	$⋮$	$⋮$	$⋮$	$\ddots$	$⋮$
$s_n$	$\pi (s_n\mid a_1)$	$\pi (s_n\mid a_2)$	$\pi (s_n\mid a_3)$	$\cdots$	$\pi (s_n\mid a_m)$

• Now, policies can be represented by parameterized functions:

$$\pi (a\mid s,\theta )=\pi (a,s,\theta )={\pi }_{\theta }(a\mid s)={\pi }_{\theta }(a,s)$$

where $\theta \in {\mathbb{R}}^m$ is a parameter vector.

• the function can be, e.g., a neural network, whose input is $s$, output is the probability to take each action, and parameter is $\theta$.
• advantage: when the state space is large, the tabular representation will be of low efficiency in terms of storage and generalization.

Difference between tabular and function representations

1. how to define optimal policy?
   • tabular case: a policy $\pi$ is optimal if it can maximize every state value
   • function case: a policy $\pi$ is optimal if it can maximize certain scalar metrics
2. how to access the probability of an action?
   • tabular case: the probability of taking action $a$ at state $s$ can be directly accessed by looking up the tabular policy
   • function case: we need to calculate the value of $\pi (a\mid s,\theta )$ given the function structure and the parameter
3. how to update policies?
   • tabular case: a policy $\pi$ can be updated by directly changing the entries in the table
   • function case: a policy $\pi$ cannot be updated in this way anymore. Instead, it can only be updated by changing the parameter $\theta$

Basic idea of policy gradient

1. Define a metric (or objective function) to represent the quality of a policy: $J(\theta )$
2. Use gradient-based optimization algorithms to search for the optimal policy: $${\theta }_{t+1}={\theta }_t+\alpha {\nabla }_{\theta }J({\theta }_t)$$

Although the idea is simple, the implementation is not trivial.

1. What appropriate metrics should be used? See PG metric.
2. How to calculate the gradient of $J(\theta )$. See PG metric gradient