Concept policy

Policy

$\text{Policy}$: At state $s$, the probability to choose action $a$ is $\pi (a\mid s)$

Policy tells the agent what actions to take at a state.

Intuitive representation: the arrows demonstrate a policy

Math representation: using conditional probability Deterministic policy at state $s_1$

$$\begin{array}{rl}\pi (a_2\mid s_1)& =1\\ \pi (a_i\mid s_1)& =0,\forall i\ne 2\end{array}$$

• ${\sum }_{i=1}^5\pi (a_i\mid s_1)=1$

Stochastic policy at state $s_1$

$$\begin{array}{rl}\pi (a_2\mid s_1)& =\pi (a_3\mid s_1)=0.5\\ \pi (a_i\mid s_1)& =0,\forall i\ne 2,3\end{array}$$

如何在编程中表示policy

一般用向量或者矩阵来表示，直观的，我们先有tabular representation, a 9x5 table，所以可以直接用矩阵表示，policy.shape=(9,5), policy.sum(dim=1)=[1,1,1,...,1]