Concept Markov Decision Process

Markov Decision Process(MDP)

Key elements of MDP

• Sets(3个集合):
  • State: the set of states $\mathcal{S}$
  • Action: the set of actions $\mathcal{A}(s)$ is associated for state $s\in \mathcal{S}$
  • Reward: the set of rewards $\mathcal{R}(s,a)$
• Probability distribution(两个分布):
  • state transition probability: $p(s^{\prime }\mid s,a)$
  • reward probability: $p(r\mid s,a)$
• Policy: $\pi (a\mid s)$
• Markov property: memoryless property

$$\begin{array}{rl}p(s_{t+1}\mid a_{t+1},s_t,\dots ,a_1,s_0)& =p(s_{t+1}\mid a_{t+1},s_t)\\ p(r_{t+1}\mid a_{t+1},s_t,\dots ,a_1,s_0)& =p(r_{t+1}\mid a_{t+1},s_t)\end{array}$$

为什么要写成 $\mathcal{A}(s)$

意味着不同state可以有不同action space

如果以后别人问什么是MDP怎么办？

就回答：MDP就是markov decision process，废话一样，其实可以从题意出发

• markov: 就是指 马尔可夫性质，当前状态采取动作之后，下一状态以及得到的奖励的概率分布和历史无关
• decision: 就是指policy，策略就是指当前状态采取某一个动作的概率
• process: 包含state, action, reward三个set和state trainsition prob和reward prob

Markov decision process becomes Markov process once the process is given.