Proximal Policy Optimization

Definition

Although TRPO has achieved monotonic improvement, its implementation and computation are complicated due to KL constraint. PPO removed the KL constraint term in the optimization problem of TRPO by introducing the clipped surrogate objective, which provides penalty when the policy update is too high.

PPO apply clipping to the policy ratio $\frac{{\pi }_{\theta }(a|s)}{{\pi }_{{\theta }_{\text{old}}}(a|s)}$. The cliiped surrogate function is defined as $L^{CLIP}(\theta )=E[\min (r(\theta )A_{{\theta }_{\text{old}}}(s,a),\mathrm{clip}(r(\theta ),1-ϵ,1+ϵ)A_{{\theta }_{\text{old}}}(s,a))]$ where $r(\theta )=\frac{{\pi }_{\theta }(a|s)}{{\pi }_{{\theta }_{\text{old}}}(a|s)}$ is the policy ratio, and $ϵ$ is a hyperparameter that determines the clipping range.

$L^{CLIP}(\theta )$ satisfies $L^{CLIP}(\theta )\le L^{TRPO}(\theta )$, so we can still use MM Algorithm.

To minimize the value function loss simultaneously and to encourage exploration, PPO appends additional terms to the objective function. $L^{CLIP+VF+S}(\theta )=L^{CLIP}-c_1{L}^{VF}(\theta )+c_{2}S[{\pi }_{\theta }](s)$ Where

• $L^{CLIP}(\theta )$ is the clipped surrogate objective
• $L^{VF}(\theta )=(V_{\text{target}}-V_{\theta }(s){)}^2$ is the value function loss
• $S[{\pi }_{\theta }](s)=-\sum _a{\pi }_{\theta }(a|s)\ln {\pi }_{\theta }(a|s)$ is the Entropy bonus terms.
• $c_1,c_2$ are hyperparameter that controls the weight of the losses.