Proximal Policy Approximation

Let $r_t(\theta )$ denote the probability ratio $r_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t|s_t)}$, so $r({\theta }_{old})=1$. TRPO maximizes a “surrogate” objective

$$L^{\text{CPI}}(\theta )={\hat{E}}_t\left[\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t|s_t)}{\hat{A}}_t\right]={\hat{E}}_t[r_t(\theta ){\hat{A}}_t].$$

The superscript CPI refers to conservative policy iteration. Without a constraint, maximization of $L^{\text{CPI}}$ would lead to an excessively large policy update; hence, we now consider how to modify the objective, to penalize changes to the policy that move $r_t(\theta )$ away from 1. The main objective proposed is the following:

$$L^{\text{CLIP}}(\theta )=\hat{E}\left[\min \{r_t(\theta ){\hat{A}}_t,\text{clip}(r_t,1-ϵ,1+ϵ)(\theta ){\hat{A}}_t\}\right]$$

where $ϵ$ is a hyperparameter. The motivation for this objective is as follows. The first term inside the min is $L^{\text{CPI}}$. The second term, $\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t$, modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving $r_t$ outside of the interval $[1-ϵ,1+ϵ]$. Finally, we take the minimum of the clipped and unclipped objective, so the final objective is a lower bound (i.e., a pessimistic bound) on the unclipped objective. With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse.

Using a truncated version of generalized advantage estimation(GAE), the advantage estimator is:

$${\hat{A}}_t={\delta }_t+(\gamma \lambda ){\delta }_{t+1}+\cdots +(\gamma \lambda {)}^{T-t+1}{\delta }_{T-1}$$

where ${\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)$ is the TD residual.

Readme

• The N Implementation Details of RLHF with PPO