Reinforcement Learning Control

PPO Algorithm

for iteration = 1, 2, ... do:
    Collect trajectories using policy π_θ
    Compute advantages A_t using GAE(λ)

    for epoch = 1, ..., K do:
        for minibatch in trajectories do:
            Compute ratio r_t(θ) = π_θ(a|s) / π_θ_old(a|s)
            L_CLIP = min(r_t A_t, clip(r_t, 1-ε, 1+ε) A_t)
            Update θ by maximizing L_CLIP - c₁ L_VF + c₂ S[π_θ]