Proximal policy optimization

This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Proximal policy optimization" – news · newspapers · books · scholar · JSTOR (October 2022) (Learn how and when to remove this message)

Proximal policy optimization (PPO) is an algorithm in the field of reinforcement learning that trains a computer agent's decision function to accomplish difficult tasks. PPO was developed by John Schulman in 2017,^[1] and had become the default reinforcement learning algorithm at American artificial intelligence company OpenAI.^[2] In 2018 PPO had received a wide variety of successes, such as controlling a robotic arm, beating professional players at Dota 2, and excelling in Atari games.^[3] Many experts called PPO the state of the art because it seems to strike a balance between performance and comprehension.^{[citation needed]} Compared with other algorithms, the three main advantages of PPO are simplicity, stability, and sample efficiency.^[4]

PPO is classified as a policy gradient method for training an agent’s policy network. The policy network is the function that the agent uses to make decisions. Essentially, to train the right policy network, PPO takes a small policy update (step size), so the agent can reliably reach the optimal solution. A too-big step may direct policy in the false direction, thus having little possibility of recovery; a too-small step lowers overall efficiency. Consequently, PPO implements a clip function that constrains the policy update of an agent from being too large or too small.^[4]