Brain

Episodic Policy Gradient Training

Published in AAAI (Oral), 2022

We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly. Unlike other hyperparameter searches, we formulate hyperparameter scheduling as a standard Markov Decision Process and use episodic memory to store the outcome of used hyperparameters and their training contexts. At any policy update step, the policy learner refers to the stored experiences, and adaptively reconfigures its learning algorithm with the new hyperparameters determined by the memory. This mechanism, dubbed as Episodic Policy Gradient Training (EPGT), enables an episodic learning process, and jointly learns the policy and the learning algorithm’s hyperparameters within a single run. Experimental results on both continuous and discrete environments demonstrate the advantage of using the proposed method in boosting the performance of various policy gradient algorithms.

Code•Talk•Slides•PDF•Link

epgt


Hyper-RL structure. The hyper-state (green circle) is captured from the PG models’ parameters and gradients at every Hyper-RL step (1). Given the hyper-states, the hyper-agent takes hyper-actions, choosing hyperparameters for the PG method to update the models (2). The update lasts U steps. After the last update step (3), the RL agent starts environment phase with the current policy, collecting an empirical return G after T environment steps (4). G is used as the hyper-reward for the last policy update step (blue diamond) (5). Other update steps (red diamond) are assigned with hyper-reward 0