Curious Agents Saga: Part 3, Beyond Surprise: Direct and Causal Exploration in Deep Reinforcement Learning

Table of Content

• Reflection on Intrinsic Motivation
• Direct Exploration
  • Replay Memory Focused on Exploration
  • Performance-based Replay Memory
• Causal Exploration
  • What is Causality?
  • Dependency Test
  • Potential Outcome
  • Structural Causal Model

Reflection on Intrinsic Motivation

In the previous article, we reviewed an essential exploration framework called intrinsic motivation, which is widely used in deep RL due to its scalability. Within the framework, surprise and novelty are the medium for exploration. Regarding surprise, memory is often hidden within dynamics models, memorizing observed data to enhance predictive capabilities. This type of memory tends to be long-term, semantic, and slow to update, akin to a careful archivist meticulously preserving information.

On the other hand, novelty takes a more straightforward approach to memory. Memory is delineated here, resembling a slot-based matrix, a nearest neighbor estimator, or a simple counter. This memory is typically short-term, instance-based, and highly adaptive to environmental changes, acting more like a dynamic and responsive agent ready to adjust to new inputs swiftly.

They all begin with the memory origin, employing surprise or novelty mechanisms to create intrinsic rewards that guide the exploration of the RL agent. While being so convenient and easy to use, two major issues have hindered the ability of the framework to explore effectively:

Detachment: lose track of interesting areas to explore.

Derailment: prevent it from utilizing previously visited states.

🧠 But what other alternatives could there be to overcome these issues?

Read the full article