Temporal-Difference Learning and the Importance of Exploration: An Illustrated Guide

Temporal-Difference (TD) learning methods are a popular subset of Reinforcement Learning that combine aspects of Monte Carlo and Dynamic Programming to learn without a perfect model of an environment. This approach is demonstrated by comparing different TD algorithms, specifically model-free Q-learning and model-based Dyna-Q and Dyna-Q+, within a custom grid world. The environment is designed to change after a set number of episodes, creating a new optimal path to test the algorithms' ability to adapt. The experiment highlights how model-free Q-learning can converge to a sub-optimal strategy, while model-based methods incorporating planning are more sample-efficient and better at handling changes, underscoring the importance of continuous exploration.