Guide · Open source
How to use reinforcement-learning-an-introduction: Python code for Sutton & Barto
Updated · written with AI from the project's own documentation and checked against our catalogue data
reinforcement-learning-an-introduction is an open-source Python repository that replicates experiments and figures from Sutton and Barto's book Reinforcement Learning: An Introduction (2nd Edition). The code is organized by chapter, so you can match each script to the book's text. This guide shows you how to get the code, which chapters and figures it covers, and what it can and cannot help you with.
reinforcement-learning-an-introductionShangtongZhang/reinforcement-learning-an-introduction · ★ 14.8k · MIT Install options & downloads Quick start
- 1Clone the repository. Run the git clone command from the install section to copy the code to your machine.
- 2Read the chapter that matches your book chapter. Pick the chapter folder that corresponds to the section of Sutton and Barto you are studying.
- 3Compare the output with the book figures. Run the code and check the plotted results against the figures listed in the README.
- 4Change one parameter at a time. Adjust values such as step size or epsilon, then observe how the curves change compared with the book.
- 5Open an issue if something breaks. If the code does not behave as described, report it on the GitHub issues page as the README requests.
What the repository contains
The repository is a Python replication of the book, arranged by chapter. Its README lists figures and examples from Chapter 1 (Tic-Tac-Toe) through Chapter 12, including bandit problems, gridworld dynamic programming, Monte Carlo methods, temporal-difference learning, n-step methods, Dyna-Q, and function approximation on tasks such as Mountain Car and Baird's Counterexample.
The README links each figure to a plot image in the repository's images folder, so you can compare your output against the book's published figures.
- Chapter 2: 10-armed testbed, epsilon-greedy, UCB and gradient bandit figures
- Chapter 6: Sarsa on windy grid world, cliff walking, Q-learning vs Double Q-learning
- Chapter 8: Dyna-Q, prioritized sweeping and Dyna maze figures
- Chapter 10: semi-gradient Sarsa on Mountain Car and access-control queuing
Install the code by cloning the repository
The README's install method is to clone the repository with git. Python is the only prerequisite the source names. Run the following command in a terminal, then open the cloned folder.
The README does not list any extra packages, environment setup or run commands, so check the chapter files for their imports and dependencies before you run them.
- Make sure Python and git are installed on your machine.
- Clone the repository using the command below.
- Open the chapter folder that matches the part of the book you are reading.
git clone https://github.com/ShangtongZhang/reinforcement-learning-an-introduction.gitUse it alongside the book
The code is most useful when read next to the book. Find the figure number in the book, look up the matching chapter folder, and run the code to see the plotted result. Seeing the curves for Sarsa, Q-learning or Dyna-Q change as you alter a parameter often makes the trade-offs easier to follow.
The author says there are no exercise answers in the repository. Use the code to explore the algorithms, and work exercises yourself.
- Start with Chapter 2 bandit figures if you are new to RL.
- Compare Sarsa and Q-learning on the windy grid world and cliff walking figures.
- Check Dyna-Q results in Chapter 8 against the planning-step experiments.
Get help and report problems
The README asks readers to open an issue on GitHub for questions about the code or to report a bug. It asks people not to email the author directly about these matters.
The repository has a Travis CI build badge in the README, which indicates it has been set up for automated builds.
- Open an issue on the GitHub repository for code questions or bugs.
- Do not email the author directly about code problems, per the README.
What people use it for
- Reproducing the 10-armed testbed bandit figures from Chapter 2 while reading the book.
- Comparing Sarsa and Q-learning on the windy grid world and cliff walking tasks from Chapter 6.
- Studying how Dyna-Q's planning steps affect learning curves in Chapter 8.
- Viewing semi-gradient Sarsa results on the Mountain Car task in Chapter 10.
Frequently asked questions
Is reinforcement-learning-an-introduction free to use?
The repository is released under the MIT licence, as listed on its project page. Check the LICENSE file in the repository for the full terms before you reuse the code.
What do I need to install reinforcement-learning-an-introduction?
The source names Python as the prerequisite and gives a git clone command for getting the code. It does not list specific Python packages, so check the imports in each chapter file.
Does this repository include answers to the exercises in the book?
No. The author states in the README that they do not have exercise answers for the book. The repository replicates the book's examples and figures instead.
Can I use this for a production reinforcement learning project?
The README presents the project as a replication of the book, not a general-purpose library. For production work, consider a library such as Stable-Baselines3 or Gymnasium, which are comparable projects listed for this item.
What algorithms does the code cover?
The chapters cover bandit methods such as epsilon-greedy, UCB and gradient bandits, plus dynamic programming, Monte Carlo methods, Sarsa, Q-learning, Double Q-learning, Dyna-Q and semi-gradient methods.
Where should I report a bug or ask a question?
Open an issue on the GitHub repository. The README asks readers not to email the author directly about code confusion or bugs.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This repository suits students reading Sutton and Barto who want runnable Python versions of the book's examples and plots. It is not a good choice if you need exercise answers or a maintained library for production use, where Stable-Baselines3 or Gymnasium are better fits.
Install options & downloads