The Reinforcement Learning Toybox Build Status

A set of games designed for testing deep RL agents.

If you use this code, or otherwise are inspired by our white-box testing approach, please cite our NeurIPS workshop paper:

  author={Foley, John and Tosch, Emma and Clary, Kaleigh and Jensen, David},

How accurate are your games?

Watch four minutes of agents playing each game. Both ALE implementations and Toybox implementations have their idiosyncracies, but the core gameplay and concepts have been captured. Pull requests always welcome to improve fidelity.


Play the games (using pygame)

cargo build --release
cd ctoybox
./start_python breakout
./start_python amidar
./start_python space_invaders

Mac Dev Setup Instructions

Lints and Formatting in Rust

Follow the readme instructions to get rustfmt and clippy.

Then you can check automatically format your files with cargo fmt and peruse the best lints with cargo clippy.

A pre-commit hook will ensure that your code is always properly formatted. To do this, run

git config core.hooksPath .githooks

from the top-level directory. This will ensure that your files are formatted properly pior to committing.


Tensorflow, OpenAI Gym, OpenCV, and other libraries may or may not break with various Python versions. We have confirmed that the code in this repository will work with the following Python versions:

Developing New Games

Initial Setup

  1. Run cargo new tb_<gamename> --lib in the top level of the repository.
  2. Edit the Cargo.toml in the top level of the repository to add tb_<gamename> to the list of members.
  3. Create <gamename>.rs in tb_<gamename>/src.
  4. Create in tb_<gamename>/src.

tb_<gamename> should now have three files in it:,, and <gamename>.rs.

Basic Game Components is the top-level API. Only modules declared here will be exposed to other modules. Copy the following to the top of your

extern crate serde;
extern crate serde_json;
extern crate toybox_core;
extern crate serde_derive;
extern crate lazy_static;
extern crate ordered_float;
extern crate rand;

Every will also have the following declarations:

mod types;
mod <gamename>;

You will export other structures as needed. We recommend that you update this on an as-need basis – Rust will tell you when it can’t find a module, and you should use this to guide when something needs to be exported.


The types module contains all of the intervenable structs. You should update this file concurrently with <> (we recommend having both open at the same time).

Nearly every struct will use the macros: #[derive(Debug, Clone, Serialize, Deserialize)]. The following are the necessary structs for every game:

<GameName>: This struct should be structurally the same as the Config object. It is used to instantiate the game and is required for restarting the game. It contains the initial values of fields that are updated during gameplay.

StateCore: This struct contains any per-frame state snapshots. It will have duplicated fields from <GameName> if those fields are updated during gameplay.

State: This struct will have the form:

pub struct State {
  pub config: `GameName`,
  pub state: StateCore


Necessary components

use serde_json; use rand::seq::SliceRandom;

* Default instantiation: you will need to specify what the default instantiation of the game looks like: 

impl Default for { fn default() -> Self { // any needed computation { // set values here } } }

While magic numbers are okay, a default module (e.g., `mod default`) is better. Including that default module in `` and the `<GameName>` configuration struct is even better.

* `impl`s with `new` methods on each of the structs defined in ``

* `impl toybox_core::Simulation for <GameName>`. The documentation for this trait is in `core/` at the top level of this repository. You will need to consult the ALE documentation in order to implement `legal_action_set`. Each Atari game has a predefined legal action set. If you are writing a new game, you will need to map to the ALE set. 
  * The JSON loading and manipulation code can be dropped in with one textual change:

fn new_state_from_json( &self, json_str: &str, ) -> Result<Box, serde_json::Error> { let state: StateCore = serde_json::from_str(json_str)?; Ok(Box::new(State { config: self.clone(), state, }))

fn from_json(&self, json_str: &str) -> Result<Box, serde_json::Error> { let config: = serde_json::from_str(json_str; Ok(Box::new(config)) }

fn to_json(&self) -> String { serde_json::to_string(self).expect(“ shoulbe JSON-serializable!") } ```

Optional components

Adding human play

Get starting images for reference from ALE

./scripts/utils/start_images --help