Skip to content
Get packs

BDH-CQ

Generates fresh ARC-AGI-1-style puzzle tasks to test AI models on unseen problems.pathwaycom/arc-task-gen

GitHub Download source
Stars
10.9k
Forks
76
Licence
MIT
Language
Python
Last update
3 months ago
Type
Dev tool

What it is

This repository generates new ARC-AGI-1-style puzzle tasks that look statistically similar to the public ARC-AGI-1 dataset. ARC tasks ask a model to infer a visual rule from a few input-output examples. The generated tasks are meant to give researchers a private test set, so model results can be compared against the public benchmark on puzzles the model is less likely to have seen. The output uses the standard ARC JSON format.

  • Generate a set of ARC-format tasks in tasks.json for evaluating a model
  • Run an existing ARC evaluation harness against the generated tasks
  • Compare model performance on public ARC-AGI-1 tasks versus generated tasks

Great for

  • Researchers evaluating few-shot reasoning models on ARC-style puzzles
  • Teams who want a private test set alongside the public ARC-AGI-1 evaluation set
  • People comparing public benchmark scores against results on newly generated tasks
  • Readers who want to see how the BDH-CQ paper's evaluation was set up

Not ideal for

  • People looking for a general-purpose AI model or chatbot; the README describes a task generator, not a model
  • Anyone expecting a ready-made puzzle game; the output is data for evaluation harnesses
  • Users who need a fully documented install path; the README points to a separate instructions.md for generation steps that were not included here
Step-by-step guide: how to use BDH-CQInstall, first run, examples, tips and FAQs

How to get it

  1. 1

    Clone and build from source

    You'll need: Python

    Clone the repository

    git clone https://github.com/pathwaycom/arc-task-gen
  2. 2

    Run it

    • Clone the repository
    • Open instructions.md in the repo and follow its steps to generate tasks
    • Check that the output tasks.json has train and test fields in ARC format
    • Feed the generated tasks into an ARC evaluation harness
  3. 3

    Get the most out of it

    • Use the generated tasks as a private complement to the public ARC-AGI-1 evaluation set, as the README suggests
    • Keep generated tasks separate from any training data to preserve their value as unseen test items
    • Read the accompanying paper linked in the README for the evaluation methodology

Downloads

Source code (.zip)main branch · straight from GitHub

Strengths

  • Output follows the standard ARC JSON format, so it is said to work with existing evaluation harnesses
  • Aims to provide a private evaluation set with properties similar to the public one
  • MIT licensed

Trade-offs

  • The README is truncated and the generation steps live in instructions.md, which was not provided, so setup details are unverified
  • Framing is tied to the BDH-CQ model and its paper; the tool's value outside that context is less clear from the README
  • Results and performance claims in the README come from the project's own authors and have not been verified here

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.