How to use the BigCode code model benchmarking agent skill

What is an agent skill and what does this one do?

An agent skill is a folder of instructions and reference files that an AI agent loads when a task matches the skill's description. This skill's main file, SKILL.md, tells the agent how to run the BigCode Evaluation Harness: how to set it up, choose benchmarks such as HumanEval (Python) or MultiPL-E (18 languages), launch the evaluation with the right flags, and read the pass@k scores it produces.

The skill also includes reference notes in references/benchmarks.md, references/custom-tasks.md and references/issues.md covering benchmarks, creating custom tasks and known problems.

• Sets up the harness and its environment
• Picks benchmarks and launches runs with the correct flags
• Reports pass@1, pass@10 and pass@100 results in a JSON file
• Explains known installation and troubleshooting issues

Install the evaluating-code-models skill

Download the ZIP for this skill from this page and unzip it. For a personal setup, place the unzipped folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

The skill folder should contain SKILL.md and the references folder with benchmarks.md, custom-tasks.md and issues.md.

• Personal install: ~/.claude/skills/
• Project install: .claude/skills/ inside your project
• Claude apps: Settings → Capabilities → Skills, then upload the ZIP

Check the requirements before you start

The skill runs scripts and uses the network, and the benchmarks need real hardware. The skill's source lists the following requirements. Confirm them before asking the agent to run an evaluation.

• Python with pip and git
• The bigcode-evaluation-harness repository and its Python package
• accelerate, configured with accelerate config
• transformers>=4.25.1, datasets>=2.6.1, accelerate>=0.13.2
• GPU hardware suitable for the chosen model size
• Docker, for the recommended sandboxed MultiPL-E execution
• A HuggingFace access token for gated models, or the --use_auth_token flag

Trigger the skill with natural requests

You do not need to type a command name. Ask for the task in plain language that matches the skill's description. Useful phrasings include benchmarking a code model, comparing coding abilities across models, testing multi-language support, or measuring code generation quality.

Be specific about the model and the benchmark. The agent will still need the model ID, the tasks you care about and whether you want results saved to a file.

• Benchmark bigcode/starcoder2-7b on HumanEval and report pass@1 and pass@10.
• Compare these three code models on HumanEval and MBPP and make a comparison table.
• Check whether my fine-tuned code model regressed on coding tasks using MultiPL-E Python and JavaScript.
• Set up the BigCode evaluation harness on this machine and tell me what I need to install first.

Run an evaluation and read the results

A typical HumanEval run uses accelerate launch main.py with the model, the task, generation settings and --allow_code_execution. Adding --save_generations and --metric_output_path writes the scores to a JSON file. The source example writes to results/starcoder2-humaneval.json, and that file holds pass@1, pass@10 and pass@100 values plus the config used.

The skill's workflows cover four cases: a standard code benchmark run, a multi-language MultiPL-E run that generates on the host and executes inside Docker, an instruction-tuned model run, and a multi-model comparison script.

• HumanEval: 164 handwritten Python problems, function completion
• HumanEval+: same 164 problems with 80× more tests
• MBPP: 500 crowd-sourced problems at entry-level difficulty
• MBPP+: 399 curated problems with 35× more tests
• MultiPL-E: HumanEval and MBPP translated into 18 languages

Run multi-language tests safely with Docker

For MultiPL-E, the skill recommends generating solutions on the host without execution using --generation_only, then running the tests inside the evaluation-harness-multiple Docker image. This keeps model-written code away from your main system. The skill also lists the supported languages, including Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R and Racket.

• Step 1: generate with --generation_only and --save_generations
• Step 2: pull ghcr.io/bigcode-project/evaluation-harness-multiple and run tests in the container
• Use the :ro read-only mount for the generations file, as the source example does

Safety notes: scripts, network and generated code

This skill runs scripts and uses the network. Model downloads and dataset access go over the network, and the harness executes model output when code execution is enabled. The flag --allow_code_execution is required for the unit-test benchmarks, so treat it as a deliberate choice.

Run untrusted generated code only inside a sandbox. The recommended route in the source is Docker for MultiPL-E. Do not run unfamiliar models with --trust_remote_code on a personal machine without understanding what their code does.

• Use Docker for execution-based benchmarks when possible
• Keep HuggingFace tokens out of shared files; use --use_auth_token only when needed
• Check GPU memory against the model size before starting a long run

Tips for reliable results

Use the same generation settings for every model you compare, or the comparison is not fair. Fix the temperature, n_samples and batch size across runs. If installation fails with PyTorch or CUDA conflicts, the skill's issues notes recommend installing PyTorch separately first, matching your CUDA version, and then installing the harness.

DS-1000 has its own requirement in the source: it needs Python 3.7.10 in a separate conda environment, with the ds1000 extra installed.

• Save generations so you can re-score them without regenerating
• Install PyTorch before the harness when you hit CUDA errors
• Use a separate conda environment for DS-1000