Skip to content
Get packs

Guide · Agent skill

How to use the Harness Eval agent skill: install and run tests

Updated · written with AI from the project's own documentation and checked against our catalogue data

Harness Eval is an agent skill from the OpenHarness project that teaches an AI agent how to test the OpenHarness framework itself against a real model. It is written for maintainers, contributors, and developers building their own agent frameworks who want end-to-end tests with no mocks. This guide explains what the skill makes the agent do, how to install it, how to trigger it, and what you need in place before every run makes real, paid model calls.

Harness EvalHKUDS/OpenHarness · MIT

Quick start

  1. 1Download and unzip. Download the harness-eval ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder.
  2. 2Check prerequisites. Confirm a Python environment with OpenHarness installed, a real LLM API key and endpoint, and Git are available.
  3. 3Export API settings. Set ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, and ANTHROPIC_MODEL for the provider you are using.
  4. 4Clone an unrelated workspace. Clone an unfamiliar open-source project such as AutoAgent into a temporary directory to use as the sandbox workspace.
  5. 5Ask for the test. Request something like "run agent loop tests" or "verify end-to-end" so the agent loads the skill.
  6. 6Review traces and results. Check the tool call list and output files for each scenario, then classify any failure before changing code.

What an agent skill is and what harness-eval makes the agent do

An agent skill is a folder containing a SKILL.md file and optional reference files and scripts. When a request matches the skill's description, the agent loads those instructions and follows them for that task.

The harness-eval skill tells the agent to validate OpenHarness features by running real agent loops against an unfamiliar codebase with actual LLM API calls. Each test runs the full path from API client to model, tool calls, tool execution, and result.

  • Clone an unrelated open-source project as a sandbox workspace (the skill's example is AutoAgent), not OpenHarness itself.
  • Configure a real model API endpoint instead of mocks.
  • Run multi-turn sessions that need prior context, with at least 2 turns per scenario.
  • Check the tool call traces and output files, not only the model's text answers.
  • Classify failures, such as timeouts, MaxTurnsExceeded, or missing sandbox tools, before changing code.

Install the harness-eval skill from the ZIP

Download the ZIP for harness-eval from this page and unzip it. The skill folder contains SKILL.md, references/feature-matrix.md, and references/test-patterns.md.

For a personal install, place the unzipped folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder instead.

If you use the Claude apps rather than a local folder, upload the skill under Settings → Capabilities → Skills.

  • Personal install: ~/.claude/skills/
  • Project install: .claude/skills/ inside your project
  • Claude apps: Settings → Capabilities → Skills, then upload

Trigger the skill with natural requests

You do not run the skill as a command. You ask for the kind of work it covers, and the agent loads it when the request matches the skill's description. The description lists phrases that match, so using similar wording helps.

Good triggers come from the source description and are specific about real model calls and end-to-end checks.

  • "Test the harness"
  • "Run integration tests"
  • "Validate features with real API"
  • "Test with real model calls"
  • "Run agent loop tests"
  • "Verify end-to-end"

Set up the environment before a run

The skill's examples use Anthropic-compatible API settings. You export your key, the endpoint base URL, and the model name, then clone the workspace repository with Git. The skill's example endpoint and model are placeholders you can replace with any provider you have access to.

For sandbox scenarios, the skill asks you to install and verify the real runtime first, including Node.js with @anthropic-ai/sandbox-runtime, bubblewrap, and ripgrep. It then runs a minimal check through OpenHarness, not only the raw srt command.

  • Python environment with OpenHarness installed
  • A real LLM API key and endpoint
  • Git, to clone the external workspace
  • For sandbox scenarios: Node.js with @anthropic-ai/sandbox-runtime, bubblewrap, and ripgrep
export ANTHROPIC_API_KEY=sk-xxx export ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic # or any provider export ANTHROPIC_MODEL=kimi-k2.5 git clone https://github.com/HKUDS/AutoAgent /tmp/eval-workspace

Run the example test scenarios

The skill recommends long-horizon scenarios on an unfamiliar repository. The architecture_multiturn scenario maps architecture over several turns and succeeds only if bash, glob, grep, and read_file all appear, with no timeout and no MaxTurnsExceeded. The hook_block_and_recover scenario blocks a bash call with a pre-tool hook and checks that the model switches to glob, grep, or read_file.

The sandbox_multiturn scenario enables sandbox settings with fail_if_unavailable set to true. Its first prompt must start with one shell command, and its second must reuse those findings.

  • Keep max_turns at the product default of 200 unless you intentionally want a tighter bound.
  • Use per-prompt timeouts of roughly 240 to 600 seconds for real exploration.
  • Keep polling long-running sessions until they finish instead of stopping after a long pause.
python tests/test_merged_prs_on_autoagent.py python tests/test_real_large_tasks.py python tests/test_hooks_skills_plugins_real.py python -m pytest tests/ -q -k "not autoagent"

Read the results correctly

A pass only counts when tool calls back it up. A pass with no tool calls means the model answered from its own knowledge, so rewrite the prompt to force tool use. A failure with missing sandbox tools is an environment problem, not a feature regression.

If a run times out with max_turns still at 200, the next suspect is the wall-clock timeout rather than the turn count.

  • PASS with tool calls: the feature works end-to-end.
  • PASS without tool calls: rewrite the prompt to force tool use.
  • FAIL with exception: read the traceback for a code bug.
  • FAIL with wrong output: check the system prompt and tool schemas.
  • Timeout: increase the timeout or simplify the prompt.

Safety notes before you run it

Every test makes real, paid requests to your model provider, so cost grows with the number of runs. The agent also executes shell commands and modifies files in the workspace it is given, and the skill uses the network.

Use the external sandbox repository the skill specifies, never your own important repositories, and never OpenHarness itself, since the agent could modify its own running code.

  • Real API calls cost money on every run.
  • The agent runs shell commands and edits files in its workspace.
  • The workflow uses network access for the API and for cloning.
  • Keep test workspaces disposable and out of CI unless you guard them, for example with pytest.mark.skipif.

What people use it for

  • Verifying that multi-turn memory, tool chaining, and error recovery work with a real model before a release.
  • Checking that a pre-tool hook blocks a bash call and the model recovers using glob, grep, or read_file.
  • Confirming that a skill or plugin is loaded and used during a real agent loop.
  • Building a pattern for non-mocked end-to-end tests in your own agent framework.

Frequently asked questions

What is the harness-eval agent skill?

It is an agent skill from the OpenHarness project that guides an AI agent through end-to-end testing of OpenHarness features. The agent runs real multi-turn sessions with actual LLM calls against an unfamiliar repository and checks tool traces.

How do I install the harness-eval skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for a personal install or into a project's .claude/skills/ folder. You can also upload it in the Claude apps under Settings → Capabilities → Skills.

What do I need to run harness-eval?

You need a Python environment with OpenHarness installed, a real LLM API key and endpoint, and Git to clone the workspace repository. Sandbox scenarios also need Node.js with @anthropic-ai/sandbox-runtime, bubblewrap, and ripgrep.

Is harness-eval free to use?

The source does not state a price for the skill. It is listed under the MIT licence. Each test makes real model API calls that you pay for through your own provider account.

Can I use harness-eval on my own repository?

The skill advises against it. The agent executes shell commands and modifies files in the workspace it is given, so it should run against a disposable external project, not your important repositories.

Why does my test fail or time out?

Check first whether max_turns was lowered manually, since the default is 200. If the run still times out, the wall-clock limit or an overly broad prompt is the likely cause. Missing srt, bwrap, or rg means the sandbox environment is incomplete, not that OpenHarness is broken.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Harness Eval suits OpenHarness maintainers and developers who want real, non-mocked end-to-end tests and have an API key and budget for paid model calls. If you only want to use an agent for everyday work, or you cannot risk the agent modifying files in a workspace, pick a different skill or a general-purpose agent setup.

Download the skill