Skip to content
Get packs

Harness Eval

Runs OpenHarness features end-to-end against a real codebase using live LLM API calls

Source Download skill
Repository
HKUDS/OpenHarness
Stars
16,157
Licence
MIT
Files
3 files

What it does

This skill guides an AI agent through testing the OpenHarness agent framework itself. It tells the agent to clone an unrelated open-source project as a sandbox workspace, configure a real model API endpoint, and run multi-turn agent sessions that exercise tools like bash, grep, and file reading, then check the tool call traces and output rather than just the model's text answers. It also includes checklists and guidance for interpreting failures.

Great for

  • Maintainers or contributors of OpenHarness who need to verify features work with a real model
  • Developers building their own agent framework who want a pattern for real, non-mocked end-to-end tests

Skip it if

  • Ordinary users who just want to use an agent; this is a developer testing workflow, not a general capability
  • Anyone without an LLM API key and a willingness to spend on model calls, since every test makes real paid requests
  • Running against your own important repositories, because the agent executes shell commands and modifies files in the workspace it is given

Try asking

  • “Test the harness with real model calls on an unfamiliar project”
  • “Run the agent loop integration tests and report which tools were called”
  • “Validate hook blocking and recovery with a real pre-tool hook”
  • “Run a sandbox multi-turn scenario with fail_if_unavailable enabled”
Step-by-step guide: how to use Harness EvalInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 3 original files, the MIT licence and an attribution note.

    harness-eval.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip harness-eval.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip harness-eval.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Use a throwaway workspace clone under /tmp, never a repo with uncommitted work you care about, since the agent runs shell commands there
  • Set API budget limits with your provider first, because long multi-turn runs with max_turns=200 can consume many tokens
  • Read references/test-patterns.md before writing new tests so the make_engine and collect helpers are used correctly

What's inside

  • references/feature-matrix.md
  • references/test-patterns.md
  • SKILL.md

Redistributed unmodified from HKUDS/OpenHarness under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.