Eval-Driven Development for LLM Apps
Builds automated pixie-based evaluation pipelines to score a Python LLM app's behavior with real runs
- Repository
- github/awesome-copilot
- Stars
- 40,703
- Licence
- MIT
- Files
- 15 files
What it does
When installed, the agent sets up an automated quality-testing pipeline for a Python application that calls an LLM. It runs the real app with test inputs, scores outputs with evaluators such as LLM-as-judge or similarity checks, and produces pass/fail results through a pixie test run, then proposes concrete fixes. It is built around pixie-qa, a separate Python package, and starts a local web server so you can view results.
Great for
- Python developers who maintain an LLM-powered app and want regression tests for its behavior
- Teams that need eval datasets and scoring for prompt or context-assembly changes
- People who want an agent to add QA to an existing project without mocking the LLM
Skip it if
- Projects that are not Python or do not call an LLM, since the skill targets Python LLM applications
- Users who only want unit tests with mocked LLM responses, because the skill explicitly forbids mocking the LLM in evals
- Environments where installing packages, running scripts and starting a background web server are not acceptable, since the workflow does all three
Try asking
- “Set up QA with evals for my Python chatbot project”
- “Add evaluators and a golden dataset to test my RAG app's answers”
- “Benchmark my LLM app and tell me what to fix”
- “Run the existing pixie evals again and show me the scores”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 15 original files, the MIT licence and an attribution note.
eval-driven-dev.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip eval-driven-dev.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip eval-driven-dev.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Make sure the app's required environment variables and API keys are set before starting, since the skill stops and asks for help if they are missing
- Provide any spec file or dataset you want used, because the skill reads referenced files first and uses them as the basis for the eval
- Review the generated dataset and evaluators, since LLM-as-judge scores can be noisy
What's inside
- references/1-a-project-analysis.md
- references/1-b-entry-point.md
- references/1-c-eval-criteria.md
- references/2a-instrumentation.md
- references/2b-implement-runnable.md
- references/2c-capture-and-verify-trace.md
- references/3-define-evaluators.md
- references/4-build-dataset.md
- references/5-run-tests.md
- references/6-analyze-outcomes.md
- references/evaluators.md
- references/runnable-examples/cli-app.md
- references/runnable-examples/fastapi-web-server.md
- references/runnable-examples/standalone-function.md
- SKILL.md
Redistributed unmodified from github/awesome-copilot under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.