How to use the Audit a pytest suite agent skill

What is an agent skill, and what does this one do?

An agent skill is a folder of instructions and optional scripts that an AI agent such as Claude can load when a task matches it. This skill is defined in a single file, SKILL.md, and is marked with the category testing and the tags pytest, mutation-testing and flaky-tests.

When installed, the agent runs a pinned measurement tool against your pytest suite. It asks whether the suite would notice broken code (mutants), which tests cost time without adding detection, and which tests are flaky or order-dependent. It then writes a report and explains the findings.

The skill is report-only. The agent does not edit, delete, skip or quarantine tests, and does not change source code unless you ask for a specific change after reading the report.

Install the agentic-test-audit skill

Option 1: download the ZIP from this page and unzip it into your personal skills folder, ~/.claude/skills/, so it is available in all your projects.

Option 2: unzip it into a project's .claude/skills/ folder so it applies only to that repository. This is a good choice when you want teammates to get the same skill through the repo.

Option 3: in the Claude apps, open Settings, go to Capabilities, then Skills, and upload the ZIP there.

• Personal: ~/.claude/skills/
• Per project: .claude/skills/ inside the repository
• Claude apps: Settings → Capabilities → Skills

Check your project before the first run

The skill only applies to Python projects that use pytest. Confirm that the project has a working test environment. The runner needs Python 3.12 or newer, pytest-cov in the project's environment, and pytest-randomly if you want order checks. It also needs uv, because the runner is started with uvx, and network access to GitHub to fetch the pinned runner.

If the runner cannot find your test setup, you can add a test-suite-audit.toml file. The agent will ask you before writing one. Its keys are python, source, tests, pytest_args, env, workers and protected. The protected key lists test node ID prefixes or marker names that must never be proposed for removal.

If pytest-cov is missing, the agent will say that coverage is not assessed and offer the install command. It will not install anything without asking.

• Python 3.12 or newer
• pytest and pytest-cov in the project environment
• pytest-randomly for order-dependent test checks
• uv, to run the pinned runner with uvx
• Network access to GitHub

Ask for an audit in plain language

You do not need a special command. Ask the agent to audit, review, speed up, prune or strengthen your pytest tests, or ask which tests are useful, flaky or slow. The skill will then pick a mode.

The default mode is quick, which takes about 5 minutes: one baseline run, timings, coverage and overlap, with no mutants. The deep mode runs for 30 minutes by default and adds file-order and random-order runs, coverage, every deletion mutant and a seeded sample of classical mutants. The agent states the budget before running.

To check a single change instead of the whole suite, use the related agentic-verify skill.

• Audit my pytest tests and tell me which ones are flaky.
• Run a quick audit of this Python project.
• Do a deep audit of the test suite with a 30-minute budget.
• Which of my tests are slow and add nothing to detection?

Read the report and check removal candidates

The agent reads report.json for full evidence and report.md for the summary. The findings are grouped under three headings: whether the suite catches broken code, whether an agent can run it on every change, and whether it is bloated. Each finding carries a label and a next validation step.

If a deep audit proposes removal candidates, the agent treats them as an upper bound and runs prune-check on them using held-out mutants. The verdict is either refuted, with the kills or lines lost, or not refuted. A not refuted result is not the same as safe, and the skill never presents a refuted set as removable.

A surviving mutant is a candidate gap, not proof of a bug. Equivalent mutants need human review.

• Refuted: the tests named are sole killers and must be kept.
• Not refuted: the set survived the check, but this is not proof that removal is safe.
• The runner writes its output to .test-suite-audit/ in your project. Consider adding that folder to .gitignore.

Example commands the runner uses

The agent runs the pinned runner through uvx. The three entry points are quick or deep audits, prune-check for a removal set, and replay for a single mutant. The runner works in a disposable copy and writes only to .test-suite-audit/. The last line of its output is JSON with the report paths.

A deep audit followed by a removal check looks like the second example below. The first example is a 5-minute quick audit.