Guide · Agent skill
How to use the Test Flakiness Detective agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The test-flakiness skill is a Claude agent skill that reads saved test-run logs from your CI history, finds tests that sometimes pass and sometimes fail with no code change, and suggests a likely cause and whether to quarantine or fix each one. It is written for game development teams using Godot (gdUnit4), Unity or Unreal. After this guide you will know what the skill does, how to install it, how to ask for it in plain language, and what it needs before it gives useful results.
Test Flakiness DetectiveDonchitos/Claude-Code-Game-Studios · MIT Download the skill Quick start
- 1Download and unzip. Download the ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder.
- 2Confirm your project layout. Check that tests/regression-suite.md and the .claude/docs files the skill references are present in your project.
- 3Collect at least three runs. Save the output of several test runs, or save CI logs into test-results/, so there is enough history to compare.
- 4Ask for a scan. Type /test-flakiness scan or describe the task in plain language, such as asking the agent to find flaky tests in your CI logs.
- 5Review the classifications. Check each flagged test, its fail rate, and the suggested likely cause before accepting any change.
- 6Apply quarantine or fixes. Accept quarantine for high-flakiness tests and fix moderate ones directly; the root-cause work remains yours.
What the skill makes the agent do
A flaky test is one that sometimes passes and sometimes fails without any code change. When installed, the agent reads test result files from several runs and compares them, so it can point out tests whose results change between runs.
For each flaky test the skill classifies a likely cause, such as timing or async issues, test order dependency, random seeds, resource leaks, external state, floating-point comparisons, or scene and prefab load races. It then recommends either skipping the test temporarily or fixing it.
- Reads saved JUnit/NUnit XML or plain text logs from multiple runs
- Groups pass and fail results per test across runs
- Assigns a likely cause and a fix direction for each test
- Can update the quarantine section of tests/regression-suite.md
- Can optionally write a flakiness report to production/qa/
Install the test-flakiness skill
Option 1: download the ZIP from this page and unzip it into your user skills folder, ~/.claude/skills/, so it is available in every project. Option 2: unzip it into a single project's .claude/skills/ folder so it applies only to that project.
Option 3: in the Claude apps, open Settings, go to Capabilities, and upload the skill under Skills.
Trigger the skill with natural requests
You can invoke the skill with its command, or describe what you want. The skill accepts an argument: a path to a specific CI log file, the word scan to scan all available CI logs, or registry to review tests already listed as quarantined. With no argument it runs scan if CI logs are accessible, otherwise registry.
- Analyse this run: /test-flakiness [ci-log-path]
- Scan all CI logs for flaky tests: /test-flakiness scan
- Review the tests already quarantined: /test-flakiness registry
/test-flakiness scanPrepare your test history
The skill needs results from multiple runs. The source recommends running the test suite at least three times and saving the output, or saving CI pipeline output to test-results/. With fewer than three runs, every finding is labelled suspected and should not be quarantined.
Where the skill looks: for Godot projects, GdUnit4 XML results under reports/. For Unity, NUnit XML written to test-results/ by default. For Unreal, automation logs in Saved/Logs/, searched for Result={Success} and Result={Fail}.
- Keep runs as separate files so each run can be compared
- Use tests/regression-suite.md with a quarantine section, as the skill references it
Read the results and act on them
Results are sorted by fail rate. Tests failing in more than 25% of runs are high flakiness and should be quarantined immediately. Tests failing in 5 to 25% of runs are moderate and should be investigated and fixed soon. Tests failing in 1 to 5% of runs are low and should be monitored.
For quarantined tests, the skill names the skip mechanism for your engine. For Godot (GDScript) it is a skip parameter on the test function; for Unity it is an NUnit Ignore attribute with a reason. For Godot C# and Unreal the source does not document a skip method, so the skill logs the test and asks you how your CI excludes it.
- High (over 25%): quarantine now, then fix the root cause
- Moderate (5 to 25%): fix the test directly, do not quarantine yet
- Low (1 to 5%): note as suspected and collect more runs
What people use it for
- Finding tests that fail roughly one run in ten during late polish, where the team has started ignoring red builds.
- Diagnosing tests already on a quarantine list to see whether they are timing, order, seed or float problems.
- Spotting a Unity test whose NUnit results alternate between Passed and Failed across a week of CI runs.
- Checking Unreal automation logs in Saved/Logs/ for tests that flip between Success and Fail.
- Producing a dated flakiness report in production/qa/ to share with the team.
Frequently asked questions
How do I install the test-flakiness agent skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder for one project. You can also upload it in the Claude apps under Settings, then Capabilities, then Skills.
What does the test-flakiness skill require?
It needs saved test result files or CI logs (JUnit/NUnit XML or plain text) from multiple runs, limited shell access to ls on recent files and a bundled yaml-helper.sh script, and the Claude Code project layout with tests/regression-suite.md and the referenced .claude/docs files.
Is the test-flakiness skill free?
The source gives the licence as MIT. It does not state pricing, so check the listing on this page for any other terms.
How many test runs do I need before the skill is useful?
The source says at least three runs. With fewer than three, every finding is only suspected, so the skill advises collecting more data before quarantining anything.
Which game engines does it support?
It covers Godot with gdUnit4, Unity and Unreal log formats. Parts of the Godot C# guidance and Unreal skip handling are not documented in the source, so the skill asks you how those tests are skipped.
Does the skill fix flaky tests automatically?
No. It recommends fixes and can edit the quarantine section of the regression suite file, but finding and fixing the root cause is still the developer's job.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if your team has many test runs in CI history, is in a late polish phase, or has been dismissing red builds as probably flaky, and you already keep a quarantine list. Pick something else if you have fewer than three runs, your project is not Godot, Unity or Unreal, or you expect the tool to fix code for you.
Download the skill