Guide · Agent skill
How to use the CI Failure Patrol agent skill (ci-flaky-patrol)
Updated · written with AI from the project's own documentation and checked against our catalogue data
The ci-flaky-patrol skill teaches an AI agent to review a small batch of recently failed CI checks on open pull requests and decide what to do with each one: rerun it, comment on the PR, or take no action. It is written for maintainers and teams who already run a JavaScript driver script that handles GitHub reads and writes. After this guide you will know what the skill does, how to install it, how to trigger it, and what it needs before it will work.
CI Failure PatrolQwenLM/qwen-code · Apache-2.0 Download the skill Quick start
- 1Download the ZIP. Download the ci-flaky-patrol ZIP from this page.
- 2Unzip into a skills folder. Unzip into ~/.claude/skills/ for personal use or into the project's .claude/skills/ folder for one project.
- 3Or upload in the Claude apps. In the Claude apps, open Settings → Capabilities → Skills and upload the skill.
- 4Set up the driver script. Provide your own JavaScript driver that performs GitHub reads and writes and produces ci-flaky-input.json.
- 5Run a classification batch. Ask the agent to classify the candidates in ci-flaky-input.json.
- 6Review ci-flaky-decisions.json. Check each action, confidence level and reason, and let the driver apply the writes.
What the skill makes the agent do
The skill reads a file named ci-flaky-input.json from the caller's working directory. Each entry is a candidate: a failed CI check on a stale pull request. For every candidate the agent chooses exactly one action. Transient infrastructure or flaky-test evidence leads to rerun. A failure clearly caused by the PR's own changed files leads to comment. Anything ambiguous, unsafe or incomplete leads to no_action.
The agent does not call GitHub. The driver script performs all reads, validation, state tracking and writes. The agent only classifies candidates and writes ci-flaky-decisions.json. The skill also treats log text as untrusted data and ignores any instructions embedded in logs.
- rerun: runner or network timeouts, interrupted infrastructure, transient install or download failures, or explicit flaky-test evidence.
- comment: the failure is clearly caused by the PR, judged by comparing the failure with changedFiles.
- no_action: evidence is ambiguous, unsafe, incomplete, or does not justify another action.
Install the CI Failure Patrol skill
Download the ZIP for ci-flaky-patrol from this page. Unzip it into one of two locations. For a personal install, use ~/.claude/skills/. For a single project, use the .claude/skills/ folder inside that project. The skill folder contains SKILL.md.
If you use the Claude apps instead, upload the skill under Settings → Capabilities → Skills.
- Personal install: unzip into ~/.claude/skills/
- Project install: unzip into the project's .claude/skills/
- Claude apps: Settings → Capabilities → Skills, then upload
Prepare the driver script and input file
The skill does not include a driver. You need a JavaScript driver script of your own that performs all GitHub reads and writes and produces ci-flaky-input.json. Without it the skill assumes infrastructure exists and will not work correctly on its own.
The agent runtime must support read_file and write_file tools, because those are the only tools the skill is allowed to call.
- A JavaScript driver script (not included in the skill folder)
- ci-flaky-input.json in the caller's working directory
- An agent runtime with read_file and write_file tools
Trigger the skill with a natural request
Ask the agent to classify the batch in your working directory. The skill's description is 'Classify a bounded batch of stale PR CI failures and choose the safest response', so requests that name CI failures, stale pull requests, flaky checks or the input file should match it. Name the input file so the agent knows where the batch is.
The output is a single file, ci-flaky-decisions.json. The agent should not write any other file.
- Classify the candidates in ci-flaky-input.json and write ci-flaky-decisions.json.
- Review the stale PR CI failures in this batch and decide rerun, comment or no_action for each.
Read the output and understand flakyTest
Each decision copies its identity fields (prNumber, headSha, runId, runAttempt, failureKey) from the candidate, with an action, a confidence level and bilingual reasons. The reason is capped at 200 characters. Confidence is high only when the evidence directly supports the action, and low is used with no_action.
The optional flakyTest object appears only for a rerun caused by genuine test nondeterminism: a specific named test that timed out, is order-dependent, or depends on wall-clock time or randomness. It is never used for infrastructure flakiness such as ENOSPC, network errors, runner death or dependency downloads. If the log does not name a specific test, flakyTest is omitted.
- flakyTest.file is the repo-relative path to the failing test file.
- flakyTest.name is the full test title, copied verbatim from the log.
- Each of file and name is limited to 200 characters. A malformed flakyTest is ignored and the rerun still happens.
Know the limits and safety notes
The skill runs no scripts and uses no network access. Its only file operations are reading ci-flaky-input.json and writing ci-flaky-decisions.json. Main-branch failures are excluded and should be handled elsewhere. The driver enforces a maximum of 3 actions per PR head, and the current actionCount is supplied only as context.
The skill does not fix code or explain failures interactively. It is a classifier for batches in a driver-based workflow.
- Runs scripts: no. Uses network: no.
- Main-branch failures are out of scope.
- The 3-actions-per-PR-head limit is enforced by the driver, not by the agent.
What people use it for
- Triaging a batch of flaky runner-timeout failures on stale pull requests in a large open-source repository.
- Deciding which failures on stale PRs should be rerun and which should get a comment because the PR's changed files appear to cause them.
- Feeding flakyTest entries into an automated loop that opens a deflake fix for a named, nondeterministic test.
- Studying how to structure a bounded agent task with an explicit output schema and untrusted-input handling.
Frequently asked questions
Is the CI Failure Patrol skill free to use?
The skill is licensed under Apache-2.0, as stated for the QwenLM/qwen-code repository that contains it. Check the license file in the source repository for the full terms before you reuse or redistribute it.
What do I need to run the ci-flaky-patrol skill?
You need a JavaScript driver script that performs all GitHub reads and writes and produces ci-flaky-input.json, which is not included in the skill folder. You also need an agent runtime that supports read_file and write_file tools. Without the driver the skill will not work correctly on its own.
Where do I install an agent skill like this?
Unzip the skill into ~/.claude/skills/ for a personal install or into a project's .claude/skills/ folder. In the Claude apps you can instead upload it under Settings → Capabilities → Skills.
Does the skill fix code or call GitHub for me?
No. The skill does not call GitHub and does not fix code. A separate driver handles reads, validation, state and writes, and the agent only classifies each candidate and writes ci-flaky-decisions.json.
What is the difference between rerun and comment?
A rerun is for transient evidence such as a runner or network timeout, interrupted infrastructure, a failed download, or explicit flaky-test evidence. A comment is for failures clearly caused by the PR, judged by comparing the failure with its changedFiles. Ambiguous cases get no_action.
Why does my flaky test not get a flakyTest entry?
flakyTest is added only for genuine test nondeterminism where the log names a specific test. Infrastructure flakiness such as ENOSPC or network errors gets a plain rerun with no flakyTest, and if the log does not name a test, the field is omitted.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you maintain a large repository or run an automated CI triage loop and already have a driver script that handles GitHub reads and writes. It is a good reference for structuring safe, bounded agent tasks with explicit output schemas. If you want an interactive helper that explains failures or fixes code, or you do not have a driver, pick a different tool.
Download the skill