Guide · Agent skill

How to use the TestRail Sync agent skill for Playwright

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The TestRail Sync skill teaches an AI agent to link your automated Playwright browser tests with TestRail, the test management tool where teams keep their test cases and track results. This guide explains what the skill does, how to install it, how to configure it, and how to ask for each task in plain language. By the end you will be able to import TestRail cases as Playwright tests, create runs, and push results back, with the safety limits in mind.

TestRail Syncalirezarezvani/claude-skills · MIT Download the skill

Quick start

1Download and unzip the skill. Download the skill ZIP from this page and unzip it into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder.
2Or upload it in the Claude apps. Alternatively, upload the ZIP under Settings → Capabilities → Skills.
3Set the environment variables. Set TESTRAIL_URL, TESTRAIL_USER and TESTRAIL_API_KEY so the agent can reach your TestRail instance.
4Install the TestRail MCP server. Run npm install in integrations/testrail-mcp, then register the pw-testrail server in your user or project MCP config.
5Annotate your Playwright tests. Add a testrail annotation with the case ID, such as C12345, to each test you want linked.
6Run a status check. Ask the agent for the TestRail sync status to see which cases and tests are linked.
7Push results. Run Playwright with the JSON reporter, then ask the agent to push results to a TestRail run and review the summary.

What the TestRail Sync skill makes the agent do

Agent skills are folders of instructions that an AI agent (such as Claude) loads when a request matches them. This skill covers the link between Playwright tests and TestRail in both directions.

The agent can import TestRail test cases and generate matching Playwright test files, create test runs, push pass, fail and skip results back to TestRail, report which cases are linked to test code, and update case steps in TestRail from test code.

Import: fetch cases from a TestRail project and suite, then generate Playwright tests grouped by section
Push results: run Playwright and send each result to the matching TestRail case
Create run: start a new TestRail test run and collect its case IDs from test annotations
Status: compare TestRail cases with local tests and report coverage gaps
Update: write steps and expected results from a test back to its TestRail case

How do you install the TestRail skill?

Download the skill ZIP from this page and unzip it. You can install it in one of three ways.

For all of your projects, unzip it into ~/.claude/skills/. For a single project, unzip it into the .claude/skills/ folder inside that project. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

The skill's source file is SKILL.md. After installing, the skill loads when your request matches its trigger phrases.

~/.claude/skills/

Set up TestRail credentials and the MCP server

The skill needs three environment variables: TESTRAIL_URL (for example, your instance address), TESTRAIL_USER (your email) and TESTRAIL_API_KEY (an API key generated in TestRail). If they are missing, the agent will tell you how to set them and stop.

The skill also depends on TestRail MCP tools. The TestRail MCP server is not registered automatically. Setting the environment variables alone is not enough. Until you enable the pw-testrail server manually, commands fail with 'tool not found'. The plugin's CLAUDE.md file, in its Integrations section, describes the steps: install dependencies in integrations/testrail-mcp, then register the server in your own user or project MCP config.

Requires Node.js and npm to install the TestRail MCP server
Requires a Playwright test project and a TestRail account with API access
Set TESTRAIL_URL, TESTRAIL_USER and TESTRAIL_API_KEY as environment variables

cd integrations/testrail-mcp && npm install

How do you trigger the skill with natural requests?

You do not need to memorise commands. Mention TestRail, test cases, a test run, or syncing, and the agent will pick up the skill. Phrases the skill is written to respond to include 'sync test cases', 'push results to testrail' and 'import from testrail'.

The skill also defines explicit commands. Each one maps to a capability above, and the command is the same as a natural request, so you can use whichever is clearer.

'Import the regression suite from TestRail project 4, suite 12 into Playwright tests'
'Push the latest Playwright results to TestRail run 981'
'Create a TestRail run called Sprint 42 Regression in project 4'
'Show me which TestRail cases have no Playwright test yet'
'Update TestRail case C12345 from its Playwright test'

/pw:testrail import --project <id> --suite <id>
/pw:testrail push --run <id>
/pw:testrail run --project <id> --name "Sprint 42 Regression"
/pw:testrail status --project <id>
/pw:testrail update --case <id>

How do tests link to TestRail cases?

The link is a test annotation. Each Playwright test linked to TestRail carries its case ID, for example C12345. When results are pushed, the agent reads these annotations to match each test to its case. When a run is created, it collects the IDs the same way.

When the agent pushes results, it maps Playwright outcomes to TestRail status IDs: pass to 1, skip to 2 and fail to 5. For failures, it includes the error message. To produce the results, it runs Playwright with the JSON reporter and parses the output.

Pass uses status_id 1
Skip uses status_id 2
Fail uses status_id 5 and includes the error message

test('should login successfully', async ({ page }) => {
  test.info().annotations.push({
    type: 'testrail',
    description: 'C12345',
  });
  // ... test code
});

What should you check before you let the agent write?

The skill runs scripts and uses the network. It talks to your TestRail instance through the MCP tools, and it writes to a shared test management system. An agent error could overwrite cases or results that your team relies on.

Keep your API key scoped to what the agent needs, and review the operation summary and any unmatched cases before you rely on the output. Only use the skill with a TestRail account you trust the agent to change.

The skill can create cases and runs, add results and update case steps
Do not use it if you cannot safely hand a TestRail API key to an agent
Review the summary, errors and unmatched cases after each operation

Tips for getting reliable results

Start with read-only actions such as status and import before you push results or update cases. The status report shows how many TestRail cases are unlinked and how many Playwright tests lack a TestRail ID, which helps you spot gaps before writing anything.

Make sure every linked test has its annotation, because the agent matches results only through these IDs.

Run status first to see coverage gaps
Check that each test carries a testrail annotation with its case ID
Expect the output to include counts, errors, unmatched cases and a link to the run or results

What people use it for

QA engineers generating automated Playwright coverage from manual regression cases stored in TestRail
Development teams reporting Playwright pass and fail results to TestRail after each CI or local run
Teams auditing which TestRail cases have no automated test yet
Creating a named regression run for a sprint and collecting its results

Frequently asked questions

How do I install the TestRail skill?

Download the skill ZIP from this page and unzip it into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

Why do my testrail commands fail with tool not found?

The TestRail MCP server is not registered automatically. You need to run npm install in integrations/testrail-mcp and register the pw-testrail server in your own user or project MCP config. Setting the environment variables alone does not fix it.

What environment variables does the TestRail skill need?

It needs TESTRAIL_URL, your TestRail instance address, TESTRAIL_USER, your email, and TESTRAIL_API_KEY, an API key from TestRail. If any are missing, the agent will tell you how to set them and stop.

Is the TestRail skill free to use?

The skill is published under the MIT licence. You still need your own TestRail account with API access, and Node.js and npm for the MCP server.

How does the skill match test results to TestRail cases?

Each linked Playwright test has a testrail annotation containing its case ID, such as C12345. The agent reads these annotations when pushing results or creating runs, so tests without an annotation are not matched.

Does the TestRail skill work with tools other than TestRail?

No. The skill is specific to TestRail's API and its MCP tools, so it is not useful for teams that do not use TestRail. Teams using other test management tools would need a different integration.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Choose this skill if you already keep test cases in TestRail, use Playwright, and want the agent to import cases, push results or report coverage gaps. Skip it if you do not use TestRail, or if you are not comfortable giving an agent an API key that can change shared cases and results.

Download the skill