Guide · Agent skill

How to use the Writing Evals for Axiom agent skill

Updated 2026-11-02 · written with AI from the project's own documentation and checked against our catalogue data

Writing Evals is an agent skill for the Axiom AI SDK. Once it is installed, your coding agent can generate automated tests (called evals) for AI features in your codebase, including the eval files, scorers, flag schemas and an axiom.config.ts file. This guide explains what the skill does, how to install it, how to ask for help with natural requests, and what you need before it will work.

Writing Evals for Axiomopenclaw/clawhub · MIT Download the skill

Quick start

1Download and unzip the skill. Download the writing-evals ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder. In the Claude apps, you can upload the ZIP under Settings → Capabilities → Skills instead.
2Complete the Axiom AI SDK Quickstart. Set up instrumentation and authentication following the Axiom AI SDK Quickstart so the SDK is working in your project.
3Confirm the SDK is installed. Run ls node_modules/axiom/dist/. If nothing is there, install the axiom package with your package manager, such as pnpm add axiom.
4Ask for evals in plain language. Name the AI function you want tested, for example its file path, and describe what correct output looks like.
5Review the generated files. Check the .eval.ts files, scorers, flag schema and axiom.config.ts. Make sure the imports point at your real code.
6Set credentials and run. Put AXIOM_URL, AXIOM_TOKEN and AXIOM_DATASET in environment variables, then run the evals in your project.

What is an agent skill and what does this one do?

An agent skill is a folder of instructions and reference files that an AI coding agent loads when a task matches the skill's description. This skill is named writing-evals. Its description says it scaffolds evaluation suites for the Axiom AI SDK and generates eval files, scorers, flag schemas and config from natural-language descriptions.

Evals are tests for non-deterministic AI systems. Each eval checks whether a capability still behaves correctly after changes, so you can see if a model swap or prompt edit broke something.

Reads the AI function you point to and traces its inputs, outputs and model call
Picks an eval type (classification, text quality, retrieval, structured output, tool use or streaming) based on the output
Writes .eval.ts files with at least two scorers each
Creates test data covering happy path, adversarial, boundary and negative cases
Sets up flag schemas so you can compare models or temperatures without code changes

Install the writing-evals skill

You can install the skill by downloading the ZIP from this page and unzipping it into one of two locations. Use ~/.claude/skills/ for a personal install available across projects, or a project's .claude/skills/ folder to share it with one codebase.

In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

Personal: unzip into ~/.claude/skills/
Single project: unzip into your project's .claude/skills/ folder
Claude apps: Settings → Capabilities → Skills, then upload the ZIP

Prepare your project before you ask for evals

The skill assumes the Axiom AI SDK is already set up. Complete the Axiom AI SDK Quickstart (instrumentation and authentication) first. The skill's SKILL.md says to verify the SDK is installed by checking that node_modules/axiom/dist/ exists. If it is missing, the skill says to install it with your project's package manager, for example pnpm add axiom.

The skill also tells the agent to read the bundled docs in node_modules/axiom/dist/docs/ before writing code, because those docs are treated as the source of truth for the installed version.

Node.js project with a package manager such as pnpm or npm
Axiom AI SDK (axiom package) installed, with the Quickstart completed
Axiom credentials for sending eval results: AXIOM_URL, AXIOM_TOKEN and AXIOM_DATASET

ls node_modules/axiom/dist/

Trigger the skill with natural requests

You do not need special syntax. The skill is meant to be triggered by describing what you want. Its description lists these cases: creating evals, writing scorers, setting up flag schemas, or configuring axiom.config.ts.

The skill tells the agent to read the code first and to avoid asking many questions. It only asks when it cannot determine what correct means for an ambiguous output, whether you want pass/fail or partial credit, or which parameters should be tunable.

"Write evals for the categorizeMessages function in src/lib/capabilities/support-agent."
"Add a flag schema so I can compare two models and a few temperatures for the ticket extractor."
"Set up axiom.config.ts for this project with the instrumentation hook."
"Write a scorer that checks the tool calls my support agent makes."

Review what the skill generates

A generated eval follows a standard shape: it imports your real function, defines scorers, lists data records of input, expected and metadata, and passes a task function that calls your code. The skill places .eval.ts files next to their source files and uses the default glob **/*.eval.{ts,js} to find them. axiom.config.ts stays at the project root.

Review the output before you run it. Confirm the imports point at your real function, that the scorers match your output type, and that the test data reflects real cases.

Eval file colocated with its implementation file
Scorers chosen by output type, with a correctness scorer plus a quality or reference-free scorer
Flag schema built with createAppScope and typed Zod schemas

Eval('my-eval-name', {
  capability: 'my-capability',
  data: [
    { input: '...', expected: '...', metadata: { purpose: '...' } },
  ],
  task: async ({ input }) => {
    return await myFunction(input);
  },
  scorers: [MyScorer],
});

Safety: scripts, network and credentials

This skill runs scripts and uses the network. Eval runs send traces and data to an Axiom account through AXIOM_URL, AXIOM_TOKEN and AXIOM_DATASET. Keep tokens in environment variables rather than in source files, and review what gets exported before you run evals against real data or production credentials.

Also check that the skill is pointed at the correct installed SDK version. The skill's own guidance is that its examples should give way to the bundled docs when they conflict.

Store AXIOM_TOKEN in environment variables, not in committed code
Review exported data before running evals on sensitive or production records
Confirm the SDK version and docs in node_modules match what the generated code uses

Tips for better eval output

Give the agent a precise function name or file path. The skill works from code, so the clearer the target, the better the result. Ask for adversarial and edge cases explicitly if the default data feels thin. Use flags to sweep models and temperatures instead of editing code for each variant. Keep the scorers focused: the skill's guidance is that each scorer checks one property of the output.

Name the exact function or file to test
Ask for adversarial, boundary and negative cases in the test data
Use flag schemas to compare models or temperatures without code changes
Run the eval, read the scores, then refine scorers rather than the data first

What people use it for

Create a classification eval with exact-match and confidence-threshold scorers for a support message categorizer
Build a tool-use eval that checks whether an agent calls the expected tools with valid arguments
Set up a flag schema to compare two model versions and several temperatures without editing code
Generate structured-output evals that check each field of an extracted object and its completeness
Write retrieval evals that check result sets with set-match scoring

Frequently asked questions

How do I install the writing-evals skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for a personal install, or into a project's .claude/skills/ folder. In the Claude apps, you can upload the ZIP under Settings → Capabilities → Skills.

What do I need before using this skill?

You need a Node.js project, the Axiom AI SDK (axiom package) installed, the Axiom AI SDK Quickstart completed, and Axiom credentials (AXIOM_URL, AXIOM_TOKEN and AXIOM_DATASET) for sending eval results. Optional OpenTelemetry packages are used for token usage instrumentation.

Is the writing-evals skill free?

The skill is listed under the MIT licence. The source does not state pricing for the Axiom platform, so check Axiom's own terms for any account costs.

Can I use this skill without the Axiom AI SDK?

No. The templates and API assumptions target the Axiom AI SDK, so the skill is not a fit for projects that do not use it. The skill tells the agent to check the installed docs first.

Why does the skill need network access and scripts?

The skill runs scripts and uses the network because eval runs send traces and data to an Axiom account. Keep your tokens in environment variables and review what gets exported.

What should I do if the generated eval imports fail?

Check that the axiom package is in node_modules/axiom/dist/ and read the docs in node_modules/axiom/dist/docs/. The skill's guidance is to follow those bundled docs over its own examples when they conflict.

Does this skill compare to writing evals by hand?

The source does not include a benchmark or comparison with other tools. It is useful when you want consistent eval structure, scorers and test data generated from your code, and you can review and edit the output as you would any hand-written file.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you build AI features with the Axiom AI SDK and want consistent, repeatable eval files, scorers, flag schemas and test data generated from your code. Skip it if your project does not use the Axiom AI SDK or if you need a general AI writing or testing tool that does not require a codebase.

Download the skill