Guide · Agent skill
How to use the Trace To Training Data agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The Trace To Training Data skill is an agent skill for Claude that converts evaluation traces and production logs into supervised fine-tuning (SFT) examples and preference pairs for DPO. It is written for ML engineers who already have graded runs from an eval harness. After this guide you will know how to install the skill, how to ask for conversions in plain language, and which hygiene checks it expects before any row is merged into a training set.
Trace To Training Datawshobson/agents · MIT Download the skill Quick start
- 1Download the ZIP. Download the skill ZIP from this page and unzip it on your machine.
- 2Place the skill folder. Copy the trace-to-training-data folder into ~/.claude/skills/ for personal use or into .claude/skills/ in your project.
- 3Or upload in the Claude apps. In the Claude apps, open Settings, go to Capabilities, and upload the skill under Skills.
- 4Prepare graded traces. Make sure eval/goldens.jsonl and runs/<run-id>/results.json exist with verdicts and rewards from eval-harness-first.
- 5Ask for a conversion. Describe the run and the target format, for example SFT messages rows or DPO pairs.
- 6Check holdout and hygiene. Confirm golden IDs are excluded, secrets and PII are redacted, and each row keeps run_id and trace_id.
What is an agent skill?
An agent skill is a folder of instructions, reference files and optional scripts that an AI agent loads when a task matches the skill's description. This skill's SKILL.md tells the agent how to convert graded traces into training data and points to a references file with worked JSONL-to-JSONL examples.
What does the skill make the agent do?
The skill covers three jobs. For SFT, it keeps the top-reward fraction of passing trajectories rather than every pass, routes expert corrections of failing traces straight into the SFT set, and masks loss on bad steps in multi-step traces instead of discarding the whole trajectory. For preference pairs, it builds chosen-versus-rejected pairs from passing and failing runs on the same task, and filters candidate pairs by chosen-minus-rejected judge delta. For hygiene, it scans for secrets and PII, keeps eval goldens out of training data, deduplicates against existing training data, and records provenance back to each run_id and trace_id.
The skill does not grade traces. It expects grading to have been done upstream by the eval-harness-first skill, and it routes any trace without a verdict or reward back there instead of hand-labeling it.
Install the skill from the ZIP
Download the ZIP for this skill from this page and unzip it. For a personal install, place the skill folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder. Alternatively, in the Claude apps, open Settings, go to Capabilities, and upload the skill under Skills.
The files in the package are SKILL.md and references/conversion-recipes.md. The skill's frontmatter name is trace-to-training-data.
- Personal: ~/.claude/skills/
- Project: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload
Prepare your inputs
The skill expects graded traces: eval/goldens.jsonl and runs/<run-id>/results.json, produced by the eval-harness-first skill. Each row needs a task_id, a verdict from the grader, and a reward when the task supports a scalar score. The skill also refers to dataset-curation and preference-optimization for output formats and the pair-selection formula. Those two skills are not included in this package, so you need them separately if you want the exact formats and selection formula.
- eval/goldens.jsonl
- runs/<run-id>/results.json with task_id, verdict and reward
- dataset-curation and preference-optimization skills (not included)
{"task_id": "t-042", "trace_id": "t-042-a3", "messages": [{"role": "user", "content": "..."}], "verdict": "pass", "reward": 0.91, "grader": "exact_match"}Trigger the skill with natural requests
You do not need a special command. Describe what you have and what you want. The skill's description matches requests about converting graded traces or failure examples into training data, applying rejection sampling to model outputs, or building DPO pairs from passing and failing runs.
Requests that should match include converting the graded traces in runs/<run-id>/results.json into SFT rows, building DPO pairs from passing and failing runs for one task, or checking that no golden IDs appear in a training set.
- "Convert the graded traces in runs/<run-id>/results.json into SFT messages rows."
- "Build DPO prompt/chosen/rejected pairs from passing and failing runs on the same task."
- "Check that no IDs from eval/goldens.jsonl appear in the training set before merging."
Safety notes: scripts and network
The skill does not run scripts and does not use the network, according to its listing. Its secret and PII scan is described in the instructions, but the source does not verify how reliable that scan is. Treat it as a step you review, not a guarantee. If your traces come from production logs, have legal or privacy review the data before it is converted.
The skill also states that conversion fails closed: if sensitive fields remain after redaction, the row is dropped rather than shipped.
- No scripts and no network access listed for this skill.
- The PII scan is not verified for reliability; check outputs yourself.
- Production data needs legal or privacy review before use.
Tips for better results
Give the agent the exact file paths and run ID so it reads the right results.json. Ask for the dedup check against existing training data, not only within the new batch. Keep the goldens holdout check as a separate step you can inspect.
- Name the run ID and file paths explicitly.
- Ask for provenance fields (run_id and trace_id) on every row.
- Review the dropped-row count after redaction.
What people use it for
- Turning a graded eval run into SFT messages rows, keeping only the top-reward fraction of passing traces.
- Building DPO pairs from passing and failing attempts on the same task.
- Converting an expert-corrected failing trace directly into an SFT example.
- Checking that no golden IDs from eval/goldens.jsonl leak into a training set before a merge.
Frequently asked questions
Is the trace-to-training-data skill free?
The skill is published under the MIT licence, as listed on this page. The source does not mention pricing for the Claude apps or any other service, so check those separately.
What do I need before using this skill?
You need graded traces: eval/goldens.jsonl and runs/<run-id>/results.json produced by eval-harness-first, with a verdict and, where relevant, a reward on each row. Without a grader, the skill will not hand-label traces and will route you back upstream.
Where do I install the trace-to-training-data skill?
Unzip the downloaded ZIP into ~/.claude/skills/ for personal use or into .claude/skills/ in a project. You can also upload it in the Claude apps under Settings, then Capabilities, then Skills.
Does this skill run scripts or use the internet?
The listing says it runs no scripts and uses no network. Its PII scan is described but not verified for reliability, so review outputs before merging.
Can the skill grade my traces for me?
No. It assumes grading was done earlier by eval-harness-first. If a trace has no verdict or reward, the skill sends it back upstream rather than labeling it by hand.
What are the alternatives if this skill is not right for me?
If you do not have graded traces, start with an evaluation workflow that produces verdicts first. The source does not name specific alternative tools, so compare options on your own criteria.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits ML engineers who already run an eval harness and want graded traces turned into SFT or DPO rows with holdout and dedup checks. It is not useful for raw logs without grading, and production data still needs privacy review because the PII scan is not verified.
Download the skill