Skip to content
Get packs

Trace To Training Data

Turns graded agent traces and logs into SFT and preference-pair training data, with hygiene checks.

Source Download skill
Repository
wshobson/agents
Stars
42,095
Licence
MIT
Files
2 files

What it does

This skill teaches an AI agent how to convert already-graded evaluation runs and production logs into training datasets for fine-tuning a model. It covers which successful examples to keep, how to build chosen-versus-rejected pairs for preference training, and how to strip secrets and PII, keep evaluation test items out of training data, deduplicate, and record where each row came from. It expects grading to have been done earlier by a separate evaluation skill and points to other skills for the exact output formats and pair-selection formula.

Great for

  • ML engineers preparing SFT or DPO datasets from evaluation runs
  • Teams running an eval-harness workflow who want labeled traces turned into training rows
  • People who need a checklist for keeping eval goldens out of training data

Skip it if

  • Anyone without already-graded traces or a grader; the skill refuses to hand-label and will route back upstream, so it does nothing useful on raw logs
  • Production data handling without legal or privacy review; the scan is described but its reliability is not verified, and raw customer data could still leak into training sets

Try asking

  • “Convert the passing traces in my eval run results.json into SFT messages rows, keeping only the top-reward fraction”
  • “Build DPO pairs from passing and failing runs on the same task, using the mean-minus-two-sigma rule for the rejected member”
  • “Check my converted dataset against eval/goldens.jsonl so no golden IDs leak into training”
  • “Add provenance run_id and trace_id fields to each converted row for the dataset card”
Step-by-step guide: how to use Trace To Training DataInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the MIT licence and an attribution note.

    trace-to-training-data.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip trace-to-training-data.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip trace-to-training-data.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Run the eval-harness-first skill before this one so every trace already has a verdict and reward
  • Review the secret and PII redaction output yourself before any dataset is shared, since the skill describes the scan but does not ship a scanner
  • Read references/conversion-recipes.md for worked JSONL examples before converting a large batch

What's inside

  • references/conversion-recipes.md
  • SKILL.md

Redistributed unmodified from wshobson/agents under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.