Guide · Agent skill
How to use the Backtest Diagnosis agent skill (backtest-diagnose)
Updated · written with AI from the project's own documentation and checked against our catalogue data
Backtest Diagnosis is an agent skill, a packaged set of instructions that an AI agent loads when a task matches it. This one is for the case where a trading strategy backtest crashes, produces no trades, or returns odd numbers. After reading this guide you will know what the skill makes the agent check, how to install it, how to ask for it in plain language, and what it needs to run.
Backtest DiagnosisHKUDS/Vibe-Trading · MIT Download the skill Quick start
- 1Download the ZIP. Download the backtest-diagnose ZIP from the skill page.
- 2Unzip into the skills folder. Unzip it into ~/.claude/skills/ for personal use, or into your project's .claude/skills/ folder.
- 3Or upload in the Claude apps. In the Claude apps, go to Settings, then Capabilities, then Skills, and upload the skill.
- 4Check the workspace files. Make sure your backtest workspace has artifacts/metrics.csv, equity.csv, trades.csv, code/signal_engine.py and config.json.
- 5Describe the failure. Ask the agent in plain language what went wrong, including any error text or symptom.
- 6Review the edit and rerun. Check the edit_file change the agent makes, then confirm the rerun and the new metrics.csv.
What the backtest-diagnose skill makes the agent do
The skill gives the agent a fixed workflow for investigating a failed or underperforming backtest. The agent reads the result files (artifacts/metrics.csv, equity.csv and trades.csv) and the strategy code (code/signal_engine.py and config.json). It then sorts the problem into a known category, makes a targeted edit with edit_file, and reruns the backtest.
The skill separates runtime errors (exit_code not 0) such as ImportError, KeyError, IndexError and TypeError, from logic bugs where the backtest finishes but results look wrong. Logic bugs include zero trades, a first trade more than 2 years after the start, capital mostly in cash below 50% utilization, and an open position at the end of the run.
Install the backtest-diagnose skill
Download the ZIP from the skill page and unzip it into one of two places. For a personal setup, use ~/.claude/skills/. For a single project, use the .claude/skills/ folder inside that project. The skill's main file is SKILL.md.
You can also upload the skill in the Claude apps. Go to Settings, then Capabilities, then Skills, and upload it there.
- Personal install: unzip into ~/.claude/skills/
- Project install: unzip into your project's .claude/skills/ folder
- Claude apps: Settings → Capabilities → Skills
Trigger the skill with natural requests
You do not need to name the skill. The skill applies when you tell the agent that a backtest failed, raised an error, or gave poor results. Describe the symptom and point the agent at your backtest workspace.
Useful details to include are the symptom (error text, zero trades, strange equity curve), the workspace folder, and whether the strategy downloads market data.
- "My backtest crashed with a KeyError, please diagnose and fix it."
- "The backtest ran but produced zero trades. Find the root cause in signal_engine.py."
- "The first trade happens three years after the start date. Why?"
- "Diagnose why this run ends with an open position and fix the exit logic."
What the skill checks before it counts a run
The skill applies a hard-gate checklist. A run only counts as valid evidence if metrics.csv and equity.csv exist and are non-empty, trade_count is greater than 0, the equity series has no NaN values, and exit_code is 0.
A run that fails a gate is skipped rather than partly recorded. The source lists the stable tokens hard-gate:exit-nonzero, hard-gate:metrics-missing, hard-gate:zero-trades, hard-gate:equity-empty and hard-gate:equity-nan. After a fix, the source says to repopulate the evidence cache with refresh_strategy_evidence, and to see the strategy-discovery skill for the manifest format.
Safety notes: scripts and network access
This skill runs scripts and uses the network. The agent runs bash commands such as pip install for missing dependencies, and a Python AST syntax check on signal_engine.py. If the strategy fetches market data, the backtest uses the data-provider API token stored in config.json.
The skill tells the agent not to change code when errors come from the data provider, such as rate limits, API limits, daily limits, a no-data response, or Information messages from Tushare. Those need the account or data source fixed by you.
- Review any pip install the agent proposes before approving it.
- Keep API tokens in config.json on your own machine and do not share the workspace publicly.
- The skill limits the agent to at most 3 repair iterations.
Tips for getting good results
The skill asks for one fix at a time, with a rerun after each. Prefer edit_file over full rewrites, and ask for the bug to be fixed without changing strategy logic. After a fix, the agent checks that the file parses, contains class SignalEngine and def generate, and then reruns the backtest.
The skill also asks for action_items, which are specific suggestions such as changing a threshold in a named file. Ask for at least two if you want options.
- Give the agent the exact error text from the run.
- Ask it to fix only the bug if you do not want the strategy changed.
- Check the new metrics.csv yourself before trusting the result.
python -c "import ast; ast.parse(open('code/signal_engine.py').read()); print('OK')"What people use it for
- Fixing a KeyError caused by a column name mismatch in data_map inside signal_engine.py.
- Finding why a mean-reversion or momentum signal never fires and produces zero trades.
- Spotting a lookback window or dropna step that delays the first trade by years.
- Catching a backtest that ends with an open position because forced liquidation is missing.
- Preparing a run so it passes the hard gates before it is added as strategy evidence.
Frequently asked questions
How do I install the backtest-diagnose skill?
Download the ZIP from the skill page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. You can also upload it in the Claude apps under Settings, then Capabilities, then Skills.
What do I need to run backtest-diagnose?
You need a Python environment, a Vibe-Trading backtest workspace containing artifacts/metrics.csv, equity.csv, trades.csv, code/signal_engine.py and config.json, and agent tools read_file, edit_file and bash. A data-provider API token in config.json is needed if the strategy fetches market data, such as Tushare.
Is backtest-diagnose free to use?
The source lists the skill under the MIT licence. Check the repository's licence file for the terms that apply to your use.
Does this skill tell me if a strategy will make money?
No. The skill fixes code and checks structure against the gates. It does not validate that a strategy will be profitable, so it should not be the basis for real-money trading decisions.
Why does the agent refuse to change my code when the backtest fails?
If the error comes from the data provider, such as a rate limit, API limit, daily limit, no-data response, or a Tushare Information message, the skill tells the agent not to modify code. You need to check your API token, switch data sources, or wait for the quota to reset.
How is this different from just asking a chatbot to fix my code?
The skill gives the agent a fixed diagnostic workflow, a set of error categories, a hard-gate checklist for valid runs, and post-fix checks such as the AST syntax test. It also limits repairs to at most 3 iterations and asks for one fix at a time.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use backtest-diagnose if you run Vibe-Trading backtests or edit signal_engine.py files and want an agent to work through errors and suspicious results in a disciplined, checkable way. Pick something else if you need a tool to judge whether a strategy is profitable or safe to trade, since this skill only fixes code and checks structure.
Download the skill