Guide · Agent skill
How to use the paper-claim-audit agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
Paper Claim Audit is an agent skill, a folder with a SKILL.md file that tells an AI agent how to perform one specific task. This one checks a research paper draft against the raw experiment output files, flagging numbers, comparisons and scope claims that the data does not support. It is aimed at researchers preparing a submission, teams running automated research pipelines, and students checking their reported results against their logs. After this guide you will know how to install the skill, trigger it with a plain request, and read the audit report it produces.
Paper Claim Auditwanshuiyin/Auto-claude-code-research-in-sleep · MIT Download the skill Quick start
- 1Download the ZIP. Download the paper-claim-audit ZIP from the page you are reading.
- 2Unzip into a skills folder. Unzip it into ~/.claude/skills/ for all projects, or into the project's .claude/skills/ folder for one project.
- 3Or upload it in the Claude apps. In the Claude apps, open Settings → Capabilities → Skills and upload the skill there instead.
- 4Confirm your reviewer. Make sure the mcp__codex__codex reviewer tool or your configured alternative backend is available.
- 5Prepare the project. Check that the paper .tex files and raw result files in json, csv, tsv, jsonl or yaml form are in the project.
- 6Ask for the audit. Say something like 'verify numbers' or 'check paper claims', or run /paper-claim-audit with the paper directory.
- 7Review PAPER_CLAIM_AUDIT.md. Read the overall verdict and check each non-exact status against the cited evidence file before you revise the paper.
What the skill makes the agent do
The skill checks paper-to-evidence fidelity: whether every number, comparison and scope claim in the paper matches the raw result files. It pulls each claim out of the .tex files and traces it to the evidence, looking for failure modes such as rounding beyond standard display precision, reporting the best seed instead of the average, deltas that are arithmetically wrong, and wording like 'consistently outperforms' when only two datasets were tested.
To avoid confirmation bias, the checking is done by a separate reviewer that starts with zero prior context. The reviewer receives only the paper .tex files and the raw results, not the executor's experiment logs, trackers, review notes or summaries. The skill does not judge whether the experiment design or the scientific conclusion is sound; for that, the source points to the separate experiment-audit and result-to-claim audit skills.
- Checks numbers, percentages and relative improvements against raw result files
- Flags best-seed cherry-picking, aggregation mismatches and config mismatches between compared methods
- Checks caption-to-table and caption-to-figure consistency
- Writes a PAPER_CLAIM_AUDIT.md report with an overall PASS, WARN or FAIL verdict
Install the paper-claim-audit skill
Download the ZIP for this skill from the page you are reading. The skill is distributed as a folder containing SKILL.md, so unzip it into one of the two locations below. Use the user-level folder to make it available in every project, or a project folder to limit it to one repository.
Alternatively, if you use the Claude apps, upload the skill under Settings → Capabilities → Skills instead of copying files manually.
- User-wide: unzip into ~/.claude/skills/ so the skill folder sits at ~/.claude/skills/paper-claim-audit/
- Single project: unzip into the project's .claude/skills/ folder
- Claude apps: Settings → Capabilities → Skills, then upload the skill
Check the requirements before you run it
The skill needs an agent runtime that supports SKILL.md skills. It also needs access to a reviewer. The source names the mcp__codex__codex reviewer tool and describes a reviewer model called GPT-6-Astra. The skill's shared reviewer reference also allows an alternative configured reviewer backend, selected through a .aris/reviewer.json file, with Claude Code's default Codex MCP route as the fallback.
Your project must contain the paper .tex files and the raw result files. Without raw result files there is nothing to trace claims to, so the audit cannot produce useful results.
- An agent runtime that supports SKILL.md skills
- A reviewer tool: mcp__codex__codex, or another reviewer backend set up through .aris/reviewer.json
- Paper files: main.tex, sections/*.tex and tables/*.tex if they are separate
- Raw result files in json, csv, tsv, jsonl or yaml form, such as results/*.json, outputs/*.csv, wandb-summary.json, metrics.json, eval_results.json, config.yaml or args.json
Trigger the skill with a natural request
You do not need to remember a command. The skill's description lists the phrases that should start it, in English and Chinese. You can also run it as /paper-claim-audit and pass a paper directory as the argument, since the skill accepts [paper-directory] as its argument hint.
Run it once the draft is in a state you want to check, and again only when the paper or results change. Before submission is the typical moment.
- 'check paper claims'
- 'verify numbers'
- '审查论文数据'
- '论文数字核对'
- 'Check the numbers in my paper against the result files before I submit.'
/paper-claim-audit [paper-directory]Read the audit report
The executor side collects the paper and result files without interpreting them, then the fresh reviewer returns a claim-by-claim list. The skill writes these results into PAPER_CLAIM_AUDIT.md, which starts with an overall verdict and counts of each status. Each claim is listed with its location, quoted text, the claimed value, the evidence file, the value found there, and a status.
Statuses range from exact_match and rounding_ok through to number_mismatch, scope_overclaim and unsupported_claim. Treat any non-exact status as something to check by hand against the file the reviewer cited, because the reviewer's verdict is an input to your judgement, not a final ruling.
- exact_match and rounding_ok: the claim agrees with the evidence
- ambiguous_mapping and missing_evidence: the reviewer could not link the claim to a clear source
- config_mismatch and aggregation_mismatch: settings differ or the run count does not match the claim
- number_mismatch, scope_overclaim and unsupported_claim: the wording or figure needs fixing
Safety notes: scripts and network
This skill runs scripts and uses the network. Its tool list includes Bash, and the reviewer step calls an external reviewer model, so paper and result content is sent to that reviewer. If your data is confidential or under an embargo, confirm that the reviewer route you configure is permitted for it before you run the skill.
The skill's own guidance says not to wrap it in /loop, /schedule or CronCreate. Re-running it on a timer adds no new information, because its verdict only changes when the paper or results change. Schedule the wait for the draft to be ready, then run the audit once.
- Runs scripts and uses the network: check your reviewer route and data policy first
- Do not run it on a timer or in a loop
- Keep executor summaries such as EXPERIMENT_LOG.md out of the audit: the skill excludes them on purpose
Tips for a useful audit
Make sure every number in the paper can be found in a result file in the project. Numbers that exist only in a notebook or in your head will be reported as missing evidence. State explicitly in the paper whether you report the average, median or best across seeds, because the reviewer checks that wording against the data.
Keep the raw result files in the repository and up to date with the paper. Fix the flagged items, then run the audit once more when the draft changes.
- Name the aggregation (average, median or best) wherever a seed-based number appears
- Keep result files in the repository so the reviewer can read them
- Rerun only after the paper or results change
What people use it for
- Catching rounding that goes beyond the displayed precision, such as 84.7% reported as 85.3%
- Checking that a relative improvement such as 'improves by 15%' matches the arithmetic from the result files
- Verifying that a claim of 'consistently outperforms' is supported by the datasets actually evaluated
- Confirming that a reported mean over seeds matches the number of runs present in the results
- Cross-checking that figure and table captions describe what the data actually shows
Frequently asked questions
How do I install the paper-claim-audit skill?
Download the skill's ZIP from this page and unzip it into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder for one project. In the Claude apps you can instead upload it under Settings → Capabilities → Skills.
Is paper-claim-audit free to use?
The source material for this page states the skill is released under the MIT licence. It does not state pricing for the reviewer model or any other service, so check the terms of whichever reviewer backend you configure.
What does paper-claim-audit need to run?
It needs an agent runtime that supports SKILL.md skills, access to the mcp__codex__codex reviewer tool or another configured reviewer backend, and the paper .tex files plus raw result files in the project. Without raw result files it has nothing to check claims against.
How is paper-claim-audit different from experiment-audit or result-to-claim?
Paper-claim-audit checks whether the paper reports the data truthfully and precisely. Experiment-audit checks whether the experiment code is honest, and result-to-claim checks whether the data scientifically supports a claim. The source says the other two are separate skills for those questions.
Why does the skill use a fresh reviewer instead of the same agent?
The executor that wrote the experiments and the paper may expect certain results, which creates confirmation bias. The reviewer starts with zero context and sees only the .tex files and raw results, so it compares what the paper says against the files without those expectations.
Can I run paper-claim-audit on a schedule?
No. The skill says not to wrap it in /loop, /schedule or CronCreate, because its verdict only changes when the paper or results change. Run it once when the draft is ready, and again only after the paper or results change.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use paper-claim-audit if you are preparing a paper for submission, running an automated research pipeline, or checking your own results against your logs, and you can provide raw result files and a reviewer. Do not use it to judge whether an experiment design or scientific conclusion is sound, or on a schedule, and do not use it on confidential data unless your reviewer route is permitted to receive it.
Download the skill