NeMo Evaluator SDK
Run standardized LLM benchmarks (MMLU, GSM8K, HumanEval, safety) across Docker, Slurm or cloud.
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 5 files
What it does
Once installed, the agent can set up and run NVIDIA's NeMo Evaluator launcher to score language models on many standard benchmarks such as MMLU, GSM8K, HumanEval and IFEval, plus safety and vision-language tests. It writes the YAML config files, launches runs on a local Docker setup, a Slurm HPC cluster or cloud, checks job status, and exports results to formats like JSON, MLflow or Weights & Biases. Results are meant to be reproducible because each task runs in a container.
Great for
- ML engineers who need to compare several models on the same benchmark suite
- Researchers who want containerized, repeatable benchmark runs
- Teams with Slurm HPC clusters or local GPUs who need large evaluation jobs
Skip it if
- Quick one-off checks of a single prompt, since setting up the launcher, containers and config is heavy for that
- Users without Docker or cluster access, because the workflows depend on them
- Anyone expecting results to be valid without checking the config, since sample limits and temperature settings change scores
Try asking
- “Evaluate my local vLLM endpoint on IFEval, GSM8K and MMLU-Pro and give me the results file path”
- “Set up a Slurm config to run humaneval on 8 GPUs with a 4-hour walltime”
- “Compare Llama 3.1 8B and Mistral 7B on the same task list and export both runs to MLflow”
- “Run a quick 10-sample smoke test of the safety tasks before a full evaluation”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 5 original files, the MIT licence and an attribution note.
nemo-evaluator.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip nemo-evaluator.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip nemo-evaluator.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Start with a small limit_samples run to confirm the endpoint and API key work before launching full benchmarks.
- Keep API keys in environment variables and reference them by name in config files rather than pasting them in.
- Review the generated config and the Slurm hostname and account before running, since jobs will execute on external infrastructure.
What's inside
- references/adapter-system.md
- references/configuration.md
- references/custom-benchmarks.md
- references/execution-backends.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.