Guide · Agent skill
How to use the NeMo Evaluator SDK agent skill: install and run
Updated · written with AI from the project's own documentation and checked against our catalogue data
The nemo-evaluator-sdk skill teaches an AI agent to set up and run NVIDIA's NeMo Evaluator launcher, which scores language models on many standard benchmarks. This guide is for ML engineers, researchers and teams with clusters who want containerized, repeatable benchmark runs. By the end you will know how to install the skill, ask for an evaluation in plain language, and check what the agent needs from you before it starts.
NeMo Evaluator SDKOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the skill ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. In the Claude apps, upload it under Settings → Capabilities → Skills instead.
- 2Install the launcher and Docker. Install the launcher with pip install nemo-evaluator-launcher and make sure Docker is available. The workflows depend on both.
- 3Set your API key. For NVIDIA-hosted endpoints, export NGC_API_KEY. Add HF_TOKEN only if a task such as gpqa_diamond needs it.
- 4Ask for the evaluation. Describe the model, the benchmarks and the backend (local Docker, Slurm or cloud) in one request. The agent writes the YAML config for you.
- 5Launch and watch the run. The agent runs the launcher. Use the invocation ID to check status and read the results file when the run finishes.
- 6Export and compare. Export each run to local JSON, MLflow or Weights & Biases so models can be compared side by side.
What the skill makes the agent do
Agent skills are folders of instructions that an AI agent loads when a task matches them. This one covers NeMo Evaluator, which evaluates LLMs across 100+ benchmarks from 18+ harnesses, including MMLU, HumanEval, GSM8K, safety tests and vision-language (VLM) tests.
Once the skill is loaded, the agent can write the YAML config files, launch runs on local Docker, a Slurm HPC cluster or cloud (Lepton is named in the source), check job status, and export results. Each task runs in a container, which is why the source describes results as reproducible.
Install the skill
Download the skill ZIP from the page where you found this guide, then choose one of the two routes below.
For Claude Code, unzip the folder into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder to share it with that project. For the Claude apps, upload the ZIP under Settings → Capabilities → Skills.
- Personal install: unzip into ~/.claude/skills/
- Project install: unzip into .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Install the launcher the skill depends on
The skill drives the NeMo Evaluator launcher, which is installed with pip. The skill's dependencies list also includes Docker. Install the launcher before asking for a run.
pip install nemo-evaluator-launcherAsk for an evaluation in plain language
The skill description says to use it when you need scalable evaluation on local Docker, Slurm HPC or cloud platforms. Name the model, the benchmarks and where you want the run to happen, and the agent will pick the matching workflow.
Requests that work well follow that pattern, for example: evaluate an endpoint on MMLU, GSM8K and IFEval using local Docker, or set up a Slurm run with vLLM deployment for a checkpoint on shared storage. Mention the output format you want, such as JSON or MLflow.
- Compare two or more models on the same tasks
- Run safety tasks such as aegis, wildguard or garak
- Run vision-language tasks such as ocrbench, chartqa or mmmu
- Check the status of a run you already started
Run a standard benchmark
For a standard run, the agent writes a config that sets the execution backend, the model endpoint and the task list. The source's minimal example uses the local execution backend, no deployment, an NVIDIA-hosted endpoint and the ifeval task. You set your key first, then run against the config directory.
For a self-hosted endpoint such as vLLM, the source uses a localhost URL and an empty api_key_name. You can add -o overrides on the command line, for example to change the output directory or to limit samples for a quick test.
export NGC_API_KEY=nvapi-your-key-here
nemo-evaluator-launcher run --config-dir . --config-name config
nemo-evaluator-launcher ls tasksCheck status and read results
Each run has an invocation ID. Use it to check status, list runs, or open the results file. The source notes that status queries sacct on Slurm runs. Results for each task are written under the output directory.
- Status: nemo-evaluator-launcher status <invocation_id>
- All runs: nemo-evaluator-launcher ls runs
- Results file: results/<invocation_id>/<task>/artifacts/results.yml
nemo-evaluator-launcher status <invocation_id>
cat results/<invocation_id>/<task>/artifacts/results.ymlExport results and compare models
To compare models, the skill's workflow runs the same base config with different model_id and url overrides, then exports each invocation. The source shows exports to MLflow, to local JSON, and to Weights & Biases. Those destinations need your own accounts or local setup.
- Keep temperature and parallelism identical across runs so the comparison is fair
- Export each invocation to MLflow or W&B, or to local JSON
nemo-evaluator-launcher export <invocation_id> --dest local --format json
nemo-evaluator-launcher export <invocation_id> --dest mlflow
nemo-evaluator-launcher export <invocation_id> --dest wandbRun on a Slurm cluster
For large jobs, the agent writes a config that uses the slurm execution backend and a deployment such as vllm. The config holds cluster settings like hostname, account, partition, walltime, nodes and GPUs per node. Fill these in with your cluster's real values; the source uses placeholders such as cluster.example.com.
- Settings include hostname, account, partition, output_dir, walltime, nodes and gpus_per_node
- To stop a run: nemo-evaluator-launcher kill <invocation_id>
- For details: nemo-evaluator-launcher info <invocation_id>
What people use it for
- Comparing several open models on the same suite of MMLU-Pro, GSM8K and IFEval tasks
- Running a containerized benchmark on a lab GPU machine with Docker
- Launching a large evaluation on a Slurm HPC cluster with a vLLM deployment
- Running safety tasks such as aegis, wildguard and garak against a model endpoint
- Scoring a vision-language model on ocrbench, chartqa or mmmu
Frequently asked questions
Is the NeMo Evaluator SDK agent skill free?
The skill is published under the MIT licence in the Orchestra-Research/AI-Research-SKILLs repository. The source does not state pricing for NVIDIA-hosted endpoints, so check those terms yourself before running paid or metered calls.
What do I need to use this skill?
You need Python with pip, the nemo-evaluator-launcher package, and Docker. NGC_API_KEY is needed for NVIDIA-hosted endpoints, and HF_TOKEN is optional for some tasks such as gpqa_diamond. Slurm, W&B or MLflow accounts are only needed for those specific workflows.
How do I install an agent skill like this one?
Download the ZIP and unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder. In the Claude apps, upload the ZIP under Settings → Capabilities → Skills.
Can I run this without Docker or a cluster?
Not as described. The source says the workflows depend on Docker or cluster access, so users without either will find the setup difficult.
Does the skill run scripts and use the network?
Yes. The skill's requirements state that it runs scripts and uses the network. It launches containerized runs and calls model endpoints, so review the config and endpoint URL before you start a job.
How does NeMo Evaluator compare with lm-evaluation-harness?
The source says lm-evaluation-harness has a simpler setup for quick local evaluation, while NeMo Evaluator covers 100+ benchmarks across many harnesses, supports Slurm and cloud runs, and offers containerized reproducibility and result export.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you regularly compare models on a shared benchmark suite and have Docker, a GPU machine or a Slurm cluster. Pick something lighter, such as lm-evaluation-harness, if you only want a quick local check of a few prompts, or if you do not have Docker or cluster access. Always review the config, since sample limits and temperature change scores.
Download the skill