Skip to content
Get packs

Hugging Face Local Model Evals

Runs benchmark evaluations on Hub models on your own GPU or via inference providers

Source Download skill
Repository
huggingface/skills
Stars
11,253
Licence
Apache-2.0
Files
5 files

What it does

Teaches the agent how to run standard AI model benchmarks (such as MMLU or GSM8K) against models hosted on the Hugging Face Hub, on local hardware. It explains which of three provided scripts to use, how to check the GPU and token setup, how to start with a small smoke test, and how to choose between vLLM, Transformers, and accelerate backends. It deliberately does not handle remote cloud jobs, publishing results, or model-card edits.

Great for

  • ML engineers who want to measure a Hub model's benchmark scores on their own GPU
  • Researchers comparing open models on tasks like MMLU or GSM8K
  • Developers who want a quick local smoke test of a model before scaling up

Skip it if

  • Users without a GPU who need heavy evaluations: the skill expects local NVIDIA hardware (nvidia-smi) or a provider-backed script, and large models may need remote compute handed off to another skill
  • Anyone wanting to publish scores to model cards or the community evals leaderboard: the skill explicitly excludes publishing and PR creation
  • Running untrusted models: the Transformers fallback can use --trust-remote-code, which executes custom code from the model repository

Try asking

  • “Run a quick 20-sample MMLU smoke test on meta-llama/Llama-3.2-1B using my local GPU”
  • “Evaluate microsoft/phi-2 on GSM8K with the Transformers backend because vLLM doesn't support it”
  • “Run leaderboard-style lighteval tasks on Llama-3.2-3B-Instruct with the chat template enabled”
Step-by-step guide: how to use Hugging Face Local Model EvalsInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 5 original files, the Apache-2.0 licence and an attribution note.

    huggingface-community-evals.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip huggingface-community-evals.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip huggingface-community-evals.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Always run a smoke test with --limit 10 or --max-samples 10 before a full evaluation to catch setup errors cheaply
  • Only use --trust-remote-code for model repositories you have reviewed, since it runs code from the model repo
  • Keep HF_TOKEN in your environment rather than pasting it into prompts, and check that it is scoped to the models you need

What's inside

  • examples/USAGE_EXAMPLES.md
  • scripts/inspect_eval_uv.py
  • scripts/inspect_vllm_uv.py
  • scripts/lighteval_vllm_uv.py
  • SKILL.md

Redistributed unmodified from huggingface/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.