Skip to content
Get packs

Guide · Agent skill

How to use the Vision Model Trainer agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The Vision Model Trainer is an agent skill that lets an AI agent set up and launch training runs for computer-vision models on Hugging Face Jobs, which use cloud GPUs. It covers object detectors, image classifiers and SAM/SAM2 segmentation models. This guide explains what the skill does, how to install it, how to ask for training in plain language, and what you need and should watch out for before running a job.

Vision Model Trainerhuggingface/skills · Apache-2.0

Quick start

  1. 1Download and unzip the skill. Download the skill ZIP from this page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder.
  2. 2Confirm your account and token. Make sure you have a Pro, Team or Enterprise Hugging Face plan and a token with write permission.
  3. 3Validate your dataset. Ask the agent to check the dataset, or run the dataset inspector, and fix any NEEDS FORMATTING result before training.
  4. 4Choose options. Answer the agent's questions about a quick test run on a subset and whether to create a validation split.
  5. 5Launch and monitor the job. Let the agent write and submit the Hugging Face Jobs training run with a sufficient timeout, and track progress with Trackio.
  6. 6Check the model on the Hub. Once the job finishes, confirm the trained model is saved to your Hugging Face Hub account.

What does the skill make the agent do?

Once installed, the agent can prepare and launch training for three kinds of vision model. Object detectors find and box objects in images (for example D-FINE, RT-DETR v2, DETR and YOLOS). Image classifiers label whole images, using timm models such as MobileNetV3, MobileViT, ResNet and ViT/DINOv3, or any Transformers classifier. SAM and SAM2 models produce masks from bounding-box or point prompts.

The agent first checks your dataset format with a helper script, then writes and submits a training job to Hugging Face Jobs. The finished model is saved to your Hugging Face Hub account, because the training environment is temporary and files are deleted when a job ends. The skill also includes a cost estimator and guidance on hardware, timeouts and monitoring with Trackio.

  • Object detection: COCO-style or Hub-hosted datasets with an objects column holding bbox and category fields
  • Image classification: a dataset with an image column and a label column
  • SAM/SAM2 segmentation: an image column, a mask column and a prompt (bbox or point)

How do you install the Vision Model Trainer skill?

Download the skill ZIP from the page where you found this guide. Then choose one of two install routes. For a personal setup, unzip it into ~/.claude/skills/. For a single project, unzip it into that project's .claude/skills/ folder.

If you use the Claude apps instead, open Settings, go to Capabilities, then Skills, and upload the skill there.

  • Personal install: unzip into ~/.claude/skills/
  • Project install: unzip into the project's .claude/skills/ folder
  • Claude apps: Settings → Capabilities → Skills, then upload

What do you need before you start?

You need a Hugging Face account with a Pro, Team or Enterprise plan, because Jobs require a paid plan. The skill itself is free to read. Your Hugging Face token must have write permission, and it must be passed to the job as a job secret, not pasted into the script.

The helper scripts use uv to run and pull their Python dependencies at runtime. These include transformers, accelerate, timm, datasets, albumentations, torchmetrics, pycocotools, trackio, huggingface_hub, evaluate, scikit-learn and torchvision. The scripts also use the network and run code, so review them before use.

  • Hugging Face account on a Pro, Team or Enterprise plan
  • Hugging Face token with write permission, passed as a job secret
  • uv installed locally to run the helper scripts
  • A dataset that already exists on the Hub, or is ready to upload

How do you trigger the skill with a natural request?

You do not need a special command. Describe the vision task in ordinary words and mention the model family or the kind of task. The skill's triggers include phrases such as training object detection, image classification, SAM, SAM2, segmentation, image matting, DETR, D-FINE, RT-DETR, ViT, timm, MobileNet, ResNet, bounding box models, or fine-tuning vision models on Hugging Face Jobs.

Before any GPU time is spent, the agent validates your dataset format. Its workflow also asks you preferences, such as whether to run a quick test on a subset of the data and whether to create a validation split from the training data.

  • "Fine-tune D-FINE on my COCO-format dataset username/my-boxes and push the model to my Hub account."
  • "Train a ResNet classifier on my image dataset with a label column."
  • "Fine-tune SAM2 with bounding-box prompts on my mask dataset."
uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train

What should you check before a training run?

The dataset inspector reports either READY, meaning the dataset can be used directly, or NEEDS FORMATTING, meaning preprocessing is needed and mapping code is provided in the output. For object detection, the script converts between xywh (COCO) and xyxy (Pascal VOC) boxes automatically and remaps string categories to integer IDs. For SAM, bounding boxes should be in xyxy absolute pixel coordinates.

Set a job timeout longer than the expected training time, because the skill notes the default 30 minutes is too short. Check that Hub push is enabled, otherwise the trained model is lost when the job ends. You can estimate cost with the included script by running it with --help first.

  • Run the dataset inspector before paying for GPU time
  • Set the job timeout above expected training time
  • Confirm the model will be pushed to the Hub
  • Use the cost estimator to plan spending
uv run scripts/estimate_cost.py --help

What are the safety and privacy limits?

Training runs remotely on cloud infrastructure, and results are pushed to the Hugging Face Hub. Do not use this skill with sensitive or private images that you do not want uploaded to a cloud provider and the Hub. Use a token with only the permissions the job needs, and keep it in job secrets.

The skill is also not for language-model or text fine-tuning. The skill says that work belongs to hugging-face-model-trainer, which uses TRL-based training.

  • Images are uploaded to the cloud and results go to the Hub
  • Scripts run code and the skill uses the network
  • Keep your token in job secrets, never in shared files

What people use it for

  • Fine-tune a detector such as D-FINE or RT-DETR v2 on your own labelled product or wildlife photos without owning a GPU.
  • Train a timm classifier like MobileNetV3 or ResNet to sort images into your own categories.
  • Fine-tune SAM2 to produce masks from bounding-box prompts for image matting on a mask dataset.
  • Set up a repeatable training pipeline for a team whose COCO-style dataset lives on the Hub.

Frequently asked questions

Is the Vision Model Trainer skill free?

The skill itself is free to read and install. Running it depends on Hugging Face Jobs, which require a Pro, Team or Enterprise plan and cloud GPU usage, so training has a cost.

What do I need to install the huggingface-vision-trainer skill?

Download the ZIP and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder, or upload it in the Claude apps under Settings → Capabilities → Skills. You also need uv to run the helper scripts.

Do I need a paid Hugging Face plan to use this skill?

Yes for training. The skill relies on Hugging Face Jobs, which need a Pro, Team or Enterprise plan. Without one you can read the skill but cannot run the cloud training jobs it submits.

Can this skill fine-tune language models or LLMs?

No. The skill covers vision models only. Text and language-model fine-tuning belongs to the separate hugging-face-model-trainer skill, which uses TRL-based training.

Why does my training job fail or time out?

Common causes are a dataset format mismatch and a timeout shorter than the training time. Run the dataset inspector first, fix any NEEDS FORMATTING result, and set the job timeout above the expected training time, since the default 30 minutes is too short.

Is my data private when I use this skill?

Not necessarily. Training runs remotely on cloud GPUs and results are pushed to the Hugging Face Hub, so avoid sensitive or private images you do not want uploaded to a cloud provider and the Hub.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you want to fine-tune detectors, classifiers or SAM models on your own labelled images, you have a paid Hugging Face plan, and your images can be shared with a cloud provider and the Hub. Pick something else if you need to keep images fully private or you want to fine-tune language models, which the skill says belongs to a different skill.

Download the skill