Guide · Agent skill
How to use the NeMo Curator Data Curation agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The nemo-curator skill teaches an AI agent how to use NVIDIA's NeMo Curator toolkit to clean and prepare large datasets for training language models. This guide explains what the skill does, how to install it from the ZIP file, how to ask for help in plain language, and what you need before it is useful. By the end you will know whether it fits your data work and how to run it safely.
NeMo Curator Data CurationOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download and unzip. Download the nemo-curator ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder.
- 2Or upload the ZIP. In the Claude apps, open Settings → Capabilities → Skills and upload the ZIP.
- 3Install NeMo Curator. Run uv pip install "nemo-curator[text_cuda12]" for text work on CUDA 12, or use the all or cpu extra if needed.
- 4Describe your task. Ask the agent in plain language, naming your data type, the problem and the output format.
- 5Review the code. Check file paths, model names and any network or cloud data sources in the generated script.
- 6Run on a sample. Test the pipeline on a small slice of your data and inspect the filtered and deduplicated output.
What does the nemo-curator skill make the agent do?
Agent skills are folders of instructions and reference material that an AI agent loads when a task matches them. This one covers data curation for LLM training: removing low-quality or repeated documents, finding near-duplicate text on GPUs, redacting personal information, and filtering images, video and audio.
The skill mainly provides code patterns and API examples. The agent uses them to write pipelines that fit your data rather than running a fixed tool for you.
- Quality filtering with heuristic filters such as word count, repeated lines, symbol-to-word ratio and URL ratio
- Exact, fuzzy (MinHash + LSH) and semantic (embedding-based) deduplication
- PII redaction for entities such as emails, phone numbers, persons and locations
- Classifier-based quality filtering and NSFW filtering
- Image, video and audio curation examples
Install the nemo-curator skill from the ZIP file
Download the nemo-curator ZIP from this page and unzip it. The folder contains SKILL.md, references/deduplication.md and references/filtering.md. Put the unzipped folder in one of these locations:
For a personal install, place it in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
- Personal: ~/.claude/skills/nemo-curator/
- Project: .claude/skills/nemo-curator/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Install the NeMo Curator package
The skill's examples need the NeMo Curator Python package. The source recommends CUDA 12 with the text extra for text work. Choose the extra that matches your data. The CPU option is described as slower.
- Text curation on CUDA 12: nemo-curator[text_cuda12]
- All modalities on CUDA 12: nemo-curator[all_cuda12]
- CPU only (slower): nemo-curator[cpu]
uv pip install "nemo-curator[text_cuda12]"Ask the agent for curation work in plain language
Once the skill is installed, you do not need to name it. Describe the task and mention your data format and goal. Requests that match the skill's topics, such as deduplicating a corpus or cleaning web text, should load it automatically.
Good requests name the data, the problem and the output you want. For example, you might say: 'Write a NeMo Curator pipeline that removes exact and near-duplicate documents from my parquet files of web text, then saves the result to a new parquet folder.'
- Name the data type: text, images, video or audio
- Name the problem: duplicates, low-quality pages, personal information, unsafe images
- Name the output: a parquet folder, a filtered dataset or a script you can rerun
Example: a basic text pipeline
The skill includes a short text pipeline that loads a DataFrame, applies a quality filter, removes exact duplicates and saves to Parquet. Your agent will adapt this pattern to your own columns and file paths.
For large web corpora, the skill also shows a Common Crawl style pipeline that chains word count, repeated lines, symbol ratio, URL ratio, language identification, deduplication, PII redaction and NSFW filtering. Review each stage before running it on real data.
from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates(id_field="id", text_field="text")(dataset)
deduped.to_parquet("curated_data/")Safety and what the skill needs
The skill runs scripts and uses the network. Examples load embedding and classifier models from Hugging Face, such as sentence-transformers/all-MiniLM-L6-v2 and nvidia/quality-classifier-deberta. The source also mentions reading data from S3 paths in some examples. Check any path, model name and data source before running code the agent writes.
The PII redactor and NSFW filters are shown as basic examples. The source does not verify their accuracy, so do not rely on them alone for compliance.
- Requirements: Python, the nemo-curator package, an NVIDIA GPU with CUDA 12 recommended, RAPIDS, cuDF, Dask and dask-cuda
- Network: model downloads from Hugging Face and possible cloud data paths
- Review generated code before running it on production or sensitive data
Tips for better results
Start on a small sample so you can check filter thresholds before processing the full corpus. Tune the fuzzy deduplication settings to your data: the reference notes that more hashes are more accurate, and more buckets are faster but may miss some matches. Ask the agent to explain why each threshold was chosen.
- Test on a small slice first
- Check the fuzzy deduplication parameters: num_hashes, num_buckets and jaccard_threshold
- Keep the original data so you can compare results
What people use it for
- Deduplicating a large parquet corpus of web-scraped text before LLM training
- Filtering low-quality or repetitive documents using heuristic thresholds
- Finding near-duplicate documents with MinHash and LSH on GPUs
- Redacting emails and phone numbers from a text dataset as a first pass
- Filtering image or video datasets with the provided curation examples
Frequently asked questions
Is the nemo-curator skill free?
The skill is listed under the MIT licence in its source. NeMo Curator itself is installed as a Python package, and the source does not state pricing for it or for any cloud resources you might use.
Do I need an NVIDIA GPU to use it?
The GPU features and most examples depend on NVIDIA GPUs with CUDA. The source recommends CUDA 12 and describes the CPU install as slower, so CPU-only users will have a harder time.
How do I install the nemo-curator skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or a project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
How do I install the NeMo Curator package?
The source gives uv pip install "nemo-curator[text_cuda12]" for text work on CUDA 12. Other extras are all_cuda12 for all modalities and cpu for CPU-only installs.
Is NeMo Curator better than datatrove or dolma?
The skill lists datatrove as a CPU-based open-source option, dolma as Allen AI's data toolkit, and Ray Data for general ML processing without a curation focus. Choose NeMo Curator when you need GPU acceleration on large datasets; choose a CPU tool for smaller jobs.
Can I rely on the PII redaction for compliance?
No. The examples are basic and their accuracy is not verified in the skill files. Treat redaction as a first pass and check the output yourself before relying on it for compliance.
Why does my agent need network access?
Some examples download embedding and classifier models from Hugging Face, such as sentence-transformers/all-MiniLM-L6-v2 and nvidia/quality-classifier-deberta, and some show reading data from S3 paths.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you are an ML engineer or data team with NVIDIA GPUs who needs to deduplicate or clean large text, image, video or audio datasets for training. Skip it for small or one-off cleaning jobs, for machines without NVIDIA GPUs, or if you need a verified compliance tool for personal data.
Download the skill