Skip to content
Get packs

Guide · Agent skill

How to use the TorchTitan distributed pretraining agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

This page explains how to install and use the distributed-llm-pretraining-torchtitan agent skill, a set of instructions and reference files that teach an AI agent how to pretrain large language models with torchtitan, PyTorch's distributed training platform. It is written for readers who have not used agent skills before. By the end you will know how to add the skill to Claude, phrase a request that triggers it, and what hardware, accounts and safety checks to consider before launching a training run.

TorchTitan Distributed PretrainingOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the ZIP. Download the skill ZIP from this page so you have the SKILL.md folder and its reference files.
  2. 2Install the skill. Unzip into ~/.claude/skills/ or a project's .claude/skills/, or upload the ZIP under Settings → Capabilities → Skills in the Claude apps.
  3. 3Install dependencies. Install torchtitan with pip and torchao from git for Float8, with PyTorch 2.6.0 or newer.
  4. 4Describe your training job. Ask in plain language, naming the model, GPU count and whether you need Float8 or multi-node training.
  5. 5Download the tokenizer. Use your Hugging Face token to fetch the gated Llama tokenizer assets with the provided script.
  6. 6Launch and monitor. Start a debug run, check TensorBoard logs in ./outputs/tb/, then scale up with SLURM if needed.

What is this skill and what does it make the agent do?

An agent skill is a folder with a SKILL.md file and optional reference files. When your request matches the skill's description, the agent loads these instructions and follows them. This skill covers pretraining LLMs from scratch with torchtitan, using 4D parallelism (FSDP2, tensor parallelism, pipeline parallelism and context parallelism).

With the skill installed, the agent can write TOML configuration files, launch single-node or multi-node jobs with torchrun or SLURM, enable Float8 on H100 GPUs, and create or resume distributed checkpoints. The reference files cover checkpointing, adding custom model definitions, and Float8 and FSDP details.

Install the skill in Claude

Download the ZIP file from this page. Then choose one of two options.

For Claude Code, unzip the folder into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder to share it with that project. The skill folder should contain SKILL.md at its top level.

For the Claude apps, open Settings, go to Capabilities, then Skills, and upload the ZIP.

  • Personal install: ~/.claude/skills/
  • Project install: .claude/skills/ inside your project
  • Claude apps: Settings → Capabilities → Skills

Install the torchtitan dependencies

The skill expects PyTorch 2.6.0 or newer, torchtitan 0.2.0 or newer, and torchao 0.5.0 or newer. Install torchtitan from PyPI for the stable release. For Float8, torchao is installed from git as shown in the source.

The source also describes a from-source option for the latest features, which requires PyTorch nightly.

pip install torchtitan USE_CPP=0 pip install git+https://github.com/pytorch/ao.git

How do you trigger the skill with natural requests?

You do not need special syntax. Describe the training job you want in plain language and mention the model or scale. Requests that match the skill's description work best, for example asking to pretrain Llama 3.1 or DeepSeek V3, or a custom model, on many GPUs with Float8 or distributed checkpointing.

Be specific about hardware. The skill's workflows assume multi-GPU clusters, so say how many GPUs or nodes you have and whether a scheduler such as SLURM is available.

  • Set up a single-node Llama 3.1 8B pretraining run on 8 GPUs with checkpoints every 500 steps.
  • Write a SLURM script to train a 70B model across 32 nodes with 8 GPUs each.
  • Enable Float8 for my H100 cluster and show me the TOML changes.
  • Create a seed checkpoint before I run pipeline parallelism on 405B.

Walk through a single-node run

The skill's basic workflow has four steps: download the tokenizer, write a TOML config, launch training, then monitor and checkpoint. The tokenizer download uses a Hugging Face access token, because the Llama tokenizers are gated.

Launch on 8 GPUs with the included run script, or with torchrun directly. TensorBoard logs are written to ./outputs/tb/.

  • Step 1: download the tokenizer with your Hugging Face token.
  • Step 2: configure training in a TOML file.
  • Step 3: launch the run.
  • Step 4: monitor with TensorBoard and rely on checkpoints for recovery.
python scripts/download_hf_assets.py \ --repo_id meta-llama/Llama-3.1-8B \ --assets tokenizer \ --hf_token=YOUR_HF_TOKEN CONFIG_FILE="./llama3_8b_custom.toml" ./run_train.sh

Scale to multi-node with SLURM

For multi-node work, the skill sets parallelism degrees in the TOML file and provides a SLURM batch script that calls torchrun. Training auto-resumes if a checkpoint already exists in the configured folder, so resubmitting a job continues from the last saved state.

Parallelism degrees must match your hardware. The source example for a 70B model on 256 GPUs uses FSDP across 32 ranks with tensor parallelism of 8 inside each node.

sbatch multinode_trainer.slurm

Safety notes before you run it

This skill runs scripts and uses the network. The workflows download tokenizer assets from Hugging Face using your token, and they launch jobs that can occupy many GPUs. A misconfigured run can waste expensive compute, so check the TOML file, parallelism degrees and GPU count before submitting. Read any script the agent writes before executing it, and keep your Hugging Face token out of shared files.

The skill is not suitable for fine-tuning or small experiments on one consumer GPU. The source recommends Axolotl or TRL for those.

  • Verify parallelism degrees multiply to your total GPU count.
  • Start with a small debug run before a long cluster job.
  • Do not commit your access token to a repository.

Tips for better results

Give the agent your exact model flavor, GPU count, node count and whether you have H100s. Those details decide the parallelism settings and whether Float8 makes sense. If training runs out of memory, the source suggests switching activation checkpointing from selective to full and lowering the local batch size.

If Float8 is not faster, the source notes it only benefits large matrix multiplications, so small layers can be filtered out.

  • Ask for a debug config first, then scale up.
  • If you change parallelism, ask the agent to check whether the checkpoint needs converting with DCP.
  • Pipeline parallelism needs a seed checkpoint first.

What people use it for

  • Pretraining a Llama 3.1 8B model from scratch on a single 8-GPU node.
  • Scaling a 70B model across 32 nodes with FSDP and tensor parallelism using a SLURM script.
  • Enabling Float8 training on an H100 cluster with torch.compile.
  • Preparing a 405B-scale 4D parallelism run with a seed checkpoint.
  • Adding a custom model definition following the repository's directory pattern.

Frequently asked questions

How do I install the TorchTitan agent skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or a project's .claude/skills/ folder. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

What are the requirements for this skill?

You need Python with PyTorch 2.6.0 or newer, torchtitan 0.2.0 or newer, and torchao 0.5.0 or newer. NVIDIA GPUs are needed, with H100 recommended for Float8, and multi-node work needs SLURM or a similar scheduler.

Is the TorchTitan skill free to use?

The skill is published under the MIT licence. The source does not state any price for it, so check the directory page for any terms beyond the licence.

Do I need a Hugging Face account?

Yes, for the workflows that download gated Llama tokenizers and assets. You need a Hugging Face account and an access token, which the tokenizer download script takes as a parameter.

What should I use instead of TorchTitan for fine-tuning?

The skill itself points to Axolotl or TRL for fine-tuning rather than pretraining. It also lists Megatron-LM, DeepSpeed and LitGPT as alternatives in other situations.

Why is my Float8 training not faster?

The source says Float8 only benefits large matrix multiplications. Filtering small layers such as attention.wk, attention.wv and output out of Float8 conversion can help.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits ML engineers pretraining Llama-style or custom LLMs on clusters of 8 or more GPUs who want a PyTorch-native stack. It is the wrong choice for fine-tuning, small experiments on a single consumer GPU, or anyone without multi-GPU access, where the source recommends simpler tools.

Download the skill