TorchTitan Distributed Pretraining
Guides an agent through large-scale LLM pretraining with PyTorch torchtitan across many GPUs
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 5 files
What it does
When installed, the agent gains a playbook for pretraining large language models from scratch using torchtitan, PyTorch's distributed training platform. It can write TOML configuration files, launch single-node or multi-node training jobs (torchrun, SLURM), split work across GPUs with parallelism strategies (FSDP2, tensor, pipeline and context parallelism), enable Float8 on H100 GPUs, and create or resume checkpoints. The reference files cover checkpointing, custom model definitions, Float8 and FSDP details.
Great for
- ML engineers pretraining Llama-style or custom LLMs on clusters of 8 or more GPUs
- Teams who want a PyTorch-native stack without third-party training frameworks
- Researchers planning multi-node H100 runs who need config and SLURM templates
Skip it if
- Fine-tuning or small experiments on a single consumer GPU, where simpler tools such as Axolotl or TRL are more appropriate (the skill itself says so)
- Anyone without access to multi-GPU hardware, since the workflows assume clusters and can waste expensive compute if misconfigured
- Non-developers looking for a general chatbot or writing help; the skill only covers training infrastructure
Try asking
- “Set up a TOML config to pretrain Llama 3.1 8B on my 8-GPU node with selective activation checkpointing”
- “Write a SLURM script to run a 70B model across 32 nodes with tensor parallel degree 8”
- “Enable Float8 training on my H100s and explain how to launch it with torch.compile”
- “Help me debug out-of-memory errors when scaling my model's batch size”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 5 original files, the MIT licence and an attribution note.
torchtitan.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip torchtitan.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip torchtitan.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Always test configs on a small GPU count or with a short step limit before launching multi-day, multi-node jobs, since mistakes waste expensive compute
- Keep your Hugging Face token out of shared files and scripts; the examples show it inline and should be replaced with an environment variable
- Check the referenced files (checkpoint, FSDP, Float8) for version-specific flags, because the skill notes that some features need PyTorch nightly
What's inside
- references/checkpoint.md
- references/custom-models.md
- references/float8.md
- references/fsdp.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.