Skip to content
Get packs

Guide · Agent skill

How to use the Long Context Extension agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The long-context agent skill gives an AI agent reference knowledge and code patterns for making transformer language models handle much longer inputs, such as 32k to 128k tokens, than they were originally trained on. This guide is for ML engineers and researchers who work with pre-trained models, and it explains what the skill does, how to install it, how to trigger it, and where its limits are. By the end you will know how to add the skill to your agent, ask it useful questions about RoPE, YaRN, ALiBi and position interpolation, and avoid the mistakes the source warns about.

Long Context ExtensionOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the ZIP. Download the long-context skill ZIP from this page.
  2. 2Install the skill. Unzip it into ~/.claude/skills/ for personal use or .claude/skills/ inside a project, or upload it under Settings → Capabilities → Skills in the Claude apps.
  3. 3Prepare your environment. Install Python, PyTorch and HuggingFace transformers, and add optional packages such as einops if you need them. Make sure you have a GPU for the examples.
  4. 4Ask a specific question. Describe your base model, target context length and preferred method, such as YaRN or position interpolation.
  5. 5Test and measure. Run any example code on your own model and measure long-context quality before relying on it.

What does the long-context skill make the agent do?

Once installed, the agent can explain positional encoding methods used to extend context windows: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases) and Position Interpolation. It can also write example code for these methods and suggest how to adjust a model's rope_scaling settings before fine-tuning.

The skill is a technical guide for model engineers rather than an end-user tool. It does not add document-processing ability by itself, so it will not summarise a long file for you on its own.

  • Explains RoPE, YaRN, ALiBi and position interpolation, including rotary embeddings, attention biases and interpolation methods
  • Provides illustrative code for rotary embeddings, ALiBi bias creation and a LLaMA rope_scaling example
  • Includes reference files on the extension methods, RoPE and fine-tuning for context extension

How do you install the long-context skill?

Download the ZIP file for the skill from this page. You can install it in one of two ways, depending on where you use your agent.

For Claude Code, unzip the download into your user skills folder ~/.claude/skills/, or into a project-specific folder .claude/skills/ inside your project. For the Claude apps, upload the ZIP under Settings → Capabilities → Skills.

  • Personal install: unzip into ~/.claude/skills/ so the skill is available across your projects
  • Project install: unzip into .claude/skills/ in the project folder so it applies only there
  • Claude apps: upload the ZIP under Settings → Capabilities → Skills

What does the skill need before you run the examples?

The skill's examples assume Python with PyTorch and HuggingFace transformers. Some optional packages add features, and one needs a special build. The examples also assume a GPU, because they use device='cuda'. Example code that loads a model needs access to its weights, such as meta-llama/Llama-2-7b-hf.

The skill itself runs no scripts and uses no network. Installing packages and downloading model weights do require internet access, though.

  • Required: Python, PyTorch, HuggingFace transformers
  • Optional: einops, rotary-embedding-torch, flash-attn (flash-attn needs a CUDA build)
  • A GPU is assumed by the examples
  • Access to model weights such as meta-llama/Llama-2-7b-hf for the example code
pip install transformers torch pip install einops pip install rotary-embedding-torch pip install flash-attn --no-build-isolation

How do you trigger the skill with a natural request?

You do not need special syntax. Describe the task in plain language and mention the context length you need or the method you are interested in. The skill's description says it applies when you are processing long documents (32k to 128k+ tokens), extending a pre-trained model beyond its original context limit, or implementing efficient positional encodings.

Requests that name the model, the target length or the method tend to give the most useful answers.

  • "Explain how YaRN extends a RoPE model to 128k tokens and what its main parameters do."
  • "Show me how to set rope_scaling on a LLaMA model to reach a 32k context window."
  • "Write a minimal ALiBi bias function for 8 attention heads and explain how it is added to the scores."
  • "Compare RoPE, YaRN, ALiBi and position interpolation for extending an existing model."

How do you use the rope_scaling example?

The source includes a Position Interpolation example for a LLaMA model. It sets the rope_scaling configuration to a linear or dynamic scaling type with a factor, so that positions are scaled down into the trained range. The example assumes an original context of 2048 tokens, and a factor of 16 gives 32768 tokens.

Treat this as a starting point. The source says the examples are illustrative, and it does not show a measured result for your model. Fine-tuning on long documents is part of the approach, and you should evaluate long-context quality on the exact model you plan to use.

  • Choose the scaling type and factor so that original context times factor matches your target length
  • Check the model's native context length in its own documentation before applying the factor
  • Measure output quality at your target length before relying on it
from transformers import LlamaForCausalLM model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf") model.config.rope_scaling = {"type": "linear", "factor": 16.0}

What safety and reliability points should you know?

The skill runs no scripts and uses no network, so it is low risk as a set of reference files. The main risks are practical ones: the code is reference material and not a tested production implementation, and the examples have not been validated for your model or data.

Before you deploy a model with extended context, measure long-context quality on the target model. Do not assume the examples will work unchanged.

  • No scripts are run by the skill and it makes no network calls
  • The reference code is not a tested production implementation
  • Long-context quality must be measured on the target model before production use
  • Check licence and access terms for any model weights you download

What tips help you get the most from it?

Start from the method that fits your situation. The skill's method comparison describes YaRN as aimed at extending existing models, while RoPE is described as suited to new models. Ask the agent for the trade-offs before choosing.

Read the reference files when you need detail. The extension methods file compares YaRN, ALiBi and position interpolation, and the fine-tuning file covers data preparation and evaluation.

  • Name your base model and target context length in the same request
  • Ask for the trade-offs between methods rather than only for code
  • Use the fine-tuning reference when you plan to train on long documents
  • Ask the agent to point out assumptions in any example before you run it

What people use it for

  • Explaining how RoPE, YaRN, ALiBi and position interpolation differ before choosing one for a project
  • Drafting example code for rotary embeddings or an ALiBi attention bias to adapt in your own codebase
  • Planning how to set rope_scaling on a LLaMA-style model before a fine-tuning run
  • Comparing extension approaches for researchers who are writing a literature review

Frequently asked questions

How do I install the long-context skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or into .claude/skills/ inside a project. In the Claude apps, upload the ZIP under Settings → Capabilities → Skills.

Is the long-context skill free?

The skill is listed under the MIT licence in its source repository. Check the licence text in the download for the full terms, and check separate access terms for any model weights you use.

What do I need to run the long-context examples?

You need Python, PyTorch and HuggingFace transformers. The examples assume a GPU because they use device='cuda', and some code needs access to model weights such as meta-llama/Llama-2-7b-hf.

Does the skill summarise long documents?

No. It gives the agent knowledge about extending a model's context window, not document-processing ability. People who only want summaries of long files should use a tool built for that task.

How does this compare with other context extension approaches?

The skill covers RoPE, YaRN, ALiBi and position interpolation. YaRN is described as suited to extending existing models, while ALiBi relies on attention biases instead of positional embeddings. Ask the agent for a comparison based on your model and target length.

Why does my extended model give poor results?

The examples are illustrative and not tested production code, and the source notes that long-context quality must be measured on the target model. Check your scaling factor against the model's native context length, then evaluate output quality at your target length.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits ML engineers, researchers and developers who want a clear starting point for extending a pre-trained LLaMA or Mistral-style model to a longer context window, or who want to compare RoPE, YaRN, ALiBi and position interpolation. It is not a good choice for people who only want to summarise long documents, or for production deployments that have not been validated on their own model and data.

Download the skill