Guide · Open source

How to use DeepEP: install and run expert-parallel communication

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

DeepEP is an open-source library from deepseek-ai that moves data between GPUs for mixture-of-experts (MoE) models, where different parts of a model handle different inputs. It provides GPU kernels for sending tokens to the right experts and gathering results back, including FP8 dispatch. This guide explains who it suits, what your system needs, how to install it and how to set up an EPBuffer for expert-parallel communication.

DeepEPdeepseek-ai/DeepEP · ★ 10.4k · MIT Install options & downloads

Quick start

1Check your hardware. Confirm you have Linux, NVIDIA Hopper or newer GPUs, and an NVLink or RDMA network for multi-GPU communication.
2Install the toolchain. Install Python 3.10 or newer, CUDA Toolkit 13.1 or newer, a C++20 compiler, and PyTorch 2.10 or newer with CUDA support.
3Clone and initialize the submodule. Clone the repository and run git submodule update --init --recursive to fetch DeepJIT.
4Install NCCL. Install the matching NCCL package, for CUDA 13.x with python -m pip install "nvidia-nccl-cu13>=2.32.3" --no-deps.
5Build and install. Run bash install.sh to build a wheel and install it into your current Python environment, then import deep_ep.
6Create a buffer. Initialize an EPBuffer per EP group with num_max_tokens_per_rank, hidden size, top-k and FP8 dispatch settings that match your model.

What DeepEP does and who it is for

DeepEP is a communication library for training and running large MoE models. Its core job is expert-parallel (EP) all-to-all communication: dispatching tokens to the GPUs that hold their assigned experts, then combining the results back.

The README describes it as suited to teams training or serving MoE models across multiple NVIDIA GPUs, and to engineers who need high-throughput or low-latency kernels. It also lists experimental pieces for pipeline, context and data parallelism, and for Engram remote memory access.

Good fit: MoE models spread across several NVIDIA GPUs, with NVLink inside a node and RDMA between nodes.
Poor fit: dense (non-MoE) models, or hardware older than NVIDIA Hopper.

Check the requirements before you install

DeepEP has a demanding setup. The README lists Linux, Python 3.10 or newer, NVIDIA Hopper or newer GPUs, CUDA Toolkit 13.1 or newer for DeepJIT compilation, a C++20 compiler with std::format support, PyTorch 2.10 or newer with CUDA support, and NCCL 2.32.3 or newer.

NVLink for intranode communication
RDMA network for internode communication
Hopper or newer GPU: installation does not need a visible GPU, because kernels compile at runtime for the current device

Install NCCL and build DeepEP

Clone the repository, then initialize the DeepJIT submodule. Install the NCCL package that matches your CUDA version. For CUDA 13.x the README uses the command below; for CUDA 12.x it says to use nvidia-nccl-cu12 instead.

Build a wheel and install it into your current Python environment with install.sh. After that, import deep_ep in your project. If you use a custom NCCL, set EP_NCCL_ROOT_DIR to a directory containing include/ and lib/, and make sure PyTorch and DeepEP load the same NCCL shared library.

Initialize the DeepJIT submodule with git submodule update --init --recursive
Install the NCCL dependency for CUDA 13.x
Build the wheel and install it with bash install.sh

git submodule update --init --recursive
python -m pip install "nvidia-nccl-cu13>=2.32.3" --no-deps
bash install.sh

Create an EPBuffer for your expert-parallel group

High-throughput and low-latency EP operations share one EPBuffer interface. You initialize it with MoE settings such as hidden size, top-k and the maximum tokens per rank. SM and QP counts are estimated analytically and can be overridden per call.

Create and reuse one buffer per EP group. All ranks must agree on num_max_tokens_per_rank. Finish outstanding operations before replacing a buffer.

Reuse the buffer if the group matches and the existing size is large enough.
Call get_theoretical_num_sms to estimate the communication SM count.

import torch.distributed as dist
from typing import Optional

from deep_ep import EPBuffer

_buffer: Optional[EPBuffer] = None
_num_comm_sms: int = 0

def get_buffer(group: dist.ProcessGroup,
               num_max_tokens_per_rank: int,
               hidden: int,
               num_topk: int,
               num_experts: int,
               use_fp8_dispatch: bool = False) -> EPBuffer:
    global _buffer, _num_comm_sms
    required_bytes = EPBuffer.get_buffer_size_hint(
        group, num_max_tokens_per_rank, hidden,
        num_topk=num_topk, use_fp8_dispatch=use_fp8_dispatch,
    )
    if _buffer is not None and _buffer.group == group and _buffer.num_bytes >= required_bytes:
        _num_comm_sms = _buffer.get_theoretical_num_sms(num_experts, num_topk)
        return _buffer
    _buffer = EPBuffer(
        group,
        num_max_tokens_per_rank=num_max_tokens_per_rank,
        hidden=hidden,
        num_topk=num_topk,
        use_fp8_dispatch=use_fp8_dispatch,
    )
    _num_comm_sms = _buffer.get_theoretical_num_sms(num_experts, num_topk)
    return _buffer

Choose training or decoding settings

Training, prefill and decoding use the same dispatch and combine APIs. The difference is the do_cpu_sync flag. Set it to True in training and prefill, where exact output sizes and CPU-side expert counts are needed. Set it to False for decoding with GPU-side receive counts; outputs are then allocated to the configured capacity, and expert GEMMs must use handle.psum_num_recv_tokens_per_expert to find valid ranges.

Set expert_alignment to your grouped GEMM's token alignment. For DeepGEMM, the README uses deep_gemm.get_mk_alignment_for_contiguous_layout(). A new routing decision needs a fresh dispatch, even during decoding.

do_cpu_sync=True: training and prefill
do_cpu_sync=False: decoding with GPU-side receive counts

Run the tests

Build into the source tree with bash develop.sh, then run the test scripts. The tests need NumPy and spawn local GPU workers. For multi-node tests, run the same script on each node with a shared MASTER_ADDR and MASTER_PORT, setting WORLD_SIZE to the node count and RANK to the node index.

Weight-prefetch and gradient-reduction tests use a single NVLink domain, and PP tests require an RDMA-only group.

python tests/ep/test_ep.py
python tests/ep/test_prefetch_weights.py
python tests/ep/test_reduce_grads.py
python tests/engram/test_engram.py
python tests/pp/test_pp.py

bash develop.sh
python tests/ep/test_ep.py

What people use it for

Training large MoE models across several NVIDIA GPUs with expert-parallel all-to-all communication
Serving MoE models where decoding needs GPU-side receive counts and a fixed output capacity
Experimenting with FP8 dispatch to reduce data moved between experts
Research on expert load balancing, using the experimental weight-prefetch and gradient-reduction primitives on a single NVLink domain

Frequently asked questions

Is DeepEP free to use?

DeepEP is released under the MIT licence, according to the directory listing. The MIT licence generally allows use, modification and redistribution, but check the repository's licence file for the exact terms before you rely on it.

What hardware do I need to run DeepEP?

The README requires NVIDIA Hopper or newer GPUs, Linux, and NVLink for intranode communication plus an RDMA network for internode communication. Installation does not require a visible GPU, because kernels compile at runtime.

Does DeepEP work with dense models?

No. The library targets expert-parallel communication for mixture-of-experts models, so dense non-MoE models do not need it.

Why does the install fail with NCCL errors?

Make sure PyTorch and DeepEP load the same NCCL shared library. If you use a custom install, set EP_NCCL_ROOT_DIR to a directory with include/ and lib/. The README also notes that NCCL headers older than 2.31 need an exact version match.

What are the alternatives to DeepEP?

The directory lists DeepSpeed-MoE, Megatron-LM, NCCL and DeepEP-Ascend as comparable projects. DeepEP-Ascend is the README's version for HUAWEI Ascend 950 NPUs.

Can I use DeepEP without a multi-GPU setup?

The README's examples and tests are built around multi-GPU expert-parallel groups, and the tests spawn local GPU workers. The source does not describe a single-GPU workflow, so check the repository before assuming one works.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

DeepEP suits teams training or serving MoE models on NVIDIA Hopper-or-newer GPUs with NVLink and RDMA networking, and engineers who want high-throughput or low-latency expert-parallel kernels. If you need a simple plug-in, run dense models, or lack this hardware and a CUDA 13.1 toolchain, pick a more general option such as DeepSpeed-MoE or Megatron-LM.

Install options & downloads