Skip to content
Get packs

DeepEP

A GPU communication library that speeds up data exchange between experts in mixture-of-experts AI models.

GitHub Download v1.2.1
Stars
10.4k
Forks
1.5k
Licence
MIT
Language
Cuda
Last update
16 days ago
Type
Library

What it is

DeepEP is a library for moving data between GPUs during training and running of large AI models that use mixture-of-experts (MoE) designs, where different parts of a model handle different inputs. It provides GPU kernels for sending tokens to the right experts and gathering results back, with options such as FP8 dispatch. The README also describes experimental pieces for other parallelism styles, but the core use is fast expert-parallel communication.

  • Dispatch and combine tokens across experts in an MoE training loop
  • Run prefill and decoding inference with the same EPBuffer dispatch and combine APIs
  • Prefetch expert weights over NVLink for dynamic redundant experts (experimental)
  • Run batched all-gather, reduce-scatter or all-reduce through BucketBuffer (experimental)

Great for

  • Teams training or serving MoE models across multiple NVIDIA GPUs
  • Engineers working on expert-parallel communication who need high-throughput or low-latency kernels
  • Groups with NVLink within a node and RDMA networking between nodes

Not ideal for

  • Anyone without NVIDIA Hopper-or-newer GPUs, since the README lists these as a requirement
  • Users running dense (non-MoE) models, since the library targets expert-parallel communication
  • Beginners looking for a simple plug-in, because setup needs CUDA 13.1+, NCCL, PyTorch 2.10+ and a C++20 compiler
Step-by-step guide: how to use DeepEPInstall, first run, examples, tips and FAQs

How to get it

  1. 1

    Clone and build from source

    You'll need: Linux · Python 3.10 or newer · NVIDIA Hopper or newer GPU · CUDA Toolkit 13.1 or newer · C++20 compiler

    Clone the repository and initialize the DeepJIT submodule

    git submodule update --init --recursive

    Install NCCL dependency for CUDA 13.x

    python -m pip install "nvidia-nccl-cu13>=2.32.3" --no-deps

    Build a wheel and install it

    bash install.sh
  2. 2

    Run it

    • Install the NCCL dependency that matches your CUDA version
    • Initialize the DeepJIT submodule and run bash install.sh
    • Import deep_ep in your Python project
    • Create an EPBuffer for your EP group and run the example dispatch and combine calls
  3. 3

    Get the most out of it

    • Use bash develop.sh to link the extension into the source tree for development
    • Set EP_NCCL_ROOT_DIR to point at a custom NCCL installation
    • Use do_cpu_sync=False with GPU-side receive counts for decoding workloads
    • Set expert_alignment to match your grouped GEMM, such as DeepGEMM's alignment helper
    • Run the included tests under tests/ to verify your setup

Downloads

Source code (.zip)main branch · straight from GitHubRelease v1.2.1Release notes and source

Strengths

  • Provides high-throughput and low-latency expert-parallel all-to-all GPU kernels in one EPBuffer interface
  • Released under the MIT license with a public test suite
  • Installation does not require a visible GPU, since kernels are compiled at runtime for the current device

Trade-offs

  • Many features (Bucket, Engram, PP) are labeled experimental in the README
  • Has strict hardware and software requirements, including Hopper-or-newer GPUs and RDMA for multi-node use
  • The README is truncated, so some API details and the full setup path are not visible here

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.