Skip to content
Get packs

bitnet.cpp

Official C++ inference framework for running 1.58-bit (ternary) LLMs fast on CPUs and GPUs.microsoft/BitNet

GitHub Download source
Stars
41.4k
Forks
3.9k
Licence
MIT
Language
C++
Last update
3 months ago
Type
Framework

What it is

bitnet.cpp is Microsoft's official inference framework for 1-bit large language models, such as the BitNet b1.58 family that stores weights with roughly 1.58 bits. It provides optimized kernels so these compressed models can run on ordinary CPUs, and the README also mentions GPU support. People use it to run language models locally with less memory and energy than full-precision models.

  • Run the official BitNet-b1.58-2B-4T model on a local x86 or ARM CPU
  • Convert and run BitNet embedding models using the I2_S kernel
  • Benchmark inference speed and energy use against full-precision models
  • Try chat-style interaction with the 2B model, per the README's conversational mode note

Great for

  • Developers who want to run BitNet b1.58 models locally on a CPU
  • Researchers exploring low-bit or ternary model inference
  • Teams testing CPU-based embedding models with I2_S conversion
  • Engineers comparing speed and energy use against full-precision inference

Not ideal for

  • Users who need to run standard non-BitNet models without conversion, since the project targets 1-bit models specifically
  • Non-technical users looking for a ready-made chat app, because setup requires building from source and using the command line
  • Anyone needing NPU support today, since the README says NPU support is coming next
Step-by-step guide: how to use bitnet.cppInstall, first run, examples, tips and FAQs
bitnet.cpp alternativesbitnet.cpp vs llama.cppbitnet.cpp vs Ollamabitnet.cpp vs ONNX Runtime

How to get it

  1. 1

    Clone and build from source

    Clone the repository

    git clone https://github.com/microsoft/BitNet
  2. 2

    Run it

    • Open the online demo to see what the models can do
    • Clone the repository and follow the build-from-source section of the README
    • Download a supported model from the Hugging Face collection linked in the README
    • Run inference on your CPU and compare results with the README benchmarks
  3. 3

    Get the most out of it

    • Use the I2_S kernel for embedding models on x86 CPUs, as described in the I2_S guide
    • Check the supported models table to pick the right kernel (I2_S, TL1, or TL2) for your CPU
    • Try the GPU kernel documentation if you have a supported GPU
    • Read the technical report for details on the CPU optimization approach

Downloads

Source code (.zip)main branch · straight from GitHub

Strengths

  • Official project from Microsoft with published technical reports and benchmarks
  • Supports both x86 and ARM CPUs with listed kernel options (I2_S, TL1, TL2)
  • MIT license with an online demo available for trying it before installing

Trade-offs

  • Setup requires building from source and comfort with the command line
  • Speedup and energy figures come from the project's own benchmarks and depend on hardware and model size
  • Kernel support varies by model and CPU architecture, as shown in the supported models table

How it compares

ProjectStarsTypeLevelUpdatedBest for
BitNet (this)41.4kFrameworkadvanced3 months agoDevelopers who want to run BitNet b1.58 models locally on a CPU
llama.cpp133kLibraryintermediate4 days agoDevelopers who want to run open-weight models locally without a cloud API
ollama186kAppbeginner4 days agoDevelopers who want a local model API for their own apps
onnxruntime22.2kLibraryintermediate4 days agoDevelopers who want to deploy a trained model inside an application or service

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.