Skip to content
Get packs

Guide · Open source

How to use xiaozhi-esp32-server: self-host the ESP32 voice backend

Updated · written with AI from the project's own documentation and checked against our catalogue data

xiaozhi-esp32-server is the backend service for the open-source xiaozhi-esp32 hardware project, which runs voice assistants on ESP32 chips. It handles speech recognition, large language model replies, speech synthesis and device control, so you can run your own instance. This guide explains what you need, how to get the code, how the deployment options differ, and what to configure.

小智 (Xiaozhi)xinnan-tech/xiaozhi-esp32-server · ★ 11.1k · MIT

Quick start

  1. 1Check prerequisites. Confirm you have an ESP32 voice device, Docker and Git installed.
  2. 2Clone the repository. Run the git clone command for xinnan-tech/xiaozhi-esp32-server.
  3. 3Pick a deployment level. Choose the minimal install for configuration-file storage or the full-module install for database-backed management.
  4. 4Choose a configuration profile. Use the free profile for personal use or the streaming profile for demos and more concurrent users.
  5. 5Add API keys. Enter keys for the speech, language, vision and speech-synthesis services you configured.
  6. 6Run the model tester. Run performance_tester.py to measure response speed for configured models.
  7. 7Connect your device. Point your ESP32 device at your server's OTA and WebSocket addresses as described in the docs.

What you need before you start

The project is built to serve ESP32 voice-assistant hardware, so you need an ESP32 device running the xiaozhi-esp32 firmware. The README says it is aimed at people who have already connected such a device to the official xiaozhi backend and now want to host their own.

  • An ESP32 voice-assistant device
  • Docker, for the Docker deployment route described in the README
  • Git, to clone the repository
  • API keys for the third-party services you choose, since the default setup uses external components

Get the code with git clone

Clone the repository from GitHub. The README lists this as the install command for the project.

git clone https://github.com/xinnan-tech/xiaozhi-esp32-server

Choose a deployment: minimal or full module

The README offers two install levels. The minimal install runs the server only, stores data in configuration files and needs no database. The full-module install adds multi-user management, multi-agent management and the web control console, with data kept in a database.

Each level has a Docker route and a source-code route. The README notes that running FunASR locally uses a lot of memory, while sending all components to APIs keeps resource use low.

  • Minimal install: smart dialogue and single-agent management, no database
  • Full-module install: smart dialogue, multi-user and multi-agent management, control console UI, database required

Configure the components

The README describes two configuration profiles. The free profile is meant for personal home use and uses free components. The streaming profile suits demos, training, or more than two concurrent users and aims for faster responses.

The default free profile uses FunASR (local) for speech recognition, glm-4-flash for the language model, glm-4v-flash for vision, EdgeTTS for speech synthesis, function_call for intent, and mem_local_short for memory. The streaming profile swaps in services such as XunfeiStreamASR, qwen-flash, qwen3.5-flash and HuoshanDoubleStreamTTS. Only models with configured keys are tested by the speed tool.

  • Free profile: FunASR, glm-4-flash, glm-4v-flash, EdgeTTS, function_call, mem_local_short
  • Streaming profile: XunfeiStreamASR, qwen-flash, qwen3.5-flash, HuoshanDoubleStreamTTS, function_call, mem_local_short

Test the server and models

The repository includes two test tools. The model response tester measures speed for ASR, LLM, VLLM and TTS. The audio interaction tool checks that audio playback and reception work on the Python side. The audio tool runs from the digital-human folder and is served on a local address.

  • Model response tester: run python performance_tester.py from main/xiaozhi-server
  • Audio interaction tool: start python start.py in main/digital-human after installing its dependencies, then open the local test page
python performance_tester.py

Understand the connection options

The server accepts devices over MQTT+UDP and WebSocket, and it provides MCP access points. Speech recognition supports multiple languages, and the project includes voiceprint recognition and RAGFlow knowledge-base support. Tool calling covers IoT and MCP protocols plus custom functions.

The README also lists a plugin system: new plugin files load the next time a device connects, while edits to already loaded plugin code need a service restart.

  • MQTT+UDP gateway, WebSocket and HTTP server
  • Voiceprint registration and recognition
  • RAGFlow knowledge base
  • Custom plugins; restart the service after changing already loaded plugin code

Keep it off the public internet

The README states that the project has not passed a network security review and should not be used in production. If you deploy it on a public network for study, the README asks you to add the necessary protection. It also says the project does not hold account keys and is not a payment service; choose API providers that hold relevant licences and read their privacy policies.

What people use it for

  • Run a private voice assistant backend for an ESP32 device at home
  • Move a device off a shared backend and onto your own server
  • Compare speech recognition, language model and speech synthesis options by measuring response times
  • Experiment with tool calling, MCP access points and custom plugins on low-cost hardware
  • Test voiceprint recognition or a RAGFlow knowledge base with a voice device

Frequently asked questions

Is xiaozhi-esp32-server free?

The project is released under the MIT licence, as shown in the repository. The README also describes a free-component configuration, but third-party API services you connect to may have their own terms and costs.

What do I need to run xiaozhi-esp32-server?

You need an ESP32 voice-assistant device, Docker for the Docker route, and Git to clone the repository. The default setup also needs API keys for the external speech and language services you choose.

Can I use xiaozhi-esp32-server in production?

No. The README says the project has not passed a network security review and should not be used in production. For public-internet deployments, add the protections the README recommends.

Does xiaozhi-esp32-server work without an ESP32 device?

The README says the server is built to serve the ESP32 device project, so it is designed for that hardware. Without an ESP32 device it has little use.

What is the difference between the minimal and full-module install?

The minimal install runs the server with data stored in configuration files and no database. The full-module install adds multi-user and multi-agent management, a control console and database storage.

Are there alternatives to xiaozhi-esp32-server?

The source does not name comparable projects, so this guide cannot compare it with alternatives. Check the README and project docs before deciding.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use xiaozhi-esp32-server if you own ESP32 voice hardware, want your own backend, and are comfortable editing configuration and managing API keys. Skip it if you need a turnkey setup, don't have the hardware, or plan to run a public production service, which the README advises against.

Install options & downloads