Skip to content
Get packs

Guide · Open source

How to use LocalAI: install, run models and connect an API

Updated · written with AI from the project's own documentation and checked against our catalogue data

LocalAI is an open-source, self-hosted AI server. It runs language, vision, voice, image and video models on your own hardware and exposes an API that is compatible with OpenAI, Anthropic and ElevenLabs. This guide shows you how to start it with Docker, load a model, and send it requests, so you can use it as a private stand-in for a cloud AI service.

LocalAImudler/LocalAI · ★ 50.1k · MIT

Quick start

  1. 1Install Docker or podman. Install Docker or podman on the machine that will run LocalAI. The container route depends on one of these.
  2. 2Start the LocalAI container. Run the CPU-only docker run command from the README, which maps port 8080 and names the container local-ai.
  3. 3Load a model. Run local-ai run with a model gallery name, such as llama-3.2-1b-instruct:q4_k_m, or with a Hugging Face or Ollama path.
  4. 4Test the model. Start the terminal chat with local-ai chat in a second terminal, or send a request to the API on port 8080.
  5. 5Connect your tools. Set your OpenAI-compatible client to use your LocalAI server address as its base URL.

What you need before you install LocalAI

The container route requires Docker or podman. LocalAI's README says it can run without a GPU, so a CPU-only machine is enough to start. Models and backends are downloaded on demand and can be large, so check that you have enough free disk space before you begin.

  • Docker or podman installed and running
  • Enough free disk space for models and backends, which are downloaded when needed
  • Optional: an NVIDIA, AMD, Intel or Vulkan GPU if you want hardware acceleration

Start LocalAI with Docker

For a CPU-only machine, run the container below. It names the container local-ai and maps port 8080 on your machine to port 8080 in the container. The first run downloads the image.

If you already ran LocalAI, restart the existing container with the command in the README instead of creating a new one.

  • Use the GPU variants in the README if you have an NVIDIA, AMD, Intel or Vulkan GPU; the README lists a separate command for each.
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest

Load a model into LocalAI

LocalAI can load models from several sources. The README lists the model gallery, Hugging Face, the Ollama OCI registry, a YAML config file, and a standard OCI registry such as Docker Hub. Use the model gallery for the simplest start.

You can browse available gallery models with the local-ai models list command or on the models site linked in the README.

  • Model gallery: a name such as llama-3.2-1b-instruct:q4_k_m
  • Hugging Face: a huggingface:// path to a GGUF file
  • Ollama registry: an ollama:// path
  • YAML config: a URL to a config file
  • OCI registry: an oci:// path
local-ai run llama-3.2-1b-instruct:q4_k_m

Chat with a model from the terminal

LocalAI includes a built-in terminal agent. It answers questions, reads your files and runs commands on your machine, and asks you to approve anything that changes state. Run the model in one terminal, then start the chat in a second one.

Inside a session, /models lists installed models and /model <name> switches between them. The README links to the Terminal agent documentation for more detail.

# Terminal 1 local-ai run llama-3.2-1b-instruct:q4_k_m # Terminal 2 local-ai chat --model llama-3.2-1b-instruct:q4_k_m

Point existing tools at your LocalAI API

Because LocalAI is compatible with the OpenAI API, tools that let you set a custom API base URL can use it instead of a cloud service. The server listens on port 8080 in the Docker setup above. The README describes the API as compatible with OpenAI, Anthropic and ElevenLabs, and each backend runs behind the same API.

Check the LocalAI documentation for the exact endpoint paths and authentication setup before you connect a production tool.

  • Set the base URL in your client to your LocalAI server address on port 8080
  • Use the model name you loaded in LocalAI as the model in your requests
  • Enable API key authentication if the server is reachable by other users

Choose the right hardware route

LocalAI detects your GPU capabilities and downloads the appropriate backend automatically, according to the README. You can also pick a GPU-specific image yourself. The README lists separate commands for NVIDIA GPUs (CUDA 12 and CUDA 13), NVIDIA Jetson ARM64, AMD GPUs through ROCm, Intel GPUs through oneAPI, and Vulkan GPUs.

For NVIDIA GPUs, the README uses the --gpus all flag and a CUDA-specific image tag. For AMD and Intel, it passes the GPU device paths into the container.

  • NVIDIA (CUDA 12): localai/localai:latest-gpu-nvidia-cuda-12 with --gpus all
  • AMD (ROCm): localai/localai:latest-gpu-hipblas with --device=/dev/kfd and --device=/dev/dri
  • Intel (oneAPI): localai/localai:latest-gpu-intel with the /dev/dri device paths
  • Vulkan: localai/localai:latest-gpu-vulkan
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12

What people use it for

  • Run a private OpenAI-compatible endpoint for a development team on an internal server
  • Test text, voice and image models behind one API during development
  • Keep prompts and data inside your own infrastructure for privacy-sensitive work
  • Use an older or mixed-hardware machine, including CPU-only systems, as a local AI server

Frequently asked questions

Is LocalAI free to use?

LocalAI is open source and released under the MIT licence, as stated in the README. You still pay for your own hardware, electricity and disk space for the models.

Does LocalAI need a GPU?

No. The README says LocalAI runs without a GPU, and it offers a CPU-only Docker command. A GPU can add acceleration if you have an NVIDIA, AMD, Intel or Vulkan card.

How do I install LocalAI?

The README's main install route is Docker or podman. Run the CPU-only docker run command with port 8080 mapped, then load a model with local-ai run.

Is LocalAI a good alternative to Ollama?

LocalAI focuses on a server with an OpenAI-compatible API that covers many model types, including voice, image and video. Ollama, llama.cpp, vLLM and LM Studio are comparable projects, so pick based on which model types and API you need.

Why is my LocalAI download so large?

Models and backends are downloaded on demand, and the README notes they can be large. Free up disk space or load smaller quantized models such as those with a q4 suffix.

Is my data sent to the cloud when I use LocalAI?

The README says LocalAI keeps data on your own infrastructure. Your prompts stay on the machine where you run it, though you should confirm any model downloads you make come from sources you trust.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

LocalAI suits developers and teams who want an OpenAI-compatible API on their own machines, with privacy and support for many model types. It is less suited to people who just want a simple desktop chat app, or who want a fully managed cloud service with no setup work.

Install options & downloads