How to use localGPT: install with Docker and chat with your documents

What you need before you install localGPT

The README lists Python 3.10+ (3.11 recommended), Node.js 20+ and npm, Ollama, and Docker as optional for containerized deployment. Ollama is required for both deployment approaches.

For hardware, the README recommends 8GB+ RAM, with 16GB+ recommended for more comfort.

• Python 3.10+ (3.11 recommended, as the Docker images use python:3.11-slim)
• Node.js 20+ and npm
• Ollama (required for both approaches)
• Docker (optional, for containerized deployment)
• 8GB+ RAM (16GB+ recommended)

Install localGPT with Docker

Clone the repository, then install Ollama on your host and pull the two models the README names. The README recommends installing Ollama locally even when using Docker.

Start Ollama, then launch the Docker deployment in a new terminal and open the web interface at localhost port 3000.

• If you would rather not install Ollama on the host, run ./start-docker.sh container and pull the models inside the container with docker compose --profile with-ollama exec ollama ollama pull qwen3.5:9b.
• Running ./start-docker.sh with no argument uses local Ollama. If nothing listens on port 11434, it offers to switch to the containerized Ollama. Add -y or set NONINTERACTIVE=1 to accept that fallback without a prompt.

Manage the Docker deployment

Once containers are running, you can check their status, read logs and stop the stack using the commands the README provides.

• Check container status: docker compose ps
• View logs: docker compose logs -f
• Stop containers: ./start-docker.sh stop

Run localGPT directly with Python (developer path)

The README lists a direct development option for people who want to work on the code. It installs Python dependencies from requirements.txt and Node.js dependencies with npm, then starts the system with run_system.py.

The key Python dependencies named in the README include torch, transformers, lancedb (vector store and full-text search), rerankers (cross-encoder reranking) and docling (document parsing).

• Install Python dependencies: pip install -r requirements.txt
• Install Node.js dependencies: npm install
• Start the system: python run_system.py

Load documents and ask questions

localGPT parses PDF, DOCX, HTML/HTM, Markdown and TXT files using Docling. You build an index from your documents, then chat against it. Answers come back with the chunks they were grounded in, so you can check the source passages.

Each query can be routed either to retrieval over your documents or to a direct LLM answer, and complex questions are split into sub-questions before retrieval.

• Supported formats: PDF, DOCX, HTML/HTM, Markdown, TXT
• PDFs with no text layer are re-run through Docling's OCR pipeline
• Source attribution shows the chunks behind each answer
• Live progress streams retrieval, reranking and synthesis stages into the chat

Tune settings and use the API

Settings such as models, chunk size, retrieval mode and toggles can be changed per request. The main defaults live in rag_system/main.py and can be overridden by environment variable. The README says the RAG core exposes a REST gateway on port 8000 and the RAG API on port 8001.

Optional features such as Late Chunking, Answer Verification and multi-vector retrieval are off by default and can be enabled through config. For example, the README names the retrieval.latechunk.enabled flag and the verification.enabled setting.

• Default configuration: rag_system/main.py
• REST gateway: port 8000
• RAG API: port 8001
• Late Chunking flag: retrieval.latechunk.enabled (off by default)
• Answer Verification setting: verification.enabled (off by default)