Skip to content
Get packs

Guide · Open source

How to use latent-diffusion: setup, text-to-image and inpainting

Updated · written with AI from the project's own documentation and checked against our catalogue data

latent-diffusion is the official research code from CompVis for High-Resolution Image Synthesis with Latent Diffusion Models. It runs diffusion in a compressed latent space, which the README says makes generation more efficient than working on full-size pixels. This guide is for readers comfortable with conda, Python and the command line; by the end you will have the environment set up and know how to run text-to-image sampling and inpainting from the provided scripts.

Latent DiffusionCompVis/latent-diffusion · ★ 14.2k · MIT

Quick start

  1. 1Clone the repository. Download the CompVis/latent-diffusion code and move into its folder.
  2. 2Create the conda environment. Run conda env create -f environment.yaml to build the ldm environment.
  3. 3Activate ldm. Run conda activate ldm before running any script.
  4. 4Download a checkpoint. Fetch the text-to-image weights (5.7GB) into models/ldm/text2img-large/.
  5. 5Generate samples. Run scripts/txt2img.py with a prompt and check outputs/txt2img-samples.
  6. 6Tune settings. Adjust scale, ddim_steps and ddim_eta to trade speed, quality and diversity.

What you need before installing latent-diffusion

The repository is set up as a clone-and-build install. The README asks for a conda environment and Python, and the interface is command-line scripts only; there is no GUI described in the source.

Plan for disk space. The README lists several downloads: the text-to-image weights are 5.7GB, the Openimages retrieval database is about 11GB, and the Openimages search index is about 21GB.

  • conda installed
  • Python (the README does not specify a version)
  • Enough free disk space for the checkpoints you plan to use
  • Basic comfort with a terminal

Install latent-diffusion with conda

Clone the repository, then move into its folder. From there, create the conda environment named ldm from the provided environment.yaml file, and activate it. The README uses this environment name for every later step.

conda env create -f environment.yaml conda activate ldm

Download the text-to-image checkpoint and generate images

Download the pre-trained text-to-image weights (5.7GB) into the path the README uses, then run the txt2img script with a prompt. Each sample is saved on its own, and a grid of n_iter by n_samples is saved too, by default in outputs/txt2img-samples.

The README describes the main controls: scale, ddim_steps and ddim_eta. Higher scale gives better samples but less diversity. Raising ddim_steps generally improves quality, but the README says returns diminish above 250. Setting ddim_eta to 0.0 gives faster sampling while keeping good quality, and adding --plms gives even faster sampling.

  • scale: higher values follow the prompt more closely at the cost of diversity
  • ddim_steps: more steps usually improve quality; diminishing returns above 250
  • ddim_eta 0.0: faster sampling with good quality
  • --plms: lower step counts for faster sampling, per the README
mkdir -p models/ldm/text2img-large/ wget -O models/ldm/text2img-large/model.ckpt https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large/model.ckpt python scripts/txt2img.py --prompt "a virus monster is playing guitar, oil on canvas" --ddim_eta 0.0 --n_samples 4 --n_iter 4 --scale 5.0 --ddim_steps 50

Change the output size with H and W

The H and W arguments set the image size. The README says they are integer-divided by 8 to get the latent size. The source notes that running the model on larger sizes than it was trained on can sometimes give interesting results, but controllability is reduced compared with the 256x256 setting.

python scripts/txt2img.py --prompt "a sunset behind a mountain range, vector image" --ddim_eta 1.0 --n_samples 1 --n_iter 1 --H 384 --W 1024 --scale 5.0

Run inpainting with your own image and mask pairs

Inpainting needs the inpainting checkpoint and a folder of input files. The README says the input folder should contain images named *.png and masks named <image_fname>_mask.png, following the examples in data/inpainting_examples.

  • Put images as *.png in your input folder
  • Name each mask <image_fname>_mask.png
  • Outputs go to the folder you pass as --outdir
wget -O models/ldm/inpainting_big/last.ckpt https://heibox.uni-heidelberg.de/f/4d9ac7ea40c64582b7c9/?dl=1 python scripts/inpaint.py --indir data/inpainting_examples/ --outdir outputs/inpainting_results

Try retrieval-augmented models (optional)

The README includes inference code for retrieval-augmented diffusion models. These need extra Python packages installed into the ldm environment, a preliminary checkpoint, and retrieval databases. The source says the Openimages database cannot be used for sampling until scripts/train_searcher.py has created its index. The README caps the number of neighbors at 20.

  • Extra packages: transformers==4.19.2, scann, kornia==0.6.4, torchmetrics==0.6.0, and einops from GitHub
  • Openimages database (~11GB) plus its index (~21GB)
  • ArtBench databases (~82MB) are smaller and suited to more abstract concepts
python scripts/knn2img.py --prompt "a happy pineapple" --use_neighbors --knn <number_of_neighbors>

What people use it for

  • Researchers reproducing or building on the latent diffusion paper
  • Developers running text-to-image sampling locally with the provided checkpoints
  • Experimenting with inpainting using your own image and mask pairs
  • Testing non-square outputs with the H and W arguments

Frequently asked questions

Is latent-diffusion free to use?

The repository is released under the MIT licence, as listed on its page. The README presents it as research code, so check the licence terms for your own use.

What do I need to install latent-diffusion?

The README asks for conda and Python. You create an environment named ldm from environment.yaml and activate it. Extra packages are needed only for the retrieval-augmented models.

How much disk space does latent-diffusion need?

The README lists a 5.7GB text-to-image checkpoint, an Openimages retrieval database of about 11GB, and an Openimages index of about 21GB. ArtBench databases are about 82MB. The exact space you need depends on which models you download.

Does latent-diffusion have a graphical interface?

The README describes only command-line scripts. A web demo on Hugging Face Spaces is mentioned in the news section, but the repository itself is run from the terminal.

What is the difference between latent-diffusion and Stable Diffusion?

This source does not compare them in detail. The listed comparable projects include Stable Diffusion, Diffusers and AUTOMATIC1111 stable-diffusion-webui, which are often used with more interface options.

Why does sampling with the Openimages database fail?

The README says that sampling with the Openimages database is not possible until scripts/train_searcher.py has been run to create the index. Run that script first, then use --database openimages.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use latent-diffusion if you are a researcher or developer who wants to run the official code and checkpoints from the paper, and you are comfortable with conda and the terminal. Pick a tool with a graphical interface or a supported production service if you need a polished app or guaranteed stability, since the README describes this as research code with preliminary checkpoints.

Install options & downloads