Skip to content
Get packs

Guide · Open source

How to use promptfoo: install, run evals and red team LLM apps

Updated · written with AI from the project's own documentation and checked against our catalogue data

Promptfoo is a command-line tool and library for evaluating and red-teaming LLM-based apps. This guide walks you through installing it, running your first evaluation, and understanding where red teaming and CI/CD integration fit. By the end you will know how to run a starter example and view the results.

promptfoopromptfoo/promptfoo · ★ 26.7k · MIT

Quick start

  1. 1Check prerequisites. Confirm Node.js is installed and get an API key for the LLM provider you plan to test.
  2. 2Install promptfoo. Run npm install -g promptfoo to install the CLI globally.
  3. 3Set your API key. Export your provider key as an environment variable, such as OPENAI_API_KEY.
  4. 4Create the starter example. Run promptfoo init --example getting-started to generate a sample project.
  5. 5Run the evaluation. Move into the getting-started folder and run promptfoo eval.
  6. 6View the results. Run promptfoo view to open the results in the web viewer and compare outputs.

What you need before you install promptfoo

Promptfoo is installed with npm, so you need Node.js on your machine. You also need an API key for each LLM provider you want to test, because most providers require one.

  • Node.js (npm is used for the install command)
  • An API key for the LLM provider you want to test

Install promptfoo with npm

Install promptfoo globally with npm. The README also lists Homebrew and pip as alternatives, and offers npx for running any command without installing.

  • Alternatives: brew install promptfoo or pip install promptfoo
  • Without installing: npx promptfoo@latest
npm install -g promptfoo

Set your API key

Most LLM providers need an API key set as an environment variable. Replace the example value with your own key.

export OPENAI_API_KEY=sk-abc123

Run the getting-started example

Create a starter example, move into its folder, run the evaluation and open the results viewer. The eval compares outputs from the configured prompts, models and test cases.

promptfoo init --example getting-started cd getting-started promptfoo eval promptfoo view

Understand the config-driven workflow

Promptfoo uses a declarative config that lists prompts, models and test cases. You write the config, run it, and compare outputs across providers such as OpenAI, Anthropic, Azure, Bedrock and Ollama. Because the setup is declarative, the same evaluation can be repeated over time.

  • Prompts: the text templates you want to test
  • Models: the providers you want to compare side by side
  • Test cases: the inputs and checks run against each combination

Use red teaming and CI/CD integration

Promptfoo includes red-teaming features that scan LLM apps for security vulnerabilities and can generate vulnerability reports. Its docs describe automating checks in CI/CD pipelines and reviewing pull requests with code scanning. Scanning is a tool to help find issues, not a guaranteed security audit.

  • Red teaming guide: https://www.promptfoo.dev/docs/red-team/
  • CI/CD integration: https://www.promptfoo.dev/docs/integrations/ci-cd/
  • Code scanning: https://www.promptfoo.dev/docs/code-scanning/

What people use it for

  • Testing whether a prompt change improves answers across a fixed set of test cases
  • Comparing several models side by side on the same prompts
  • Running automated LLM checks in a CI/CD pipeline
  • Scanning an LLM app locally for security vulnerabilities through red teaming

Frequently asked questions

Is promptfoo free?

Promptfoo is open source and MIT licensed, according to the README. You still pay the LLM providers whose APIs you call, since you supply your own API keys.

What does promptfoo require to run?

You need Node.js because the install uses npm, and an API key for each LLM provider you want to test. The README notes that most providers require an API key.

How do I install promptfoo?

Run npm install -g promptfoo. The README also lists brew install promptfoo and pip install promptfoo, and npx promptfoo@latest can run commands without installing.

Does promptfoo work with Ollama or local models?

The README lists Ollama among supported providers for comparing models. It also says LLM evals run locally on your machine, and prompts never leave it.

What are alternatives to promptfoo?

The comparable projects named in our analysis are LangSmith, Ragas, DeepEval and Giskard. Choose based on whether you need a command-line workflow, red teaming, or a hosted tool.

Is promptfoo a no-code tool?

No. The workflow centres on the command line and config files, so it suits developers more than people who want point-and-click testing.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Promptfoo suits developers building on LLM APIs who want repeatable prompt tests, model comparisons and CI/CD checks from the command line. It is a poor fit for no-code users, for projects that call no LLM APIs, or for anyone who needs a guaranteed security audit.

Install options & downloads