promptfoo
Evaluation & observability · Open source
About
An open-source CLI for testing prompts against assertions and red-teaming LLM apps for security issues.
Key features
- Config-driven prompt tests
- Model comparison matrix
- Red-team scans
- CI integration
Best for
- — Prompt regression tests in CI
- — Red-teaming for jailbreaks and leaks
- Pricing
- Open source
- Platforms
- CLI, Web
- Works with
- GitHub, OpenAI, Anthropic
- Last checked
- 2026-10-29
- Status
- Listed, not yet tested
Our take
Developer-friendly and free. It is config-heavy, which suits engineers more than prompt writers.
Features and pricing change often — confirm on promptfoo.dev before you commit.
Alternatives to promptfoo
| Tool | Type | Pricing | Best for |
|---|---|---|---|
| promptfoo (this) | Evaluation & observability | Open source | Prompt regression tests in CI |
| Langfuse | Evaluation & observability | Open source | Tracing LLM apps in production |
| LangSmith | Evaluation & observability | Freemium | LangChain and LangGraph users |
| Braintrust | Evaluation & observability | Freemium | Systematic evals before shipping |
| Arize Phoenix | Evaluation & observability | Open source | Local debugging of LLM traces |
| W&B Weave | Evaluation & observability | Freemium | Teams already on Weights & Biases |
| Helicone | Evaluation & observability | Freemium | Quick cost and usage visibility |