Portkey AI Gateway vs promptfoo
Which should you use? A side-by-side look at what each one does, who it suits, what it costs, how hard it is to set up, and where each falls short.
Open source · MIT
Open-source gateway that routes requests to many AI models through one API, with guardrails.
Open source · MIT
CLI and library for testing LLM prompts and red-teaming AI apps, with declarative configs.
The short answer
Choose Portkey AI Gateway if you need…
- Developers who call several LLM providers and want one OpenAI-style interface
- Teams that want automatic retries or fallbacks when a model provider fails
- Apps that need content checks on model replies using configurable guardrails
- Developers who want a local gateway with a console showing request logs
Choose promptfoo if you need…
- Developers building apps on top of LLM APIs who want repeatable prompt tests
- Teams comparing several models side by side on the same prompts
- People who want to run automated LLM checks in CI/CD pipelines
- Teams looking for a local, command-line way to scan LLM apps for security issues
Side-by-side
| Portkey AI Gateway | promptfoo | |
|---|---|---|
| Type | Dev tool | Dev tool |
| Price | Free and open source | Free and open source |
| Licence | MIT | MIT |
| GitHub stars | 13.3k | 26.7k |
| Setup difficulty | Intermediate | Intermediate |
| How you get it | Install with a package manager | Install with a package manager |
| Platforms | — | CLI, Web |
| Works with | — | GitHub, OpenAI, Anthropic |
| Main language | TypeScript | TypeScript |
| Last code update | 6 months ago | 4 days ago |
What each one is
Portkey AI Gateway
Portkey's AI Gateway sits between your app and large language model providers. Your app sends requests to one API, and the gateway forwards them to the model provider you choose. The README says it adds routing, automatic retries, fallbacks, load balancing, and guardrails that check inputs or outputs, and it can be run locally or deployed.
promptfoo
Promptfoo is a command-line tool and library for evaluating LLM-based apps. You write a declarative config listing prompts, models, and test cases, then run it to compare outputs across providers such as OpenAI, Anthropic, and Ollama. It also includes red-teaming features that scan AI apps for security vulnerabilities.
Best for
Portkey AI Gateway
- Developers who call several LLM providers and want one OpenAI-style interface
- Teams that want automatic retries or fallbacks when a model provider fails
- Apps that need content checks on model replies using configurable guardrails
- Developers who want a local gateway with a console showing request logs
promptfoo
- Developers building apps on top of LLM APIs who want repeatable prompt tests
- Teams comparing several models side by side on the same prompts
- People who want to run automated LLM checks in CI/CD pipelines
- Teams looking for a local, command-line way to scan LLM apps for security issues
Strengths
Portkey AI Gateway
- The README describes a small footprint and low latency, and it is MIT licensed
- Works with an OpenAI-compatible client, so existing code can be pointed at it with limited changes
- Supports several deployment options including Docker, Node.js, and Cloudflare Workers
promptfoo
- The README states that evals run locally and prompts never leave your machine
- Works with many providers and any LLM API, according to the README
- Has a large community with about 26,500 stars and an MIT license
Not ideal for
Portkey AI Gateway
- People who only call a single model from one provider and do not need routing, since the gateway adds a layer to manage
- Non-developers without experience running Node.js or calling APIs, since setup assumes familiarity with code
- Organizations needing enterprise features like org management, which the README places in a separate paid Enterprise version
promptfoo
- People who want a no-code, point-and-click tool, since the workflow centers on the command line and config files
- Projects that do not call any LLM APIs, because there is nothing for it to evaluate
- Anyone who needs a guaranteed security audit, since the README describes scanning as a tool rather than a complete assurance
Trade-offs
Portkey AI Gateway
- Many headline numbers (model counts, latency, token volume) come from the README's own claims and were not independently verified here
- Some advanced features, such as enterprise governance and private deployments, are described as part of a separate Enterprise version
- The README mentions a Gateway 2.0 pre-release that is merging enterprise features into open source, so the project may change
promptfoo
- The README says the project is now part of OpenAI, which may matter to some users evaluating independence
- Requires API keys for most providers, which means usage costs from those providers
- Some claims in the README, such as usage by OpenAI and Anthropic and serving 10M+ users, are not independently verified here
Getting started
Portkey AI Gateway
Run the gateway locally
npx @portkey-ai/gatewayInstall the Python client
pip install -qU portkey-aipromptfoo
Install globally with npm
npm install -g promptfooCreate a starter example
promptfoo init --example getting-startedFrequently asked questions
What is the difference between Portkey AI Gateway and promptfoo?
Portkey AI Gateway: Open-source gateway that routes requests to many AI models through one API, with guardrails. promptfoo: CLI and library for testing LLM prompts and red-teaming AI apps, with declarative configs. Portkey AI Gateway suits developers who call several llm providers and want one openai-style interface, while promptfoo suits developers building apps on top of llm apis who want repeatable prompt tests.
Are Portkey AI Gateway and promptfoo free?
Portkey AI Gateway: Free and open source. promptfoo: Free and open source.
Is Portkey AI Gateway or promptfoo more popular?
promptfoo has more GitHub stars (26.7k) than Portkey AI Gateway (13.3k). Popularity isn't the same as fit — check which one matches your use case above.
What are other alternatives to Portkey AI Gateway and promptfoo?
Other options people compare them with include Botpress, Composio, Codebase to Tutorial, claude-context, opencodex.
Keep comparing
Built from our catalogue entries for both products: open-source details come from each project's README (AI-assisted, refreshed weekly) and GitHub data; AI-tool details are editorial. Spotted something wrong? Let us know from either product's page.