promptfoo
An open-source tool for testing, evaluating, and red-teaming prompts and models.
promptfoo evaluates prompts and models from a simple config: define test cases and assertions, run them across prompts and providers, and compare results side by side. It also does security red-teaming, probing models for unsafe behaviour.
It runs from the command line and slots into CI, making prompt changes something you measure rather than guess at.
Where it fits
A fit when you want config-driven evaluation across prompts and models, plus security red-teaming, in one CLI tool.
Where it does not fit
A testing and red-teaming tool, not a production tracing platform: pair it with one rather than replace it.
Alternatives to promptfoo
The closest options on the map, and where each fits instead.
-
DeepEval Evaluation framework
A fit when you want LLM quality checks written as tests and run automatically in CI.
Unlike promptfoo Maturity: growing
-
Ragas RAG evaluation
Essential when retrieval quality matters and you need to measure changes to a RAG pipeline rather than judge them by eye.
Unlike promptfoo Maturity: growing
Pairs with
Commonly used alongside it.
