Promptfoo

A testing framework for prompts, models, and RAGs to systematically evaluate and improve LLM output quality.

Promptfoo is a tool for AI and LLM developers to test and evaluate the quality of their models, prompts, and Retrieval-Augmented Generation (RAG) setups. It provides a systematic way to create test cases, run evaluations against various LLM providers, and compare the outputs side-by-side in a unified web viewer. By defining expected outcomes and assertions in a simple configuration file, developers can automate the process of scoring outputs based on metrics like correctness, similarity, latency, and cost. Designed for individual developers and teams, promptfoo integrates directly into the development workflow. It can be run from the command line, incorporated into CI/CD pipelines to prevent quality regressions, and used to find the optimal combination of prompts and models for a specific task. Its support for a wide range of providers—from commercial APIs like OpenAI and Anthropic to open-source models run locally via Ollama—makes it a versatile solution for anyone building reliable AI-powered applications.