Skip to main content
Promptfoo is an open source library for evaluating LLM output quality. With Portkey, you can:
  • Manage prompts with versioning and call them in Promptfoo
  • Run evals on 1600+ LLMs including private/local models
  • Track costs and metrics for all eval runs
  • Avoid rate limits with load balancing and caching

​
1. Reference Portkey Prompts

Use prompts stored in Portkey directly in Promptfoo:
  1. Set PORTKEY_API_KEY environment variable
  2. Use portkey:// prefix with your prompt ID:
prompts:
  - portkey://pp-test-prompt-669f48
providers:
  - openai:gpt-3.5-turbo-0613
tests:
  - vars:
      topic: ...
Variables from test cases are automatically plugged into the Portkey prompt.
Promptfoo doesn’t follow temperature, model, and other parameters set in Portkey. Set them in the providers configuration.

​
2. Route to Any Provider

Set PORTKEY_API_KEY and configure providers with the portkey: prefix: 
providers:
  id: portkey:gpt-4o
  config:
    portkeyProvider: openai

​
Cloud Providers (Azure, Bedrock, Vertex)

providers:
  id: portkey:xxx
  config:
    portkeyVirtualKey: "@azure-openai-prod"

​
3. Track Costs & Metrics

Add metadata to segment requests and track costs per team/project: 
providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyVirtualKey: "@anthropic-prod"
    portkeyMetadata:
      team: alpha9
      prompt: classification
    portkeyTraceId: run_1
Filter and group by these keys in Portkey dashboards.

​
4. Avoid Rate Limits with Caching

Create a config with load balancing and caching for high-volume evals: 
{
  "cache": { "mode": "simple" },
  "strategy": { "mode": "loadbalance" },
  "targets": [
    { "provider": "@openai-key-1" },
    { "provider": "@openai-key-2" },
    { "provider": "@openai-key-3" }
  ]
}
Reference the config in your YAML: 
providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyConfig: pc-your-config-id

​
Roadmap

🚧 Coming soon: View Promptfoo eval results in Portkey’s feedback section.

​
Next Steps

Model Catalog

Set up providers

Prompt Management

Version and manage prompts