Fallback Models

DocsGPT can switch to a fallback model when the primary model fails, including after an answer has started streaming. This works for streaming and non-streaming requests.

Fallback order

The agent’s other models. An agent can have several models selected; when the one answering fails, the others are tried in order.
The global fallback set with the FALLBACK_LLM_* settings below.
If none of them can answer, the error is returned.

A fallback is skipped when the request is clearly too large for the fallback model’s context window, since it would fail the same way.

Global fallback settings

Setting	Description	Default
`FALLBACK_LLM_PROVIDER`	Provider name, as in `LLM_PROVIDER` (for example `openai`, `anthropic`, `google`)	unset (no global fallback)
`FALLBACK_LLM_NAME`	Model name sent to that provider. A catalog id works, and so does any other model name the provider serves.	unset
`FALLBACK_LLM_API_KEY`	API key for the fallback provider	`API_KEY`

Set all three. Without FALLBACK_LLM_NAME the fallback has no model to call.

When FALLBACK_LLM_API_KEY is unset, DocsGPT sends API_KEY, the key of your primary provider, to the fallback provider. Provider variables such as ANTHROPIC_API_KEY are not used for the global fallback. Whenever the fallback provider differs from LLM_PROVIDER, set FALLBACK_LLM_API_KEY: otherwise your primary key is sent to another vendor, and the fallback fails to authenticate.

If FALLBACK_LLM_NAME is a catalog model that carries its own key and endpoint, such as a DeepSeek model enabled with DEEPSEEK_API_KEY, that key and endpoint are used instead.

Example

OpenAI answers; Anthropic takes over when it fails:


LLM_PROVIDER=openai
API_KEY=sk-your-openai-key
LLM_NAME=gpt-5.5

FALLBACK_LLM_PROVIDER=anthropic
FALLBACK_LLM_NAME=claude-sonnet-4-6
FALLBACK_LLM_API_KEY=sk-ant-your-anthropic-key

For the most resilience, pick a fallback provider on a different cloud from your primary one.

See Cloud Providers for provider names and model ids, and the Settings Reference for every FALLBACK_LLM_* setting.