Connecting DocsGPT to Cloud LLM Providers

DocsGPT is designed to seamlessly integrate with a variety of Cloud Large Language Model (LLM) providers, giving you access to state-of-the-art AI models for document question answering.

Configuration via .env file

The primary method for configuring your LLM provider in DocsGPT is through the .env file. For a comprehensive understanding of all available settings, please refer to the detailed DocsGPT Settings Guide.

To connect to a cloud LLM provider, you will typically need to configure the following basic settings in your .env file:

• LLM_PROVIDER: The provider whose models answer by default (for example openai, google, anthropic).
• API_KEY: Your API key for that provider. You can use the provider’s own variable instead, such as OPENAI_API_KEY (see Provider API keys).
• LLM_NAME (optional): The default model. It must be a model id from DocsGPT’s model catalog , such as gpt-5.5, gemini-3.5-flash or claude-sonnet-4-6, or from your own custom model YAML. A name that is not in the catalog is ignored, and DocsGPT logs a warning and uses the provider’s first model. Leave it unset to use that first model.

For example:

LLM_PROVIDER=openai
API_KEY=YOUR_OPENAI_API_KEY
LLM_NAME=gpt-5.5

Explicitly Supported Cloud Providers

DocsGPT offers direct, streamlined support for the following cloud LLM providers. The table below outlines the LLM_PROVIDER value and an example LLM_NAME (a catalog id) for each provider.

The Google provider uses a Gemini API key from Google AI Studio. Vertex AI credentials (a project, a location and a service account) are not supported.

DocsGPT also ships a model catalog (docsgpt/core/models/*.yaml) that the in-app model picker reads. It includes DeepSeek, Alibaba Qwen and Z.ai GLM models, which appear once their key is set; see the table below.

Provider API keys

Each provider’s models are registered when its key is set. API_KEY counts as the key of the provider named in LLM_PROVIDER; the other variables can be set together to offer several providers in the model picker.

LLM_PROVIDER still decides the default. With LLM_PROVIDER=openai, OPENAI_API_KEY and ANTHROPIC_API_KEY both set, both providers appear in the picker and gpt-5.5 is the default. To make a DeepSeek, Qwen or GLM model the default, set LLM_NAME to its id (for example LLM_NAME=deepseek-v4-flash).

To add a provider or model that is not in the catalog, write a model YAML and load it with MODELS_CONFIG_DIR.

Keys stay with their provider

Each key is sent only to the endpoint it was configured for:

• a provider’s own variable (OPENAI_API_KEY, ANTHROPIC_API_KEY …) to that provider;
• API_KEY to the provider in LLM_PROVIDER, or to your own server at OPENAI_BASE_URL;
• a catalog key such as DEEPSEEK_API_KEY to the base_url in its YAML;
• FALLBACK_LLM_API_KEY to the fallback provider.

The model decides where a request goes: it carries its provider, its key and its endpoint together. When DocsGPT cannot pair a request with an endpoint of its own, it refuses the request instead of sending it to OpenAI with whatever key is set:

A key configured for an https endpoint is never sent to the same host over http, and plain http is accepted only for a server on your own network; see Plain http and your API key.

The provider SDKs’ own endpoint variables (ANTHROPIC_BASE_URL, GOOGLE_GEMINI_BASE_URL, GOOGLE_GENAI_USE_VERTEXAI) are ignored, because they would send a key to a host DocsGPT never checked. To reach a provider through a proxy or another endpoint, give the model a base_url in a model YAML.

A model name that isn’t in the catalog still works for a provider with a fixed endpoint, such as FALLBACK_LLM_NAME=gpt-4o-mini with FALLBACK_LLM_PROVIDER=openai: it goes to that provider with that provider’s key.

Connecting to OpenAI-Compatible Cloud APIs

DocsGPT can also connect to any cloud provider that offers an OpenAI-compatible API through OPENAI_BASE_URL. Set LLM_PROVIDER=openai, the provider’s endpoint as OPENAI_BASE_URL, its key as API_KEY, and the model name as LLM_NAME. LLM_NAME is required here, because DocsGPT registers exactly the model(s) it names; separate several with commas.

Setting OPENAI_BASE_URL hides the DocsGPT and OpenAI catalogs, so only the models in LLM_NAME are offered, plus any whose provider-specific key is set.

Example for DeepSeek (OpenAI-Compatible API):

LLM_PROVIDER=openai
API_KEY=YOUR_API_KEY # Your DeepSeek API key
LLM_NAME=deepseek-v4-flash # The model name DeepSeek's API expects
OPENAI_BASE_URL=https://api.deepseek.com/v1 # DeepSeek's OpenAI API URL

DeepSeek is also in the catalog, so setting DEEPSEEK_API_KEY is usually simpler: it keeps the rest of the catalog available.

Remember to consult the documentation of your chosen OpenAI-compatible cloud provider for their specific API endpoint, required model names, and authentication methods.

The openai_compatible provider

openai_compatible is not a value for LLM_PROVIDER on its own: setting it registers nothing unless a model YAML with its key is loaded. It is the provider behind models that carry their own base_url and API key:

• catalog entries such as DeepSeek, Alibaba and Z.ai, which name their key variable (api_key_env) and endpoint in their YAML;
• your own model YAMLs in MODELS_CONFIG_DIR;
• per-user custom models that users add in Settings → Custom Models (bring your own model). Outbound requests for these use an SSRF-pinned HTTP client, so their base URL must be a public address: a server on localhost or your LAN (such as a local Ollama) is refused. Configure those as the operator, with OPENAI_BASE_URL or a model YAML, which is not checked. See Outbound network access.

OpenAI Responses API and reasoning

For OpenAI models that support it, DocsGPT can call the newer Responses API (/v1/responses) instead of Chat Completions. This is selected per model in the catalog via an api_flavor: responses capability and enables features like server-side reasoning. Related settings:

• reasoning_effort — per-model reasoning effort hint (for example medium) declared in the model catalog.
• OPENAI_RESPONSES_STORE (default false) — when true, lets OpenAI persist Responses API state server-side and chains calls with previous_response_id. When false, DocsGPT requests encrypted reasoning items and persists those ciphertext items itself so reasoning survives tool calls and later conversation turns without OpenAI retaining the response.
• OPENAI_RESPONSES_CHAIN_ACROSS_TURNS (default true) and OPENAI_RESPONSES_CHAIN_BUDGET_TOKENS (default: the model’s context_window) — in store mode a new user turn chains onto the previous response, which keeps the provider’s stored transcript and its prompt cache warm. The chain is bounded: once the previous turn’s reported prompt reaches the budget, or after the conversation history has been compressed, the next turn starts from DocsGPT’s own saved history instead. Chained tool rounds do not re-send an unchanged system message. Every turn of a conversation whose call goes out unchained logs one responses_chain_reset line at INFO with reason, conversation_id, activity_id and model; the same reason is on that call’s chat span as docsgpt.chain_reset_reason. The reasons are first_turn, disabled (store mode or cross-turn chaining off), no_previous_response (the last turn stored no response), fallback_answered (a fallback model answered the last turn), chain_key_mismatch (another model, endpoint or key), native_parts_cap, compression (before the turn or during it), chain_budget, history_mismatch (the history does not answer every tool call of the response it would chain onto), not_found (the provider no longer has that response) and previous_call_failed.
• prompt_cache_breakpoints (model capability, default false): set it to true in a model’s YAML for GPT-5.6 and later models, which cache only at prompt-cache breakpoints. DocsGPT then marks a breakpoint on each user message and puts a turn’s attachment block after the question, so a turn that resends its history unchained (after a compression, a restart or a chain past its budget) reads earlier turns back from the provider’s cache instead of writing the whole prompt again. Earlier models reject the field with a 400, so leave it off for them. Resent history replays each turn’s tool calls round by round, as they ran, and the context checks count the encrypted reasoning every replayed call carries (the provider bills it in full as input). A turn that answers a finished background job stores its response id too, so the next user turn can chain onto it.
• OPENAI_RESPONSES_TRUNCATION_AUTO (default false) — send truncation: "auto" so the provider drops the oldest conversation items instead of failing a request that exceeds the model’s window.
• OPENAI_PROMPT_CACHE_KEY (default true) and OPENAI_PROMPT_CACHE_RETENTION (default unset) — prompt-cache hints on Responses API calls: an opaque per-user prompt_cache_key (a hash, never the user id itself), and a prompt_cache_retention value such as 24h where the provider supports it.
• V1_SESSION_TTL_SECONDS (default 86400) — lifetime of the hashed Redis mapping used to associate OpenAI-compatible client session headers with hidden DocsGPT conversations. Explicit DocsGPT conversation IDs always take precedence.

See App Configuration for the full settings reference.

Adding Support for Other Cloud Providers

If you wish to connect to a cloud provider that is not explicitly listed above or doesn’t offer OpenAI API compatibility, you can extend DocsGPT to support it. Write an LLM class in docsgpt/llm/ (the existing modules are examples) and a provider plugin in docsgpt/llm/providers/ that points at it and says how to find its API key, then add the plugin to ALL_PROVIDERS in docsgpt/llm/providers/__init__.py and list its models in a YAML under docsgpt/core/models/. See Adding Support for Other Local Engines for the two parts.