Setting Up Local Language Models for Your App
Setting up local language models for your app can significantly enhance its capabilities, enabling it to understand and generate text in multiple languages without relying on external APIs. By integrating local language models, you can improve privacy, reduce latency, and ensure continuous functionality even in offline environments. Here’s a comprehensive guide on how to set up local language models for your application:
Steps:
For cloud version LLM change:
Step 1
Visit the chat screen and you will be to see the default LLM selected.
Step 2
Click on it and you will get a drop down of various LLM’s available to choose.
Step 3
Choose the LLM of your choice.
Video Demo
For Open source llm change:
Step 1
For open source version please edit LLM_PROVIDER, LLM_NAME and others in the .env file. Refer to ⚙️ App Configuration for more information.
Step 2
Visit ☁️ Cloud Providers for the updated list of online models. Make sure you have the right API_KEY and correct LLM_PROVIDER. For self-hosted please visit 🖥️ Local Inference.
Fallback LLM
DocsGPT can automatically switch to a fallback LLM when the primary model fails, including mid-stream. This works with both streaming and non-streaming requests.
Fallback order:
- Per-agent backup models (other models configured on the same agent)
- Global fallback (
FALLBACK_LLM_*env vars below) - Error returned if all fail
| Setting | Description | Default |
|---|---|---|
FALLBACK_LLM_PROVIDER | Provider name (e.g., openai, anthropic, google) | — |
FALLBACK_LLM_NAME | Model name (e.g., gpt-4o, claude-sonnet-4-20250514) | — |
FALLBACK_LLM_API_KEY | API key for the fallback provider | Falls back to API_KEY |
All three (FALLBACK_LLM_PROVIDER, FALLBACK_LLM_NAME, and an API key) must resolve for the global fallback to activate.
FALLBACK_LLM_PROVIDER=anthropic
FALLBACK_LLM_NAME=claude-sonnet-4-20250514
FALLBACK_LLM_API_KEY=sk-ant-your-anthropic-keyFor maximum resilience, use a fallback provider from a different cloud than your primary. Each agent can also have multiple models configured — the other models are tried first before the global fallback.