Understanding and Configuring Embedding Models in DocsGPT

Embedding models are a crucial component of DocsGPT, enabling its powerful document understanding and question-answering capabilities. This guide will explain what embedding models are, why they are essential for DocsGPT, and how to configure them.

What are Embedding Models?

In simple terms, an embedding model is a type of language model that converts text into numerical vectors. These vectors, known as embeddings, capture the semantic meaning of the text. Think of it as translating words and sentences into a language that computers can understand mathematically, where similar meanings are represented by vectors that are close to each other in vector space.

Why are embedding models important for DocsGPT?

DocsGPT uses embedding models for several key tasks:

• Semantic Search: When you upload documents to DocsGPT, the application uses an embedding model to generate embeddings for each document chunk. These embeddings are stored in a vector store. When you ask a question, your query is also converted into an embedding. DocsGPT then performs a semantic search in the vector store, finding document chunks whose embeddings are most similar to your query embedding. This allows DocsGPT to retrieve relevant information based on the meaning of your question and documents, not just keyword matching.
• Document Understanding: Embeddings help DocsGPT understand the underlying meaning of your documents, enabling it to answer questions accurately and contextually, even if the exact keywords from your question are not present in the retrieved document chunks.

In essence, embedding models are the bridge that allows DocsGPT to understand the nuances of human language and connect your questions to the relevant information within your documents.

Out-of-the-Box Embedding Model Support in DocsGPT

DocsGPT is designed to be flexible and supports a wide range of embedding models right out of the box:

• Local models (FastEmbed / ONNX Runtime): DocsGPT runs local embeddings through FastEmbed . Any of FastEmbed’s built-in models works, as does any Hugging Face repository that ships an ONNX export at onnx/model.onnx — which covers most popular sentence-transformers repos. A repository with PyTorch weights only will not load; serve those over EMBEDDINGS_BASE_URL instead. New installs default to ibm-granite/granite-embedding-311m-multilingual-r2; existing ones stay on huggingface_sentence-transformers/all-mpnet-base-v2 until re-embedded.
• OpenAI Embeddings: text-embedding-ada-002 is built in (EMBEDDINGS_NAME=openai_text-embedding-ada-002). Other OpenAI models such as text-embedding-3-small and text-embedding-3-large work through the remote-embeddings mode; see Using OpenAI Embeddings.
• Azure OpenAI Embeddings: Azure OpenAI is supported for embeddings only, not for chat. See Azure OpenAI Embeddings.
• Remote OpenAI-compatible Embeddings: Any server that exposes an OpenAI-compatible /v1/embeddings endpoint (for example llama.cpp, vLLM, TEI, or a hosted provider) by setting EMBEDDINGS_BASE_URL. See Remote Embeddings below.

Configuring a Local Model

Set EMBEDDINGS_NAME in your .env to a registry name or a Hugging Face repository id:

EMBEDDINGS_NAME=ibm-granite/granite-embedding-311m-multilingual-r2

The model is downloaded on first use and cached, together with its tokenizer, under models/ in the data home; set EMBEDDINGS_CACHE_DIR to use another directory. There is no folder to populate by hand, and a filesystem path is not accepted as a model name.

DocsGPT knows the pooling, vector width and context window of the models in its registry (all-mpnet-base-v2, granite-embedding-311m-multilingual-r2, granite-embedding-97m-multilingual-r2). For any other repository it reads those from the repository’s own 1_Pooling/config.json and modules.json. If a repository declares neither, mean pooling with L2 normalization is assumed and a warning is logged — pin the real values with EMBEDDINGS_POOLING (cls or mean) and EMBEDDINGS_NORMALIZE.

Models with a Dense projection layer (for example sentence-transformers/LaBSE) are refused at startup: FastEmbed cannot apply the projection, so the vectors would be the wrong width and in a different space.

For an offline or air-gapped install, pre-fetch the model at build or setup time:

python -m docsgpt.scripts.prefetch_models

Using OpenAI Embeddings

To use OpenAI’s text-embedding-ada-002 embedding model, set EMBEDDINGS_NAME to openai_text-embedding-ada-002 and give DocsGPT an OpenAI key. The embeddings client uses the first of these that is set:

1. EMBEDDINGS_KEY
2. OPENAI_API_KEY
3. API_KEY, but only when LLM_PROVIDER=openai, because only then is it an OpenAI key

Example .env configuration for OpenAI Embeddings:

LLM_PROVIDER=openai
API_KEY=YOUR_OPENAI_API_KEY # Used for both chat and embeddings
EMBEDDINGS_NAME=openai_text-embedding-ada-002

If your chat model comes from another provider, set the OpenAI key separately:

LLM_PROVIDER=anthropic
API_KEY=YOUR_ANTHROPIC_API_KEY
EMBEDDINGS_NAME=openai_text-embedding-ada-002
EMBEDDINGS_KEY=YOUR_OPENAI_API_KEY

Only text-embedding-ada-002 is built in. Any other name that DocsGPT does not know, such as text-embedding-3-small, is treated as a Hugging Face repository and fails to load. Use OpenAI’s newer models through the remote-embeddings mode instead:

EMBEDDINGS_BASE_URL=https://api.openai.com
EMBEDDINGS_NAME=text-embedding-3-small
EMBEDDINGS_KEY=YOUR_OPENAI_API_KEY

DocsGPT learns the vector width from the first response. Changing models on an existing index still needs re-embedding; see Embedding Dimensions.

Azure OpenAI Embeddings

DocsGPT switches the OpenAI embeddings client to Azure when OPENAI_API_BASE, OPENAI_API_VERSION and AZURE_DEPLOYMENT_NAME are all set. EMBEDDINGS_NAME must still name the built-in ada-002 entry, and the Azure deployment that serves it goes in AZURE_EMBEDDINGS_DEPLOYMENT_NAME:

EMBEDDINGS_NAME=openai_text-embedding-ada-002
EMBEDDINGS_KEY=YOUR_AZURE_OPENAI_KEY
OPENAI_API_BASE=https://YOUR_RESOURCE.openai.azure.com/
OPENAI_API_VERSION=YOUR_API_VERSION # an API version your Azure resource supports
AZURE_DEPLOYMENT_NAME=YOUR_DEPLOYMENT
AZURE_EMBEDDINGS_DEPLOYMENT_NAME=YOUR_EMBEDDINGS_DEPLOYMENT

The key is looked up in the same order as for OpenAI. There is no Azure OpenAI chat provider; AZURE_DEPLOYMENT_NAME is only part of the check that turns Azure on.

Remote (OpenAI-compatible) Embeddings

If you run your own embedding server, or use a provider that exposes an OpenAI-style embeddings API, point DocsGPT at it with EMBEDDINGS_BASE_URL. When this is set, all embedding calls (ingestion and querying) are sent to {EMBEDDINGS_BASE_URL}/v1/embeddings in OpenAI format instead of running a local model.

EMBEDDINGS_BASE_URL=http://localhost:8080   # your OpenAI-compatible embeddings server
EMBEDDINGS_NAME=your-model-name             # sent as the "model" field in the request
EMBEDDINGS_KEY=YOUR_API_KEY                 # optional; sent as a Bearer token

• EMBEDDINGS_BASE_URL — base URL of the remote server. Setting it switches DocsGPT into remote-embeddings mode.
• EMBEDDINGS_NAME — forwarded as the model field in each request.
• EMBEDDINGS_KEY — optional bearer token. In this mode it is the only key sent: OPENAI_API_KEY and API_KEY are not used, so set it even if it is the same OpenAI key.

If the server answers 401 or 403, DocsGPT logs an embeddings_auth_failed error naming a fingerprint of the key (the first 16 hex characters of its SHA-256), never the key itself. When the query embedding fails because the embedder itself failed (a 5xx, a timeout or a connection error), the sources are not searched, since each search would send the same query to the same failing embedder again. After any other embedding failure each source embeds the query itself. When the answer then goes ahead without retrieved context, that is logged once as a retrieval_degraded error with the status code, the number of sources, and the activity and user ids, and the trace’s retrieval step is marked docsgpt.retrieval_degraded.

Guarding against oversized inputs

Some remote servers (notably llama.cpp) reject any single input larger than their physical batch size with a 500 error. Set EMBEDDINGS_MAX_INPUT_TOKENS to clip each input to a fixed number of tokens before it is sent:

EMBEDDINGS_MAX_INPUT_TOKENS=512

When set, each input string is truncated to that many tokens and the overflow is dropped (lossy by design).

You usually do not need to set it. When EMBEDDINGS_NAME names a model DocsGPT knows, its context window is used automatically, and counting uses that model’s own tokenizer, so the limit and the count are in the same unit. Set EMBEDDINGS_MAX_INPUT_TOKENS when your server serves a model DocsGPT does not know, or when you deliberately want a limit below the model’s own.

Leaving EMBEDDINGS_NAME unset imposes no limit: the name is only forwarded as the model field in each request, so a default nobody chose is not taken as a description of your server. When no model tokenizer is available, counting falls back to tiktoken — pick a limit with headroom below the server’s true limit to absorb the skew between the two tokenizers.

Search query length

Each search query is clipped to EMBEDDINGS_MAX_QUERY_TOKENS tokens (default 512) before it is embedded, whichever embedder you use. A query needs a few hundred tokens, while the memory an embedder uses grows with the square of its input length: on a long-context model such as granite, one query of about 9,000 tokens needs more than 12 GB. Long queries come from pasted code, logs or JSON in a first chat message, or from a webhook payload. When a query is clipped, DocsGPT logs Clipping a … search query.

Documents are not affected. To bound them too, use EMBEDDINGS_MAX_INPUT_TOKENS for a remote server or EMBEDDINGS_LOCAL_MAX_TOKENS for a local model. Each drops whatever is past its limit. A local model otherwise accepts up to its own maximum, which is 32,768 tokens for granite; EMBEDDINGS_LOCAL_MAX_TOKENS=4096 keeps one input at about 3 GB. Set EMBEDDINGS_MAX_QUERY_TOKENS=0 to turn query clipping off.

Where the model runs

A local embedding model costs a few hundred megabytes of resident memory per process, and the API embeds every query it serves — so by default it would hold its own copy alongside the worker’s.

EMBEDDINGS_DELEGATE_TO_WORKER (on by default) moves that work to the Celery worker: the API sends the text over the broker and gets the vector back, holding no model. Measured on a default install, the API process drops from ~660 MB to ~285 MB, and query embedding costs one broker round trip (~60 ms on a prefork worker).

EMBEDDINGS_THREADS sets the intra-op threads of the local ONNX runner; unset, it uses every core. It scales sub-linearly, so several single-threaded workers do more than one many-threaded process on the same cores.

Retrieval then depends on a worker consuming EMBEDDINGS_QUEUE (embeddings by default). A bare celery worker with no -Q consumes it along with everything else. A worker started with an explicit -Q must list it — the bundled Compose and Kubernetes manifests run -Q docsgpt,parsing,embeddings for exactly this reason. Omit it and every search blocks for EMBEDDINGS_DELEGATE_TIMEOUT and then returns an answer with no retrieved context, without raising.

Sharing one worker also shares its concurrency with ingest, so a query can queue behind a long parse. Run a dedicated worker to isolate query latency:

celery -A docsgpt.app.celery worker -Q embeddings

Set EMBEDDINGS_DELEGATE_TO_WORKER=false if you run the API without a worker; it will load the model in-process instead.

For production, prefer EMBEDDINGS_BASE_URL. A real embedding service removes the model from both the API and the worker, and replaces the broker round trip with a network call.

Batch sizes

Two separate knobs, easily confused:

• EMBEDDINGS_BATCH_SIZE (default 32) — chunks per store transaction, and per request to a remote embeddings API. Larger means fewer round trips and fewer transactions.
• EMBEDDINGS_MODEL_BATCH_SIZE (default 1) — documents per forward pass of a local model.

For the local model, bigger batches are not faster. ONNX needs a rectangular tensor, so every input in a pass is padded up to the longest one in it, and that waste grows with the square of chunk length. Measured on a 30-document ingest at the 1250-token default chunk size:

Raise it only if your chunks are short and uniform in length.

Important: Embedding Dimensions Must Stay Consistent

Each embedding model produces vectors of a fixed dimension, and your vector store is created with that dimension. Changing EMBEDDINGS_NAME to a model with a different dimension is not compatible with an existing index — FAISS will raise a dimension-mismatch error, and pgvector/Qdrant tables are sized to the original dimension.

If you need to switch embedding models, you must re-ingest your sources so the index is rebuilt with the new dimension. This also applies to the GraphRAG graph tables, which are sized to the embedding dimension at creation time.

A matching dimension is not a matching model

The dimension check is a guard against a corrupt index, not a guarantee that a swap is safe. Two models of the same width — all-mpnet-base-v2 and granite-embedding-311m-multilingual-r2 are both 768 — raise nothing at all, and every query is then embedded by a different model than the stored vectors were. Nothing fails; retrieval quality simply degrades.

Switching between same-width models therefore still requires re-embedding:

python -m docsgpt.scripts.reembed

Run it after changing EMBEDDINGS_NAME and before serving queries. See Upgrading for the granite migration specifically.

Changing to a model of a different width is supported for FAISS: the script rebuilds the index at the new width and keeps the existing chunk ids. For pgvector the vector column is sized at creation time, so a width change there still means re-ingesting those sources.

With GRAPHRAG_ENABLED, the script also rewrites graph_nodes.name_embedding on pgvector. Those vectors seed every graph traversal, and they share the chunk vectors’ width, so leaving them behind degrades graph retrieval just as silently.

Adding Support for Other Embedding Models

To teach DocsGPT about a new model — so it carries a known pooling, width and context window rather than being inferred — add an EmbeddingModel entry to MODELS in docsgpt/vectorstore/model_registry.py. That registry is the single source of truth the local runner, the remote client, the schema bootstrap and the chunker all read, and for a model that FastEmbed can run, or one served over EMBEDDINGS_BASE_URL, the entry is usually all you need.

To plug in a different embedding library, look at these classes:

• EmbeddingsWrapper in docsgpt/vectorstore/embeddings_local.py wraps the local runner in the embed_query / embed_documents interface the vector stores use.
• EmbeddingsSingleton in docsgpt/vectorstore/base.py builds and caches one embeddings instance per model, and get_embeddings in the same file decides between the remote, OpenAI, delegated and local paths.