Observability

DocsGPT bundles the OpenTelemetry SDK and auto-instrumentation packages as core dependencies (pyproject.toml), so they install with the rest of the backend and ship in the image. OpenTelemetry export is off by default; opt in by prefixing the launch command with opentelemetry-instrument and setting OTLP env vars.

Two other things are on by default. Execution traces are stored locally in Postgres and leave the instance only through an exporter you configure. The version check is outbound: when the worker starts and every 7 hours it sends its version, a random instance_id kept in the database, the Python version, the platform (sys.platform) and a client name to https://gptcloud.arc53.com/api/check, reusing a recent answer cached in Redis instead where it has one. Security advisories in the answer appear in the worker log, and high or critical ones also print a banner to the worker’s console. Turn it off with VERSION_CHECK=0.

Auto-instrumentation covers Flask, Starlette, Celery, SQLAlchemy, psycopg, Redis, requests, and Python logging. Agent runs, LLM calls, tool calls and retrieval are recorded by DocsGPT itself and exported as OpenTelemetry GenAI spans — see Execution traces.

Enabling

Set these env vars in your .env (or compose environment: block):

OTEL_SDK_DISABLED=false
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector.example.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<token>
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_PYTHON_LOG_CORRELATION=true
OTEL_RESOURCE_ATTRIBUTES=service.name=docsgpt-backend,deployment.environment=prod

Then prefix the process command with opentelemetry-instrument. The simplest way is a Compose override file, with no image rebuild. Save this as docker-compose.otel.yaml next to your Compose file:

# docker-compose.otel.yaml
services:
  backend:
    # The image's CMD from docsgpt/Dockerfile behind opentelemetry-instrument.
    # Keep the rest in sync with the Dockerfile when you upgrade.
    command:
      - opentelemetry-instrument
      - gunicorn
      - -w
      - "1"
      - -k
      - docsgpt.gunicorn_worker.BoundedDrainUvicornWorker
      - --bind
      - 0.0.0.0:7091
      - --timeout
      - "180"
      - --graceful-timeout
      - "120"
      - --keep-alive
      - "5"
      - --worker-tmp-dir
      - /dev/shm
      - --max-requests
      - "5000"
      - --max-requests-jitter
      - "500"
      - --config
      - docsgpt/gunicorn_conf.py
      - docsgpt.asgi:asgi_app
    environment:
      - OTEL_SERVICE_NAME=docsgpt-backend
  worker:
    # The bundled worker command behind opentelemetry-instrument.
    command: opentelemetry-instrument celery -A docsgpt.app.celery worker -l INFO -B -Q docsgpt,parsing,embeddings,ingest,analytics
    environment:
      - OTEL_SERVICE_NAME=docsgpt-celery-worker

Compose loads an override file by itself only when no -f is given, and the commands in these docs all pass -f, so name it after the main file on every command, up included. For the checkout Compose files, with the override saved in deployment/:

docker compose --env-file .env -f deployment/docker-compose-hub.yaml -f deployment/docker-compose.otel.yaml up -d

For the standalone file, run docker compose -f docker-compose-standalone.yaml -f docker-compose.otel.yaml up -d in its folder. A docsgpt up stack doesn’t know about the override: docsgpt up, docsgpt restart and docsgpt upgrade start it without tracing, so start it with docker compose -f ~/.docsgpt/server/docker-compose.yaml -f ~/.docsgpt/server/docker-compose.otel.yaml up -d afterwards.

For local dev, prepend dotenv run -- so the OTEL_* vars from .env reach opentelemetry-instrument before it boots the SDK:

dotenv run -- opentelemetry-instrument uvicorn docsgpt.asgi:asgi_app --port 7091
dotenv run -- opentelemetry-instrument celery -A docsgpt.app.celery worker -l INFO -B --pool=solo -X analytics

-X analytics keeps the conversation-analytics sweep off the solo worker, where it would hold the only slot and searches would time out waiting for their query embedding. With DECISION_ANALYTICS_ENABLED on, run dotenv run -- opentelemetry-instrument docsgpt worker --analytics next to it (see the analytics queue).

Trace the ASGI app rather than flask run, which serves only the Flask app: the ASGI-only routes return 404 there and their spans never appear. -B keeps the beat scheduler running, as in production.

Execution traces

Every request records an execution trace: a timed tree of the steps behind it. Traces are recorded for chat turns (/stream, /api/answer, /v1/chat/completions, including each round of a tool-approval pause), scheduled and webhook runs, workflows, the research agent, /api/search, the MCP search_docs tool, and graph builds.

Traces are stored in the request_traces table and shown in the app: open Settings → Logs (or an agent’s Logs tab), expand an entry and choose View trace to see a waterfall of every step with its timing, tokens, cost and details. See Analytics and Logs for a user’s guide to those pages.

Stored traces keep short previews — tool arguments and results, retrieved chunk titles and snippets (with each chunk’s relevance where the decision model scored it), rephrased queries, answer excerpts — truncated and with secret-named fields redacted. Full prompts are never stored. When a guardrail fires during a request, every preview is dropped from its trace.

TRACES_ENABLED=true            # record traces at all
TRACES_CAPTURE_CONTENT=true    # keep previews in stored traces
TRACES_PREVIEW_CHARS=2000      # characters kept per preview
TRACES_MAX_SPANS=500           # steps kept per trace; the rest are counted
TRACES_RETENTION_DAYS=30       # a daily task deletes older traces
TRACES_OTEL_EXPORT=true        # also export traces as OTel GenAI spans

GenAI spans and metrics

When DocsGPT runs under opentelemetry-instrument, each finished trace is also exported as spans that follow the OpenTelemetry GenAI semantic conventions : invoke_agent {agent}, chat {model}, execute_tool {tool}, embeddings {model} and retrieval, with attributes such as gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.conversation.id, gen_ai.agent.id and gen_ai.tool.name. DocsGPT-specific details use the docsgpt.* prefix (docsgpt.request_id, docsgpt.token_source, docsgpt.cache_hit, docsgpt.ttft_ms, and on a Responses API call that did not chain onto the previous response, docsgpt.chain_reset_reason, …). The trace’s root span is a child of the request’s HTTP server span, and the stored trace keeps the OTel trace id so you can move between the two.

Two metrics are recorded for every model call: gen_ai.client.token.usage and gen_ai.client.operation.duration.

A call to the decision model is exported as a decision {feature} span: a client span in API mode, an internal one when the model runs in the process. It records the same two metrics with operation decide and provider arc-decide, plus a docsgpt.decision.calls counter by feature and status.

Relevance in retrieval

When the decision model scores retrieved chunks (a source’s relevance), each scoring call is a rerank relevance step under the retrieval step, with the decision step inside it. It carries gen_ai.operation.name=rerank, docsgpt.rerank.backend=decision, docsgpt.rerank.feature (rag_rerank in answers and agents’ searches, retrieval_test in a retrieval test), docsgpt.rerank.arm, docsgpt.candidate_count, docsgpt.kept_count, docsgpt.relevance_threshold, docsgpt.top_relevance, docsgpt.query_source (rephrased when it judged the standalone rephrase, else raw), docsgpt.relevance.passage_select (best where the model runs on a CPU and scores each chunk’s best window, else full), docsgpt.relevance.passage_tokens, and docsgpt.decision.fallback, the reason when it kept the retrieval’s own selection (timeout, loading, circuit_open, …). The retrieval step carries docsgpt.rerank.arm for the request’s rollout arm.

The same call adds an entry to the answer’s decision record, numbers and labels only:

{"feature": "rag_rerank", "round": 0, "status": "ok", "applied": true, "ms": 412.0,
 "outcome": {"candidates": 20, "kept": 6, "replaced": 3, "top": 0.81, "threshold": 0.0,
             "query_source": "rephrased", "arm": "on", "passage_select": "full", "passage_tokens": null}}

replaced counts the kept chunks that retrieval would not have returned without the model, top is the highest relevance scored, threshold is the strictest min_relevance of the sources scored together (0 reorders only), and applied is false after a failure. A request in the rollout’s control arm (DECISION_RELEVANCE_ROLLOUT_PERCENT below 100) with a source whose relevance is on asks the model nothing and records this entry, so the two arms can be counted:

{"feature": "rag_rerank", "status": "skipped", "outcome": {"arm": "control"}}

Per-day call counts are in decision_usage under the same feature labels.

Conversation analytics

The analytics sweep runs outside any request, so it leaves no execution trace. Its calls are counted in decision_usage under two feature labels: analytics_turn (answers, the replies to them and each user message) and analytics_conversation (goal met and the agent’s topics that can overlap). Each sweep logs one analytics_sweep line from the worker with its counts (messages, conversations, calls, states, rows, errors, timeouts, truncated), ms, and stopped: budget when its time ran out, unavailable (or failed) when the model could not be reached or answered nothing, timeout when chat kept the model busy, lock_lost when its lock ran out and another sweep took over, or null when it finished. Sweeps that keep stopping on timeout mean labels lag behind the traffic.

Prompt-injection scanning

The injection_model guardrail asks the decision model under the stage’s guardrail step, whose decision steps carry the feature label of the stage. At retrieval and attachment, where it scores each document or file on its own, the guardrail step also carries docsgpt.guardrail.items (documents or files checked), docsgpt.guardrail.withheld, docsgpt.guardrail.cache_hits (scores read from the cache or stored with a file), docsgpt.guardrail.live (scored during the turn) and docsgpt.guardrail.lazy (queued for background scoring), next to docsgpt.guardrail.stage and docsgpt.guardrail.checks.

Background scoring runs outside any request and leaves no trace; it is counted in decision_usage like every call. Texts answered from the score cache are counted there too, under status cached, so the texts checked per feature are the model-scored states plus the cached ones. Each chat-time evaluation also adds an entry to the answer’s decision record. A flagged or withheld document or file writes a guardrail event whose item_ref names it: chunk:<source_id>:<chunk_key> or attachment:<id>.

Backends that understand the GenAI conventions — Langfuse (/api/public/otel), Arize Phoenix, Datadog LLM Observability, Grafana — render these as LLM traces with token and cost views.

Prompt and tool content is not exported by default, because the OTLP backend may be a third party. Opt in with the standard variable:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY

This adds the same redacted previews the app stores (for example gen_ai.tool.call.arguments and gen_ai.tool.call.result).

The GenAI conventions are still in development upstream, so attribute names may change in later releases.

Backend examples

Axiom

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.axiom.co
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20xaat-XXXX,X-Axiom-Dataset=docsgpt
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

%20 is the URL-encoded space between Bearer and the token. Create the dataset in the Axiom UI before sending.

Self-hosted OTLP collector / Jaeger / Tempo

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Honeycomb / Grafana Cloud / Datadog

Each vendor publishes a single-line OTEL_EXPORTER_OTLP_ENDPOINT plus OTEL_EXPORTER_OTLP_HEADERS recipe — drop them in alongside the service-name override.

Caveats

• The Dockerfile uses gunicorn -w 1. If you raise worker count, move SDK init into a post_worker_init hook to avoid one-thread-per-process exporter contention.
• asgi.py mounts the Flask app inside a Starlette app through a2wsgi’s WSGIMiddleware. Both instrumentors are installed, so each request produces a Starlette span enclosing a Flask span. If the duplication is noisy, uninstall opentelemetry-instrumentation-flask in your image, or set OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=flask. Don’t edit docsgpt/requirements.txt: it is generated from uv.lock.
• OTEL packages add ~50 MB to the image. They install on every build — the runtime cost is zero unless you set opentelemetry-instrument on the command and set the OTLP env vars.