Document Parsing and OCR for Sources and Attachments

Parser engine

DocsGPT converts uploaded documents to Markdown before chunking, embedding or handing them to a model. The engine is selected with one setting:

DOC_PARSER_ENGINE=anydoc

• anydoc (default): firecrawl-anydoc , a Rust converter with no ML models. It reads PDF, DOCX, PPTX, XLSX and CSV in milliseconds with ~100 MB peak memory; HTML/XHTML is converted with markdownify, head-truncated at MARKUP_MAX_BYTES. Because anydoc is a core dependency, source uploads and the read_document tool now also accept its other formats — DOC, PPT/PPS/POT, XLS, ODT/ODS/ODP, RTF, XHTML and the macro-enabled Office variants — on every engine (they are in the SUPPORTED_SOURCE_EXTENSIONS whitelist; chat attachments are not whitelisted by extension and take them too). It never performs OCR: a scanned or image-only PDF is detected and handed to the fallback parser — the active OCR backend when OCR is on (see below), Docling when it is installed, the legacy text parsers otherwise — and if nothing can read the file the upload fails with a clear error instead of storing an empty document.
• docling: the previous default. Docling’s layout and table models produce structured Markdown, support OCR, and back the read_document tool’s structured output, at the cost of a large dependency tree (torch, transformers, ONNX models) and seconds to minutes per PDF. Switching back is this one variable; nothing else changes.

With DOC_PARSER_ENGINE=anydoc, Docling still handles what anydoc cannot when it is installed: the fallback for files anydoc rejects, .adoc/.vtt/.xml chat attachments (those suffixes are not in the source-upload whitelist), and — when it is the OCR backend — scanned PDFs and images. Without Docling those formats use the standard parsers; OCR still works through the native backend, and with OCR off images are only read when PARSE_IMAGE_REMOTE=true.

Installing tesseract

Nothing OCR-related ships in the base install: the default engine, tesseract, is a ~35 MB system package that you opt into like any other OCR dependency. Locally:

# Debian/Ubuntu                      # macOS
apt-get install tesseract-ocr tesseract-ocr-eng    brew install tesseract

Docker images build without it by default; opt in with the build argument:

docker build -f docsgpt/Dockerfile --build-arg INSTALL_TESSERACT=true .

deployment/docker-compose.yaml forwards the same switch, so setting INSTALL_TESSERACT=true in .env (or the shell) bakes tesseract plus the English pack into locally built backend and worker images; setup.sh writes it when you answer yes to the OCR question after choosing to build images locally.

With pre-built images the switch is the image variant: every tag is published twice, slim (arc53/docsgpt:<tag>) and -docling (arc53/docsgpt:<tag>-docling), and the latter bakes tesseract, the docling engine and its models in. Set DOCSGPT_IMAGE_VARIANT=-docling in .env for docker-compose-hub.yaml or docker-compose-standalone.yaml; setup.sh writes it when you answer yes to the OCR question with Docker Hub images. Alternatively point OCR_ENGINE=deepseek at a DeepSeek-OCR endpoint, which needs no system package. With OCR_ENABLED=true and no binary on PATH, scanned pages fail with an install hint (text-layer documents are unaffected).

Installing the docling engine

docling is not part of the base install, and OCR does not need it (see OCR backends). Add it when you want its layout-model OCR, .adoc/.vtt/.xml attachment parsing, or read_document’s structured output:

pip install -r docsgpt/requirements-docling.txt   # or: uv sync --extra docling

That file is the core set plus the docling extra, exported from the same lock. On Linux it takes torch from the CPU-only PyTorch index, so the extra costs about 1.5 GB rather than the 2.7 GB the CUDA build of torch would; a GPU deployment can reinstall torch from PyPI on top.

Pre-built images: use the -docling variant (arc53/docsgpt:<tag>-docling, DOCSGPT_IMAGE_VARIANT=-docling in .env), which also bakes docling’s layout, table-structure and RapidOCR models in so the first parse does not download them. Local builds opt in with the build argument:

docker build -f docsgpt/Dockerfile --build-arg EXTRAS=docling .

deployment/docker-compose.yaml forwards the same switch, so setting EXTRAS=docling (or the older INSTALL_DOCLING=true) in .env (or the shell) bakes docling into locally built backend and worker images; setup.sh offers it as a follow-up to the OCR question. Compose reads build arguments from the shell or from the .env you pass with --env-file .env (not from the containers’ env_file), so build with docker compose --env-file .env -f deployment/docker-compose.yaml build as setup.sh does. Of the pre-built Docker Hub images only the slim default excludes docling; the -docling variant ships it with its models. Either way no code changes are needed — docling is picked up as the fallback engine (and, under OCR_BACKEND=auto, as the OCR backend) as soon as it is importable, and DOC_PARSER_ENGINE=docling makes it the primary parser.

OCR

OCR is optional and controlled by two on/off settings, a backend and an engine choice:

OCR_ENABLED=false
OCR_ATTACHMENTS_ENABLED=false
OCR_BACKEND=auto
OCR_ENGINE=tesseract

• OCR_ENABLED: OCR behavior for Source Docs ingestion.
• OCR_ATTACHMENTS_ENABLED: OCR behavior for chat attachments uploaded from the message box.
• OCR_BACKEND: which stack performs the OCR (next section).
• OCR_ENGINE: which recognition engine it uses (Choosing the OCR engine).

The older names DOCLING_OCR_ENABLED and DOCLING_OCR_ATTACHMENTS_ENABLED are still accepted as aliases.

Under the default anydoc engine, a scanned PDF reaches OCR through anydoc’s own detection: anydoc refuses it (“OCR is required”) and the OCR backend takes over as the fallback parser. If that fallback also extracts almost nothing — OCR off, or no engine available — the upload fails with a clear message instead of silently indexing an empty document.

Mixed documents — text pages with scanned pages among them — convert through anydoc, which reads the text pages and skips the scanned ones. With OCR on, DocsGPT probes every page’s text layer, OCRs the pages that have none through the active backend, and appends their text; the document’s metadata records the count as ocr_pages. With OCR off, only the text pages are indexed.

OCR backends

Docling is not the only way to OCR. The native backend renders the pages that need it with pypdfium2 and Pillow — both already core dependencies — and feeds them straight into tesseract or a DeepSeek-OCR endpoint. No ML models load in the worker, and nothing beyond the ~35 MB tesseract binary (see Installing tesseract) is needed.

The trade-off is the layout model. Under native, multi-column scans rely on tesseract’s own page segmentation and tesseract yields tables as plain lines; DeepSeek-OCR emits Markdown tables itself, so it remains the quality path on either backend. Under DOC_PARSER_ENGINE=docling with the native backend, PDFs whose every page has a text layer still go through Docling (OCR off) for its structured Markdown; only documents with scanned pages take the native path.

Choosing the OCR engine

Benchmarked 2026-08 on English, bilingual EN/ZH, table-heavy and degraded scans (all engines driven through docling so layout handling is identical):

For deepseek, point the worker at an OpenAI-compatible endpoint:

OCR_ENGINE=deepseek
OCR_DEEPSEEK_URL=http://localhost:11434/v1/chat/completions  # Ollama default
OCR_DEEPSEEK_MODEL=deepseek-ocr:3b
OCR_DEEPSEEK_TIMEOUT=300  # seconds per page request, both backends

Ollama works out of the box (ollama pull deepseek-ocr:3b); for real throughput serve deepseek-ai/DeepSeek-OCR with vLLM on a GPU and set the URL accordingly. The native backend sends pages one at a time, so a slow laptop-hosted model only needs a generous OCR_DEEPSEEK_TIMEOUT; docling’s VLM pipeline honours the same timeout per request but keeps its own four concurrent requests.

A selected engine that is not available degrades rather than failing parses: under docling, a missing tesseract binary or non-macOS ocrmac falls back to auto with a warning; under native, a docling-only engine becomes tesseract, and a missing tesseract binary fails the scanned file with an install hint (text-layer documents are unaffected).

DeepSeek through docling versus native

OCR_ENGINE=deepseek behaves differently on the two backends, and the difference matters more than for tesseract:

• native sends one request per page that lacks a text layer and reads every other page directly. Pages go one at a time with OCR_DEEPSEEK_TIMEOUT per request, so a slow model server just takes longer.
• docling uses its VLM pipeline, which replaces the whole converter: every page of every PDF that reaches docling goes to the model, text layer or not (a 39-page text PDF measured 721 s against 20 s on the other paths), with four concurrent requests and OCR_DEEPSEEK_TIMEOUT per request. In testing it also dropped page-level elements — a title, an intro paragraph — that the same model kept through the native path, and it logs a burst of pydantic serialization warnings per conversion. Its one advantage was table structure: it was the only configuration that returned an invoice’s subtotal/VAT/total rows as table rows.

Measured on the same scans (2026-09, MacBook Air, Ollama deepseek-ocr:3b):

Recommendation: native with tesseract for throughput, native with deepseek for quality, docling with tesseract when you want its layout model, and docling with deepseek only for table-heavy scans where the costs above are acceptable.

Processing Flow

Source Docs flow (Upload and Train)

1. Files are uploaded through /api/upload.
2. Ingestion runs asynchronously in Celery (ingest_worker).
3. SimpleDirectoryReader parses files with get_default_file_extractor.
4. Documents are parsed by the DOC_PARSER_ENGINE engine; images (and, under anydoc, scanned PDFs) reach the OCR backend. OCR in this path is controlled by OCR_ENABLED.
5. Parsed text is chunked, embedded, and stored in the vector store.
6. Retrieval during chat uses this indexed text and returns source citations.

Attachment flow (Chat-only file context)

1. Files are uploaded through /api/store_attachment.
2. Celery task attachment_worker parses and stores the attachment in Postgres (attachments table).
3. OCR in this path is controlled by OCR_ATTACHMENTS_ENABLED.
4. Attachments are not vectorized and are not added to the source index.
5. During answer generation, selected attachment IDs are loaded and passed directly to the LLM pipeline.

How Docling OCR Works

With OCR_BACKEND=docling, OCR behavior is different for PDFs vs images:

• PDF parser defaults to hybrid OCR:
  • text regions: extracted directly
  • bitmap/image regions: OCR only where needed
• Image parser defaults to full-page OCR (the whole image is visual content).

The engine and its languages come from OCR_ENGINE and OCR_LANGS (see the table above). INSTALL_TESSERACT=true installs only the English tesseract pack; for other languages install their packs in the image (e.g. apt-get install tesseract-ocr-chi-sim) and list them in OCR_LANGS (eng+chi_sim).

Model compilation

Docling runs its layout, table, and OCR models through torch.compile by default. DocsGPT turns that off. On x86-64 Linux, compiling raised the first parse of a two-page PDF from 8.4s to 58.6s while steady-state parsing stayed at 1.5s either way, so the warmup is overhead a per-file parse never recovers. Compiling also fails outright on Apple Silicon, on install paths containing a space, on Windows without MSVC, and in slim images with no C compiler.

DOCLING_COMPILE_TORCH_MODELS=false

Set it to true only if you are parsing large batches on a machine where the warmup pays for itself.

Attachment Behavior by Model Support

When attachments are used in chat, behavior depends on the selected model/provider:

• If a MIME type is supported, DocsGPT sends files/images through provider-native attachment APIs.
• If unsupported, DocsGPT falls back to the parsed text content stored for the attachment.
• For providers that support images but not native PDF attachments, PDF files are converted to images (synthetic PDF support).

This means OCR quality is especially important for text fallback paths and for models without native attachment support.

Recommended Configuration

For most OCR-enabled use cases, enable both flags and leave the backend on auto:

OCR_ENABLED=true
OCR_ATTACHMENTS_ENABLED=true

After changing these settings, restart the API and Celery worker.

Legacy Fallback Notes

• If Docling is unavailable, DocsGPT falls back to the native OCR parsers (OCR on) or the legacy parsers (OCR off) for the formats anydoc does not cover, and anydoc’s refusals of scanned PDFs become upload errors only when no OCR is available.
• With OCR disabled, text-based PDFs can still parse, but scanned/image-heavy content may produce little text.
• For image parsing without OCR, the legacy image parser only extracts text when PARSE_IMAGE_REMOTE=true.