Welcome to the new DocsGPT 🦖 docs! 👋
📥 Training on docs

How to train on other documentation

This AI can utilize any documentation, but it requires preparation for similarity search. Follow these steps to get your documentation ready:

Step 1: Prepare Your Documentation video-example-of-how-to-do-it

Start by going to /scripts/ folder.

If you open this file, you will see that it uses RST files from the folder to create a index.faiss and index.pkl.

It currently uses OPENAI to create the vector store, so make sure your documentation is not too large. Using Pandas cost me around $3-$4.

You can typically find documentation on GitHub in the docs/ folder for most open-source projects.

1. Find documentation in .rst/.md format and create a folder with it in your scripts directory.

  • Name it inputs/.
  • Put all your .rst/.md files in there.
  • The search is recursive, so you don't need to flatten them.

If there are no .rst/.md files, convert whatever you find to a .txt file and feed it. (Don't forget to change the extension in the script).

Step 2: Configure Your OpenAI API Key

  1. Create a .env file in the scripts/ folder.
  • Add your OpenAI API key inside: OPENAI_API_KEY=.

Step 3: Run the Ingestion Script

python ingest.py ingest

It will provide you with the estimated cost.

Step 4: Move index.faiss and index.pkl generated in scripts/output to application/ folder.

Step 5: Run the Web App

Once you run it, it will use new context relevant to your documentation.Make sure you select default in the dropdown in the UI.


You can learn more about options while running ingest.py by running:

  • Make sure you select 'default' from the dropdown in the UI.


You can learn more about options while running ingest.py by executing: python ingest.py --help

ingestRuns 'ingest' function, converting documentation to Faiss plus Index format
--dir TEXTList of paths to directory for index creation. E.g. --dir inputs --dir inputs2 [default: inputs]
--file TEXTFile paths to use (Optional; overrides directory) E.g. --files inputs/1.md --files inputs/2.md
--recursive / --no-recursiveWhether to recursively search in subdirectories [default: recursive]
--limit INTEGERMaximum number of files to read
--formats TEXTList of required extensions (list with .) Currently supported: .rst, .md, .pdf, .docx, .csv, .epub, .html [default: .rst, .md]
--exclude / --no-excludeWhether to exclude hidden files (dotfiles) [default: exclude]
-y, --yesWhether to skip price confirmation
--sample / --no-sampleWhether to output sample of the first 5 split documents. [default: no-sample]
--token-check / --no-token-checkWhether to group small documents and split large. Improves semantics. [default: token-check]
--min_tokens INTEGERMinimum number of tokens to not group. [default: 150]
--max_tokens INTEGERMaximum number of tokens to not split. [default: 2000]
convertCreates documentation in .md format from source code
--dir TEXTPath to a directory with source code. E.g. --dir inputs [default: inputs]
--formats TEXTSource code language from which to create documentation. Supports py, js and java. E.g. --formats py [default: py]

MIT 2024 © DocsGPT