Crawl FastAPI docs into Markdown chunks for a vector database

Created by

Yukai Lin

Actor

Website to Markdown Crawler - Content Crawler for LLM & RAG

Crawl the FastAPI tutorial and split each page into Markdown chunks of about 1,500 characters with 150 characters of overlap, ready to embed into Pinecone, Qdrant, Chroma or pgvector. Pages are found via links and the sitemap; the include pattern keeps the crawl on /tutorial/.

Try for free

Website to Markdown Crawler - Content Crawler for LLM & RAGtidytools/website-markdown-crawler

URL

Title

Depth

+6 fields

Input

Start URLs:https://fastapi.tiangolo.com/tutorial/

Include URL patterns:https://fastapi.tiangolo.com/tutorial/**

Max pages:20

RAG chunk size (characters):1500

Output fields

URL

Title

Depth

Markdown

Start URL

Charged

Error type

Error

What to try

How it works

Sign up on Apify or log in

The Website to Markdown Crawler - Content Crawler for LLM & RAG Actor is ready to run with the inputs already set for "Crawl FastAPI docs into Markdown chunks for a vector database". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Crawl FastAPI docs into Markdown chunks for a vector database" task

The Website to Markdown Crawler - Content Crawler for LLM & RAG Actor will start running based on the input automatically. You can also change all the input fields, like start urls, include url patterns, and more, to match your specific use case.

Receive the "Crawl FastAPI docs into Markdown chunks for a vector database" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL, title, depth, and more.