Documentation site URL list for RAG and LLMs

Created by

Joshua White

Actor

Sitemap URL Extractor – All Pages for RAG & SEO

Collects every page URL under a documentation path, ready to load into a RAG pipeline, vector database or website-content crawler.

Try for free

Sitemap URL Extractor – All Pages for RAG & SEOoldjard/sitemap-url-extractor

URL

Last modified

Title (feeds/news)

Found via

+4 fields

Input

Websites(required):https://docs.apify.com/academy+1

Max URLs per site:50

Output fields

URL

Last modified

Title (feeds/news)

Found via

Sitemap / feed / page

Allowed by robots.txt

Site

Error (input entries that aren't web addresses)

How it works

Sign up on Apify or log in

The Sitemap URL Extractor – All Pages for RAG & SEO Actor is ready to run with the inputs already set for "Documentation site URL list for RAG and LLMs". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Documentation site URL list for RAG and LLMs" task

The Sitemap URL Extractor – All Pages for RAG & SEO Actor will start running based on the input automatically. You can also change all the input fields, like websites and max urls per site, to match your specific use case.

Receive the "Documentation site URL list for RAG and LLMs" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL, last modified, title (feeds/news), and more.

Integrate into your workflow

The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "Documentation site URL list for RAG and LLMs" task via API.