[
  "https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API",
  "https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_grid_layout",
  "https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API"
]

Max pages

maxPages

Optional

Maximum public knowledge-base pages or snapshots to process.

Type:integer

Minimum:1

Maximum:25

Default:1

Max depth

maxDepth

Optional

Maximum same-site crawl depth for future live crawling. Source-record mode uses depth 0.

Type:integer

Minimum:0

Maximum:3

Default:0

Include URL patterns

includePatterns

Optional

Only process URLs containing one of these strings when provided.

Type:string[]

Default:

[
  "/en-US/docs/"
]

Exclude URL patterns

excludePatterns

Optional

Skip URLs containing any of these strings.

Type:string[]

Default:

[
  "/login",
  "/admin"
]

Request delay

requestDelayMs

Optional

Delay between public fetches in milliseconds for polite crawling.

Type:integer

Minimum:0

Maximum:10000

Default:250

Request timeout

timeoutMs

Optional

Per-request timeout in milliseconds.

Type:integer

Minimum:1000

Maximum:60000

Default:20000

Max records

maxRecords

Optional

Maximum dataset rows to emit before stopping gracefully.

Type:integer

Minimum:1

Maximum:500

Default:10

Respect robots.txt

respectRobotsTxt

Optional

Keep enabled for public crawling. Source-record analysis does not fetch blocked pages.

Type:boolean

Default:true

Redact query strings

redactQueryStrings

Optional

Remove query strings from output URLs unless query text is core evidence.

Type:boolean

Default:true

Max charge

maxChargeUsd

Optional

Maximum local PPE budget before the actor stops before paid output.

Type:number

Minimum:0

Maximum:20

Default:20

Debug

debug

Optional

Enable extra non-secret diagnostics in logs and KVS reports.

Type:boolean

Default:false

Source records

sourceRecords

Optional

Optional authorized public knowledge-base HTML snapshots for deterministic analysis without live fetching.

Type:object[]

Front Knowledge Base

canadesk/front-knowledge-base

Get Categories and Articles from any public Front Knowledge Base. It's fast and costs little.

Canadesk Support

Website to RAG Knowledge Dataset

ghostgrid/website-to-rag-knowledge-dataset

Convert a website sitemap into clean text and markdown rows for RAG, AI search, chatbots, and knowledge base workflows.

GhostGrid

Site to LLM Knowledge Base

adambounhar/site-to-knowledge-base

Turn any website or docs into clean, LLM-ready Markdown for RAG and AI agents — one record per page, each with a token count. Sitemap- and robots.txt-aware, with predictable per-page pricing (no token credits). Simple knowledge-base ingestion.

Mohamed Adam BOUNHAR

RAG Doctor: Audit & Repair Your AI Knowledge Base

sanya_kumari/rag-doctor

Audit and repair the content you feed your AI. Finds contradictions, stale facts, duplicates, dead links, and broken chunks that quietly poison RAG, agents, and custom GPTs. Returns a scored report, a prioritized fix list, and a cleaned, ready-to-index knowledge base.

Sanya Kumari

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

Deprecated

kadiremir/deprecated

deprecated

Kadir Emir

Website To LLM Knowledge Pack

attainable_iota/website-to-llm-knowledge-pack

Crawl any website and turn it into an LLM-ready knowledge pack. This Actor extracts clean main text + metadata, follows links with depth/URL filters, and outputs per-page dataset items plus knowledge.jsonl, knowledge.md, and manifest.json for RAG/embeddings pipelines.

M Junaid Shaukat

Camera Equipment Knowledge Base (Wikidata)

crawlerbros/mpb-scraper

Look up camera bodies and lenses from Wikidata's open knowledge graph. Returns encyclopedic entries with brand, release year, category, image, Wikipedia link, and QID. Reference/knowledge data only - no prices, conditions, or live listings.

Crawler Bros