Website Intelligence Crawler

Crawl a public website and get its pages back as clean text, Markdown and chunks sized for an embedding model: title, description, language, word count, the same-site links and the status code ride along on every row. No API key, no model call, nothing to authorise.

The one limit to understand first: there is no browser here. It reads the HTML the server sends, so a page that builds its content in JavaScript after loading comes back with very little text, and no setting changes that. For ordinary documentation, blogs, marketing sites and knowledge bases, the HTML is the content.

Input	Public HTTP or HTTPS URLs to start from
Output	One row per page: readable text, Markdown, chunks with token estimates, title, description, language, links, status
Ceiling	100 pages per starting URL, 5 levels of same-site links
Account needed	None
Price	$0.60 per 1,000 pages ($0.0006 each), flat on every plan. The free plan's $5 a month covers about 8,300

🔍 What Website Intelligence Crawler does

From each URL you give it, it follows same-site links down to the depth you set and writes a record per page. It stays on the site it started on, so an outbound link to somebody else's domain is not followed.

The text is the readable part of the page, with navigation, scripts and boilerplate stripped. The Markdown keeps the heading structure, paragraphs and lists, which is the version you want if the pages are going into a document store.

Chunks are cut to the size you ask for with a little overlap, so a sentence is not sliced in half. Each one carries its character count and an estimated token count, so you can budget an embedding run before you start it.

Give it a query and every chunk gets a relevanceScore against those words. It is plain term matching, not a model, and it is there to let you keep the 20 chunks that matter instead of all 400. The chunking and the scoring are both arithmetic, so the same input gives the same output every time.

📋 What data you get from each page

What you get	Field
The page's readable text	text
The same content as Markdown, headings kept	markdown
Chunks with character count, token estimate and relevance score	chunks
Title, description, language and canonical address	title, description, language, canonicalUrl
Same-site links with their anchor text	links
Word count and HTTP status	wordCount, status
Where it came from: the starting URL, the site and the link depth	seedUrl, siteKey, depth
What was asked for and what answered after redirects	requestedUrl, url

▶️ How to crawl a website into text, Markdown and chunks

1. Open Website Intelligence Crawler and click Try for free.
2. Put your address into Start URLs.
3. Set Maximum pages and Maximum link depth. Ten pages at depth 1 is a good look before you commit to a whole site.
4. If the crawl is feeding a search or a RAG index, set Chunk size to whatever your embedding model likes.
5. Click Start, then download the dataset as JSON, CSV or Excel, or read it from the API.

Leave the URLs empty and you get one sample row showing the shape, with nothing fetched and nothing charged.

💰 How much does it cost to crawl a website?

$0.60 per 1,000 pages, which is $0.0006 each. Flat on every Apify plan, no volume tiers. The free plan's $5 a month covers about 8,300 pages.

You pay per page that came back. The sample row is not charged, and neither is a page that failed, was disallowed by robots.txt, was too large, or turned out to be a PDF. Those still arrive as rows so you know they happened.

Fifty pages is three cents. A hundred-page site crawled weekly is about a quarter a month. If you set a maximum charge for a run, it stops at the last page that fits, and the run's status says so.

📥 What you give it

{
  "startUrls": [{"url": "https://docs.example.com/"}],
  "maxPages": 50,
  "maxDepth": 2,
  "outputFormat": "both",
  "chunkSizeChars": 1200,
  "query": "pricing plans billing"
}

Field	Default	What it is
startUrls	none	The pages to start from. Normalised and deduplicated. The Console box starts with https://example.com/.
urls	none	More starting URLs, as plain strings, if that shape suits you better.
maxPages	10	Records per starting URL, 1 to 100. Failed pages count too.
maxDepth	1	Same-site link depth, 1 to 5. Setting it to 0 behaves the same as 1, so use maxPages: 1 when you want the starting page alone.
maxConcurrency	3	Pages fetched at once from each site, 1 to 10. Several starting URLs are also crawled side by side up to the same number, so 3 with three starting URLs can mean nine requests at once. Leave it low on small servers.
useRobotsTxt	true	Honour Disallow rules and crawl delay. The delay is kept between batches of pages, so use maxConcurrency 1 for a site that asks for one. Allow exceptions are not read, so a site that relies on them may be crawled less than it permits.
query	none	Words to score chunks against. No model, no key.
chunkSizeChars	1200	Target characters per chunk, 300 to 5,000.
chunkOverlapChars	120	Characters repeated between neighbouring chunks, up to 800. Zero behaves as 120.
outputFormat	both	json keeps the text, markdown keeps the Markdown, both keeps each.
requestTimeoutSecs	15	Per request, 3 to 60.
maxResponseSizeKb	2048	Biggest page body accepted, 128 to 8,192 KB.
fallbackToProxy	false	Retry a page that could not be read.
proxyConfiguration	none	Your own servers, for that retry, if you have them.

📤 What you get back

A real row from a run on 27 September 2026, the first of five pages read from python.org. The text, the Markdown and the chunk texts are cut short, and the links list is cut to two of fifteen. No query was set, so relevanceScore is null.

{
  "ok": true,
  "requestedUrl": "https://python.org/",
  "url": "https://python.org/",
  "seedUrl": "https://python.org/",
  "siteKey": "python.org",
  "depth": 0,
  "status": 200,
  "title": "Welcome to Python.org",
  "description": "The official home of the Python Programming Language",
  "language": "en",
  "canonicalUrl": "https://python.org/",
  "text": "Get Started\nWhether you're new to programming or an experienced developer, it's easy to learn and use Python. ...",
  "markdown": "## Get Started\n\nWhether you're new to programming or an experienced developer, it's easy to learn and use Python.\n\n...",
  "links": [
    { "url": "https://python.org/about/gettingstarted/", "text": "Start with our Beginner’s Guide" },
    { "url": "https://python.org/downloads/release/python-3147/", "text": "Python 3.14.7" }
  ],
  "wordCount": 295,
  "contentType": "text/html",
  "chunks": [
    { "chunkIndex": 0, "text": "Get Started\nWhether you're new to programming ...", "charCount": 1200, "estimatedTokens": 300, "relevanceScore": null },
    { "chunkIndex": 1, "text": "26\n\nSuccess Stories\nMore ...", "charCount": 1051, "estimatedTokens": 263, "relevanceScore": null }
  ],
  "query": null,
  "outputFormat": "both",
  "fetchedAt": "2026-09-27T03:49:27.558Z"
}

Field	How to read it
requestedUrl, url, canonicalUrl	What was asked for, what answered after redirects, and what the page says its own address is.
text	The readable content, boilerplate removed. Absent when outputFormat is markdown.
markdown	Headings, paragraphs and lists. Absent when outputFormat is json. A page built entirely of tables falls back to the plain text.
links	Same-site links found on the page, with their anchor text, up to 250.
chunks	chunkIndex, text, charCount, estimatedTokens, and relevanceScore when you gave a query. With a query, the best chunks come first.
wordCount, language, status	Useful for filtering out thin pages and redirect landing pages before you embed anything.
contentType	Always reads text/html, because HTML is the only thing kept.

🧾 Reading the output

Page rows carry ok: true. Failures carry diagnostics: true with an errorCode, and are never charged.

Row	How to spot it	Charged
A page	ok is true	yes
The sample row	_sample is true	no
A failed page	diagnostics is true, and errorCode says why	no

Code	What it means
BAD_INPUT	Not a public HTTP or HTTPS address. Credentials in the URL and other schemes land here.
ROBOTS_DISALLOWED	The site's own robots.txt asks crawlers not to read that path.
UNSUPPORTED_CONTENT	A PDF, an image, JSON, anything that is not an HTML page.
HTTP_ERROR	The server answered with an error status.
NETWORK	The server could not be reached, or answered badly.
TIMEOUT	Slower than requestTimeoutSecs.
RESPONSE_TOO_LARGE	Bigger than maxResponseSizeKb.
SSRF_BLOCKED	The address points somewhere private, so it was refused.

One quirk of the table view: a failed row has no url, and the address it tried sits in requestedUrl, which the view does not show. Open the row or export the dataset to see which page it was.

💡 What people use it for

• The dull half of a RAG pipeline: crawl the docs, get chunks with token counts, embed them.
• Keeping a support bot's knowledge current by crawling the help centre again on a schedule.
• Content audits: wordCount, title, description and status across every page, in one export.
• Finding the pages that actually mention a term, using query and relevanceScore.

🚧 What it does not do

• No browser. Content a script builds after loading is not there to read.
• Same site only. Links to other domains are not followed.
• HTML only. PDFs, Word files and images come back as unsupported rather than as text.
• Nothing behind a login. No cookies, no sign-in, no interactive challenges.
• 100 pages per starting URL. For a bigger site, give it several starting URLs, one per section.
• No screenshots. The screenshot generator linked below does that.
• No summarising or tagging. No model runs here, which is the point: nothing is invented and nothing is rephrased.
• Depth 0 is not honoured. It behaves as depth 1.
• It is not an official tool of any platform or company named on this page. Dami's Studio is independent and is not affiliated with or endorsed by any of them.

🧭 Which site tool do you need?

If you want	Use
A site as text, Markdown and chunks	This one
Pictures of the pages instead	Website Screenshot Generator
A site's DNS, certificate, security headers and tech signals	Domain Inspector
Full Wikipedia articles as text	Wikipedia Scraper
What changed on a rival's pricing page or changelog since last week	SaaS Competitor Monitor
Questions and answers from Stack Overflow	Stack Overflow Scraper

❓ Questions people ask

Does it run JavaScript?

No. It reads the HTML the server sends, which is the whole content on most documentation and marketing sites.

Will it hammer my site?

It keeps to robots.txt, crawl delay included, between batches of pages. Drop maxConcurrency to 1 and it fetches one page at a time.

What chunk size should I use for embeddings?

Whatever your embedding model is happiest with. 1,200 characters with 120 of overlap is a reasonable default, and estimatedTokens on each chunk tells you where you landed.

Can I crawl one page only?

Set maxPages to 1. Depth 0 does not do it.

Do I get both text and Markdown?

With outputFormat: both, yes, on the same row. Pick one if the rows are getting large.

Can I call it from code or connect it to an AI assistant?

Yes. The API tab has ready-made code for Python, JavaScript and the command line. For Claude, ChatGPT or another MCP client, connect https://mcp.apify.com/?tools=fetch-actor-details,dami_studio/website-intelligence-crawler. Either way the run happens on your Apify account at the same price.

Is crawling a public site legal?

It reads public pages and honours robots.txt. Site terms still apply to what you do with the content afterwards. Apify's write-up on the legality of web scraping is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the starting URL and the run ID. If a diagnostic row landed, its errorCode and requestedUrl usually name the reason already.