JavaScript Website Crawler

Crawl a website that builds its pages with JavaScript, and get every page as clean text, Markdown and chunks: each page is opened in a headless browser and read once its scripts have run, so the content a plain HTML read never sees is in the row. Title, description, headings, links, structured data and the status code come with every page. The catch is speed and price: a page takes from one second to nine in the browser, depending on how much script it runs, and on a site whose HTML already holds the text, a plain crawl does the same job faster and for less.

Input	Public web addresses to start from
Output	One row per page, as the browser saw it: text, Markdown, chunks with token estimates, title, description, headings, links, structured data, status
Ceiling	1,000 pages per start URL, 20 levels of links
Speed	About 1 to 5 seconds a page on most sites and 5 to 9 on script-heavy ones, at the default memory (measured on 29 October 2026)
Account needed	None
Price	$2.00 per 1,000 pages ($0.002 each), flat on every plan. The free plan's $5 a month covers about 2,500

🔍 What JavaScript Website Crawler does

It starts at each URL you give it and follows links to other pages of the same site, down to the depth you set.

Every page is read twice. First the plain way, which is where robots.txt, redirects, the size limit and the check that it is an HTML page happen. Then in a headless browser that runs the page's scripts, waits for the page to settle and hands back what ended up on it. The row is built from that second read.

Some pages need more than a settle. Five settings cover that: wait for a CSS selector, wait a few seconds more, scroll to the bottom so content that loads on scroll arrives, open collapsed sections such as FAQ answers, and skip images, fonts and styles so pages load faster.

Every row carries needsJavaScript, judged on the HTML the server sent before any script ran. It tells you which pages really needed the browser. If most of a site comes back false, Website Intelligence Crawler reads it without one.

The text, Markdown and chunks work as in that crawler: boilerplate out, headings kept, chunks cut to your size with a little overlap and an estimated token count.

📋 What data you get from each page

What you get	Field
The page's readable text, after its scripts ran	text
The same content as Markdown, headings kept	markdown
Chunks with character count, token estimate and relevance score	chunks
Title, description, language and canonical address	title, description, language, canonicalUrl
Same-site links with their anchor text, the ones scripts wrote included	links
Word count and HTTP status	wordCount, status
Where it came from: the start URL, the site, the link depth and the page that linked here	seedUrl, siteKey, depth, referrerUrl
The address asked for, and the one the page ended on	requestedUrl, url, finalUrl
Whether the page was an empty shell until its scripts ran	needsJavaScript
Time the server took, and time the browser spent	responseTimeMs, renderTimeMs
Open Graph, site name, sharing image, published date, keywords	openGraph, siteName, imageUrl, publishedAt, keywords
Headings with their level	headings
Links to other websites, and link totals by place on the page	externalLinks, linkStats
Robots tag, noindex and where the canonical points	indexability
Structured data: its types and the JSON-LD blocks	schemaTypes, jsonLd
Images with alt text and declared size	images, imageStats
Paragraph and sentence counts, reading ease, top words	contentStats
Response headers	responseHeaders
Pages in the run sharing a title or description	duplicateTitleOf, duplicateDescriptionOf
Optional: the cleaned HTML of the content	html

▶️ How to crawl a JavaScript website

1. Open JavaScript Website Crawler and click Try for free.
2. Put your address into Start URLs.
3. Set Pages per start URL and Link depth. Ten pages at depth 1 is a good first look.
4. If the content you want arrives late, put a CSS selector that only exists once it is there into Wait for this element.
5. Click Start, then download the dataset as JSON, CSV or Excel, or read it through the API.

💰 How much does it cost to crawl a JavaScript website?

$2.00 per 1,000 pages, so $0.002 a page, the same on every plan. The free plan's $5 a month covers about 2,500 pages.

You pay for the pages that came back. Note rows are free: a page the browser could not render, one the site refused, one robots.txt disallows, a PDF, a page that grew too big. So is the summary row each site ends with.

Fifty pages cost ten cents. If you set a maximum cost for the run, it is checked before the browser opens each page, and the run stops at the last page that fits.

📥 What you give it

{
  "startUrls": [{"url": "https://docsify.js.org/"}],
  "maxPages": 20,
  "maxDepth": 1,
  "waitForSelector": "#main",
  "keepSelector": "#main",
  "outputFormat": "both",
  "chunkSizeChars": 1200
}

Field	Default	What it is
startUrls	none, required	The pages to start from. The Console box starts with https://example.com/.
maxPages	10	Pages per start URL, 1 to 1,000. Pages that could not be read count too, unless countOnlyGoodPages is on.
maxDepth	1	Link depth, 0 to 20. 0 reads the start pages only.
maxPagesPerRun	none	One page limit for the whole run, across the start URLs.
maxConcurrency	3	Pages of one site worked on at once, 1 to 10.
useRobotsTxt	true	Honour Disallow rules and the crawl delay, capped at 10 seconds.
requestTimeoutSecs	15	Seconds for the first read of a page, 3 to 60. The browser gets twice this, 20 to 60 seconds.
maxResponseSizeKb	2048	Biggest page accepted, 128 to 8,192 KB. A page that grows past twice this (1 MB at least) once its scripts run is refused too.
waitForSelector	none	Wait up to 5 seconds for this CSS selector. A page where it never appears is a free SELECTOR_NOT_FOUND note.
renderWaitSecs	0	Extra seconds to wait once the page has settled, 0 to 5.
scrollToBottom	false	Scroll down a few times so content that loads on scroll is there.
expandSections	false	Open details boxes and buttons marked as collapsed. Links, forms and send buttons are never clicked.
blockMedia	false	Don't load images, video, fonts or stylesheets, so pages load faster. If a page comes back with less text with it on, run that page without it.
includeUrlGlobs	none	Follow only links matching these patterns, such as /docs/**. * stays inside a folder, ** crosses folders. Up to 20. Start URLs are always read.
excludeUrlGlobs	none	Never follow links matching these patterns, such as **?page=*.
stayUnderStartPath	false	A crawl started at /docs/ stays in /docs/.
includeSubdomains	false	docs.example.com counts as part of example.com.
followNavigationLinks	false	Also follow links in menus, headers and footers.
skipFileLinks	false	Don't request links ending in a file type: PDFs, images, archives, feeds.
countOnlyGoodPages	false	Pages that could not be read don't use up maxPages; tries stop at twice the limit.
useSitemap	false	Also start from the sitemap, indexes included. A start URL ending in .xml is read as a sitemap.
requestLanguage	US English	The language list sent with each page, such as de-DE,de;q=0.9.
outputFormat	both	json keeps the text, markdown keeps the Markdown, both keeps each.
markdownStyle	basic	rich keeps links, bold, tables and code blocks too. Chunks are always cut from the plain text.
contentExtractor	default	default reads the main or article element, article picks the block with the most paragraph text, wholePage reads the body.
keepSelector	none	Keep only what matches this CSS selector. No match keeps the normal text and sets keepSelectorMatched to false.
removeSelector	none	Remove what matches this CSS selector before the text is read.
removeCookieBanners	false	Remove cookie and consent dialogs from the text.
dropRepeatedLines	false	Drop text blocks an earlier page of the same site already had.
includeHtml	false	Add the cleaned HTML of the content, as it stood after the scripts ran, as html. Rows get several times bigger.
query	none	Words to score chunks against. Plain word matching, no AI.
chunkSizeChars	1200	Target characters per chunk, 300 to 5,000.
chunkOverlapChars	120	Characters repeated between neighbouring chunks, up to 800.
saveSkippedUrls	false	Save the links your rules left out, with the reason, in the key-value record SKIPPED_URLS.

📤 What you get back

A real row from a run on 29 October 2026: the start page of docsify.js.org, a documentation site whose server sends an empty shell and whose script writes the page, read with waitForSelector and keepSelector both set to #main. The text, the Markdown and the chunk are cut short, and most of the metadata is left out.

{
  "ok": true,
  "requestedUrl": "https://docsify.js.org/",
  "url": "https://docsify.js.org/",
  "seedUrl": "https://docsify.js.org/",
  "siteKey": "docsify.js.org",
  "depth": 0,
  "status": 200,
  "title": "docsify",
  "description": "A magical documentation generator.",
  "language": "en",
  "text": "docsify\nA magical documentation site generator.\nWhat it is\nDocsify turns your Markdown files into a documentation website instantly. ...",
  "markdown": "# docsify\n\n> A magical documentation site generator.\n\nA magical documentation site generator.\n\n## What it is\n\n...",
  "wordCount": 147,
  "chunks": [
    { "chunkIndex": 0, "text": "docsify\nA magical documentation site generator.\nWhat it is ...", "charCount": 889, "estimatedTokens": 223, "relevanceScore": null }
  ],
  "headings": [{ "level": 1, "text": "docsify" }, { "level": 2, "text": "What it is" }, { "level": 2, "text": "Features" }],
  "keepSelectorMatched": true,
  "finalUrl": "https://docsify.js.org/#/",
  "responseTimeMs": 153,
  "needsJavaScript": true,
  "renderTimeMs": 1929,
  "fetchedAt": "2026-10-29T21:46:56.171Z"
}

Field	How to read it
needsJavaScript	true: the HTML the server sent was an empty shell, and only the browser got the text. false: the page had its text before any script ran.
renderTimeMs	Milliseconds the browser spent on the page, waits included. Your extra waits show up here.
finalUrl	Where the page ended up once its scripts ran. On a single-page site this can carry a #/ route.
text, markdown	What the page showed after its scripts ran, ads and widgets the scripts added included. removeSelector takes them out.
chunks	chunkIndex, text, charCount, estimatedTokens, and relevanceScore when you gave a query. With a query, the best chunks come first.

🧾 Reading the output

Page rows carry ok: true, and they are the only rows charged. Every other row is a free note with diagnostics: true and an errorCode.

Row	How to spot it	Charged
A page	ok is true	yes
A note	diagnostics is true, and errorCode says what happened	no
The site's summary	_summary is true; one per start URL, after its pages, with the counts and stopReason	no

Code	What it means
RENDER_FAILED	The browser could not finish the page in time. Re-run it later, or give it a longer requestTimeoutSecs.
SELECTOR_NOT_FOUND	The element in waitForSelector never appeared.
HTTP_ERROR	The server answered with an error status, which status carries.
ROBOTS_DISALLOWED	The site's robots.txt asks crawlers not to read that path.
UNSUPPORTED_CONTENT	A PDF, an image, JSON: anything that is not an HTML page.
RESPONSE_TOO_LARGE	Bigger than maxResponseSizeKb, or more than twice that once its scripts ran.
NETWORK, TIMEOUT	The server could not be reached, or was slower than requestTimeoutSecs.
SSRF_BLOCKED	The address, or one the page tried to go to, points inside a private network, so it was refused.
LEFT_OUT	A page this crawler does not visit. The row carries no address.
BAD_INPUT	No start URL, or an option value it cannot read; field names it. Nothing is crawled.
COST_LIMIT	The run's maximum cost does not cover one page. Nothing is crawled.
TIME_LIMIT	The run stopped short of its time limit to keep what it had. It still ends as succeeded.
STOPPED_EARLY	The browser stopped working partway; the pages before it are delivered and charged.

To keep only the pages, filter on ok being true.

💡 What people use it for

• Feeding a RAG index from a documentation site built as a single-page app, where a plain crawl comes back with a title and no text.
• Reading product or listing pages whose prices and details are filled in by scripts after the page loads.
• Checking what a search engine's renderer would see on your own site, against what the raw HTML says, with needsJavaScript on every page.
• FAQ and help pages whose answers sit in collapsed sections, with expandSections on.

🚧 What it does not do

• Nothing behind a login. No cookies, no sign-in, no forms filled.
• No way past sites that refuse headless browsers. A page the site will not serve comes back as a free note, not a page.
• One site at a time. Links to other websites are listed in externalLinks but not followed.
• HTML pages only. PDFs, office files and images come back as notes rather than text.
• No screenshots and no clicking around. It reads pages; the only clicks are the collapsed-section ones you ask for.
• No summarising. No model runs here, so nothing is rephrased or invented.
• It is not an official tool of any platform or company named on this page. Dami's Studio is independent and is not affiliated with or endorsed by any of them.

🧭 Which website crawler do you need?

If you want	Use
Pages that only fill in once their scripts run	This one
Pages whose HTML already holds the text, faster and for less	Website Intelligence Crawler
Pictures of the pages instead of their text	Website Screenshot Generator
A site's DNS, certificate, security headers and tech signals	Domain Inspector
Full Wikipedia articles as text	Wikipedia Scraper

❓ Questions people ask

How do I know if a site needs a browser?

Crawl ten pages and look at needsJavaScript. Pages marked true were empty shells until their scripts ran; if almost every page is false, the plain crawler reads that site faster and for less.

Why is a page missing text I can see in my own browser?

Some content only arrives after a click, a scroll or a few seconds. Try scrollToBottom, expandSections, renderWaitSecs or a waitForSelector for the part you want. Content behind a sign-in is never read.

How long does a crawl take?

One to five seconds a page on most sites, five to nine on script-heavy ones, one page at a time at the default memory. So 1,000 pages of a heavy documentation site can outlast a run's default one-hour limit: raise the run's timeout, or it stops early with a free TIME_LIMIT row and keeps what it read. blockMedia cut the time by about a seventh on one such site.

Do I need an account on the site?

No. It reads public pages the way a signed-out visitor sees them.

Will it slow my site down?

It keeps to robots.txt, crawl delay included, and reads a page with the browser only after the plain read has passed those checks. Set maxConcurrency to 1 for a small server.

Can I call it from code or connect it to an AI assistant?

The API tab has code for Python, JavaScript and the CLI. MCP clients use https://mcp.apify.com/?tools=fetch-actor-details,dami_studio/javascript-website-crawler, and it bills your Apify account at the same price.

Is crawling a public website legal?

It reads public pages the way a signed-out visitor sees them, with no account, and honours robots.txt. Pages can hold personal data, which falls under GDPR and similar laws: keep only what you need. The site's own terms still apply to what you do with the content. Apify's write-up on the legality of web scraping is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the start URL and the run ID. If a note row landed, its errorCode and requestedUrl usually name the reason already.