JavaScript Website Crawler
Pricing
from $2.00 / 1,000 rendered pages
Crawl a public website in a headless browser and get each page as it looks after its scripts run: clean text, Markdown, links, metadata and chunks.
Pricing
from $2.00 / 1,000 rendered pages
Rating
0.0
(0)
Developer
Dami's Studio
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
9 hours ago
Last modified
Categories
Share
ok
Required
diagnostics
Optional
_notice
Optional
requestedUrl
Optional
url
Optional
seedUrl
Optional
siteKey
Optional
depth
Optional
status
Optional
title
Optional
description
Optional
language
Optional
canonicalUrl
Optional
text
Optional
markdown
Optional
wordCount
Optional
links
Optional
chunks
Optional
query
Optional
outputFormat
Optional
errorCode
Optional
error
Optional
fetchedAt
Optional
When the page was read.
finalUrl
Optional
The address the page ended on, after redirects and after its scripts ran.
referrerUrl
Optional
The page whose link led here, or the sitemap file. Empty for start pages.
responseTimeMs
Optional
Milliseconds the server took to send the page, before the browser opened it.
renderTimeMs
Optional
Milliseconds the browser spent on the page: loading it, the waits, scrolling and opening sections.
responseHeaders
Optional
Content type, last modified, ETag, content language and robots header of the answer.
openGraph
Optional
The page's Open Graph tags (title, description, image, type and the rest).
siteName
Optional
The site name the page declares.
imageUrl
Optional
The page's social sharing image.
publishedAt
Optional
The published date the page declares in its tags or structured data.
keywords
Optional
The page's keywords tag, split.
headings
Optional
Headings of the content, with their level.
externalLinks
Optional
Links to other websites with their anchor text, nofollow flag and where they sit, up to 250.
linkStats
Optional
Link totals: same site, other websites, nofollow, and how many sit in content, menus, headers and footers.
indexability
Optional
Robots tag, robots header, noindex and nofollow, the canonical address and whether it points elsewhere.
schemaTypes
Optional
Structured-data types found on the page.
jsonLd
Optional
The page's JSON-LD blocks, copied as published, up to 50,000 characters.
jsonLdTruncated
Optional
True when JSON-LD blocks were left out to stay under the size limit.
images
Optional
Images on the page: address, alt text and declared size, up to 100.
imageStats
Optional
How many images, and how many have no alt text or an empty one.
contentStats
Optional
Paragraph and sentence counts, a reading-ease score and the most frequent words.
needsJavaScript
Optional
True when the page the server sent was an empty shell until its scripts ran: the pages that needed the browser.
duplicateTitleOf
Optional
The first page in this run with the same title.
duplicateDescriptionOf
Optional
The first page in this run with the same description.
repeatedLinesDropped
Optional
Text blocks dropped because an earlier page had them (Drop repeated text blocks).
keepSelectorMatched
Optional
Whether the keep selector matched anything on this page.
html
Optional
Cleaned HTML of the content (Include cleaned HTML).
htmlTruncated
Optional
True when the HTML was cut at one million characters.
_summary
Optional
True on the free summary row each site ends with.
pagesDelivered
Optional
Summary, time-limit and cost-limit rows: pages delivered.
diagnosticRows
Optional
Summary row: note rows for this site.
leftOutByRules
Optional
Summary row: links your rules left out, by reason.
sitemapUrls
Optional
Summary row: start pages taken from the sitemap.
needsJavaScriptPages
Optional
Summary row: pages that were empty shells until their scripts ran.
stopReason
Optional
Summary row: why the site's crawl ended (finished, pageLimit, runPageLimit, attemptLimit, timeLimit, costLimit, browserUnavailable).
field
Optional
On a BAD_INPUT row: the input field it is about.
