Website Content Crawler avatar

Website Content Crawler

Pricing

from $1.19 / 1,000 results

Go to Apify Store
Website Content Crawler

Website Content Crawler

Website Content Crawler SD - Website Content Crawler is a web crawling tool that extracts the main content of every page of a website as clean Markdown and text, with title, language and metadata, for LLMs, RAG, vector databases and search indexes.

Pricing

from $1.19 / 1,000 results

Rating

0.0

(0)

Developer

Neuro Scraper

Neuro Scraper

Maintained by Community

Actor stats

0

Bookmarked

1

Total users

0

Monthly active users

5 days ago

Last modified

Features

Limited permissions

Share

Website Content Crawler — clean Markdown and text from whole websites

The Website Content Crawler collects public Website content data and returns it as a clean dataset you can download as JSON, CSV or Excel, or read through the Apify API. No login, no cookies and no code: give the Website Content Crawler your input, start it and get structured Website data for research, monitoring, analytics or lead generation. You pay only for the results you receive.

⭐ Featured by Neuro Scraper: Facebook Reels Scraper - scrape Facebook Reels from any page, profile or reel link with exact views and plays, every reaction type (Like, Love, Care, Haha, Wow, Sad, Angry), comments, shares, captions, music and video links. No login and no cookies. Try the Facebook Reels Scraper →

Website Content Crawler crawls a website from one or more start URLs and saves every page's main content as Markdown and plain text, with title, description, language, canonical URL, author, dates and metadata. Menus, footers and cookie banners are removed, so the output is ready for LLMs, RAG pipelines, vector databases and search indexes.

Input

FieldTypeDefaultMeaning
startUrlsURL listcrawlee.dev/python/docs/guides (prefill)Where the crawl starts
maxPagesinteger20Pages saved (one row each)
maxCrawlDepthinteger5Links away from a start URL; 0 = start URLs only
crawlScopeselectstartUrlPathstartUrlPath (URLs starting with the start URL), sameDomain, sameDomainAndSubdomains
includeUrlGlobs / excludeUrlGlobsglob lists–e.g. https://site.com/docs/**; include globs replace the scope
useSitemapsbooleanfalseAlso queue in-scope URLs from robots.txt sitemaps / /sitemap.xml
respectRobotsTxtbooleantrueSkip disallowed pages, honour Crawl-delay (max 10 s)
htmlTransformerselecttrafilaturatrafilatura, readability, or none (whole cleaned page)
saveMarkdown / saveText / saveHtmlbooleanstrue / true / falseOutput formats
keepMarkdownLinksbooleanfalseKeep [text](url) links in the Markdown
requestDelayMsinteger1000Pause between requests to one host
maxConcurrencyinteger2Pages processed at once across hosts
proxyConfigurationproxyApify datacenterUse RESIDENTIAL for sites that block datacenter IPs

Output (one row per page)

FieldMeaning
url, requestUrl, startUrl, referrerUrl, depth, statusCodeWhere the page came from
title, description, language, canonicalUrl, author, publishedAtPage metadata
markdown, text, htmlMain content in the formats you chose
wordCount, extractorWords in the main content; which extractor produced it
likelyJsRenderedTrue when the server sent almost no text (page builds itself with JavaScript)
truncatedTrue if a field was cut at 1.5 M characters
metadatakeywords, robots, modifiedAt, Open Graph, Twitter card, JSON-LD types, hreflang, content type, response time, size
crawledAtISO time (UTC)

Measured (2026-10-03, see TEST_REPORT.md)

  • crawlee.dev guides: 20 pages in 22 s; with sitemaps + globs: 60 pages in 67 s (59 found through the sitemap).
  • books.toscrape.com: 12 pages in 12 s; / and /index.html saved once (content de-duplication).
  • About 1 page per second per site with the default delay.

Limits

  • No browser: pages that render their text with JavaScript come out (nearly) empty and are flagged likelyJsRendered.
  • PDFs, images and other non-HTML files are skipped. No login or cookies.
  • Default scope follows only URLs that start with the start URL — start from a section or the homepage, or pick sameDomain.
  • Sites behind anti-bot challenges (Cloudflare, DataDome, ...) are logged as blocked and skipped; try a RESIDENTIAL proxy.

Example

{"startUrls": [{"url": "https://crawlee.dev/python/docs/guides"}], "maxPages": 50, "useSitemaps": true,
"excludeUrlGlobs": ["**/docs/next/**"]}

Why use the Website Content Crawler

The Website Content Crawler is a Website scraper and Website data extractor for anyone who needs public Website content data as clean data. The Website Content Crawler handles web scraping, proxy rotation and retries for you, so you get a ready dataset instead of raw pages. Use the Website Content Crawler for market research, competitor analysis, lead generation, analytics or monitoring, and export the results to JSON, CSV, Excel or Google Sheets.

  • No login and no code: the Website Content Crawler runs on Apify with one click or one API call.
  • Pay per result: the Website Content Crawler charges only for rows delivered - no monthly Website API fee.
  • Complete fields: the Website Content Crawler returns every public field, with a stable schema for automation.
  • Integrations: connect the Website Content Crawler to Zapier, Make, n8n, Google Sheets, a webhook or your CRM.

Website Content Crawler use cases

  • Market research: the Website Content Crawler collects Website content data at scale, so you can compare trends across weeks and niches.
  • Competitor analysis: run the Website Content Crawler on a schedule and export the dataset to Google Sheets or Excel to watch what competitors publish.
  • Lead generation and B2B prospecting: the Website Content Crawler gives a sales or agency team a clean list to load into a CRM.
  • Content and social media monitoring: the Website Content Crawler keeps a fresh copy of Website data through an Apify schedule and a webhook.
  • Analytics and dashboards: send the Website Content Crawler output as JSON through the Apify API into Python, Node.js or a BI tool.
  • Academic and data science projects: the Website Content Crawler returns a consistent schema, ready for pandas, R or a notebook.
  • Price, product and brand tracking: the Website Content Crawler turns public Website data into a dataset you can diff run by run.
  • Automation workflows: connect the Website Content Crawler to Zapier, Make or n8n so every new row starts your next step.

How to run it with the API, Python and Node.js

Start the Website Content Crawler from the Apify Console, or call it from code. The Website Content Crawler returns its dataset as JSON, CSV or Excel. With the Apify API you can run the Website Content Crawler from Python or Node.js, wait for it to finish and read the rows, which makes the Website Content Crawler easy to plug into an automation, a data pipeline or a scheduled export.

  1. Open the Website Content Crawler on Apify and fill in the input (or send the same input as JSON to the API).
  2. Start the Website Content Crawler. Follow the run log while the Website Content Crawler collects public Website data.
  3. Download the dataset as CSV, JSON, Excel or XLSX, or let a webhook send it on.
  4. Schedule the Website Content Crawler to keep your Website data fresh for monitoring and analytics.

Tips for better results

  • Start small: run the Website Content Crawler with a low maximum first, check the fields, then scale up the same input.
  • Split big jobs: several smaller Website Content Crawler runs (by keyword, URL list or date) finish faster and are easier to resume.
  • Keep a history: name a dataset or use a schedule so each Website Content Crawler run adds to your Website data over time.
  • Filter early: use the input filters of the Website Content Crawler so you only pay for the Website content data you really need.
  • Automate the hand-off: a webhook or the Zapier and Make integrations can move every Website Content Crawler result straight to Google Sheets, a CRM or a database.
  • Watch the run log: the Website Content Crawler writes what it is doing and why an item was skipped, so a surprising result is easy to explain.

Website Content Crawler FAQ

Can it replace the official Website API?

For public data, yes. The Website Content Crawler gives you Website data through the Apify API without an official Website API key, quota approval or developer account.

Which formats can I download?

The Website Content Crawler dataset downloads as JSON, CSV, Excel (XLSX), XML, RSS or an HTML table, and the Apify API returns the same Website Content Crawler data page by page.

What is the best Website scraper for content data?

The Website Content Crawler is built for exactly this job: public Website content data, complete fields, no login, pay per result and a clean export to JSON, CSV or Excel.

Can I schedule it?

Yes. Apify schedules run the Website Content Crawler hourly, daily or weekly, and a webhook can notify your system when new Website data is ready.

How much does it cost?

The Website Content Crawler is pay per result: you pay only for the rows you receive, with lower prices on higher Apify plans. You can cap the cost of every run with the maximum cost setting.

How is this different from other Website scrapers?

The Website Content Crawler combines complete public fields, no login, pay-per-result pricing and a stable output schema, so the Website Content Crawler fits both quick one-off exports and long-running automation.

Can I run it for many inputs at once?

Yes. Give the Website Content Crawler a list of inputs in one run; each row says which input it came from, so one Website Content Crawler run can cover a whole list.

Does it work with proxies?

Yes. The Website Content Crawler uses Apify Proxy by default and you can pick the proxy group or country in the input.

The Website Content Crawler collects only publicly available data. If results contain personal data, process it under GDPR or the rules that apply to you, and only for a legitimate purpose.

How do I get more results?

Raise the maximum items, add more inputs (keywords, URLs or profiles), or run the Website Content Crawler several times on a schedule. Results are de-duplicated.

How fast is it?

The Website Content Crawler runs requests in parallel and rotates proxy sessions automatically, so a typical run of a few hundred items finishes in minutes.

Do you offer custom scrapers?

Yes. If you need a field or a filter the Website Content Crawler does not have yet, contact Neuro Scraper through the Apify Store page and we can extend the Website Content Crawler for you.

What happens if a run is interrupted?

The Website Content Crawler saves its progress, so a resurrected or restarted run continues where it stopped instead of paying for the same rows again.

Can I use it through the API?

Yes. Start the Website Content Crawler and read its dataset through the Apify API from Python, Node.js or any HTTP client, or call it from Zapier, Make and n8n.

Can I send the data to my CRM?

Yes. Push Website Content Crawler rows to HubSpot, Salesforce, Pipedrive or any CRM through Zapier, Make or a webhook - a common setup for an agency or B2B sales team.

Do I need a Website account or login?

No. The Website Content Crawler works with no login and no cookies. It reads only public data, the same data any visitor can see.

Why is a field empty in my results?

The Website Content Crawler fills every field Website publishes for that item. When Website does not show a value - for example a figure the owner hid - the field stays empty rather than being guessed.

What does this Website scraper do?

The Website Content Crawler collects public Website content data and returns it as a structured dataset. It is web scraping without code: you set the input, start the Website Content Crawler and download JSON, CSV, Excel or XLSX.

Can I export the results to Excel, CSV or Google Sheets?

Yes. Every Website Content Crawler run stores a dataset you can export as CSV, JSON, Excel (XLSX), XML or HTML, or send to Google Sheets with the Apify integration.

More scrapers from Neuro Scraper

ActorWhat it extracts
⭐ Facebook Reels ScraperFacebook Reels with exact views, plays, every reaction type, comments and shares
Website Images ScraperWebsite images with all public fields, exported to JSON, CSV or Excel
Website SEO Audit ScraperWebsite seo audit data with all public fields, exported to JSON, CSV or Excel
Snapchat Profile ScraperSnapchat profile data with all public fields, exported to JSON, CSV or Excel
Amazon ScraperAmazon data with all public fields, exported to JSON, CSV or Excel
Instagram Reels ScraperInstagram reels with all public fields, exported to JSON, CSV or Excel
Facebook Video Details ScraperFacebook video details with all public fields, exported to JSON, CSV or Excel
Facebook Reels Scraper AdvancedFacebook reels advanced data with all public fields, exported to JSON, CSV or Excel
Facebook Reels DownloaderFacebook reels with all public fields, exported to JSON, CSV or Excel
Instagram Search ScraperInstagram search data with all public fields, exported to JSON, CSV or Excel
TikTok Video Details ScraperTikTok video details with all public fields, exported to JSON, CSV or Excel
Instagram Hashtag ScraperInstagram hashtag data with all public fields, exported to JSON, CSV or Excel
TikTok Profile ScraperTikTok profile data with all public fields, exported to JSON, CSV or Excel