Website Content Crawler
Pricing
from $1.19 / 1,000 results
Website Content Crawler
Website Content Crawler SD - Website Content Crawler is a web crawling tool that extracts the main content of every page of a website as clean Markdown and text, with title, language and metadata, for LLMs, RAG, vector databases and search indexes.
Pricing
from $1.19 / 1,000 results
Rating
0.0
(0)
Developer
Neuro Scraper
Maintained by CommunityActor stats
0
Bookmarked
1
Total users
0
Monthly active users
5 days ago
Last modified
Categories
Features
Share
Website Content Crawler — clean Markdown and text from whole websites
The Website Content Crawler collects public Website content data and returns it as a clean dataset you can download as JSON, CSV or Excel, or read through the Apify API. No login, no cookies and no code: give the Website Content Crawler your input, start it and get structured Website data for research, monitoring, analytics or lead generation. You pay only for the results you receive.
⭐ Featured by Neuro Scraper: Facebook Reels Scraper - scrape Facebook Reels from any page, profile or reel link with exact views and plays, every reaction type (Like, Love, Care, Haha, Wow, Sad, Angry), comments, shares, captions, music and video links. No login and no cookies. Try the Facebook Reels Scraper →
Website Content Crawler crawls a website from one or more start URLs and saves every page's main content as Markdown and plain text, with title, description, language, canonical URL, author, dates and metadata. Menus, footers and cookie banners are removed, so the output is ready for LLMs, RAG pipelines, vector databases and search indexes.
Input
| Field | Type | Default | Meaning |
|---|---|---|---|
startUrls | URL list | crawlee.dev/python/docs/guides (prefill) | Where the crawl starts |
maxPages | integer | 20 | Pages saved (one row each) |
maxCrawlDepth | integer | 5 | Links away from a start URL; 0 = start URLs only |
crawlScope | select | startUrlPath | startUrlPath (URLs starting with the start URL), sameDomain, sameDomainAndSubdomains |
includeUrlGlobs / excludeUrlGlobs | glob lists | – | e.g. https://site.com/docs/**; include globs replace the scope |
useSitemaps | boolean | false | Also queue in-scope URLs from robots.txt sitemaps / /sitemap.xml |
respectRobotsTxt | boolean | true | Skip disallowed pages, honour Crawl-delay (max 10 s) |
htmlTransformer | select | trafilatura | trafilatura, readability, or none (whole cleaned page) |
saveMarkdown / saveText / saveHtml | booleans | true / true / false | Output formats |
keepMarkdownLinks | boolean | false | Keep [text](url) links in the Markdown |
requestDelayMs | integer | 1000 | Pause between requests to one host |
maxConcurrency | integer | 2 | Pages processed at once across hosts |
proxyConfiguration | proxy | Apify datacenter | Use RESIDENTIAL for sites that block datacenter IPs |
Output (one row per page)
| Field | Meaning |
|---|---|
url, requestUrl, startUrl, referrerUrl, depth, statusCode | Where the page came from |
title, description, language, canonicalUrl, author, publishedAt | Page metadata |
markdown, text, html | Main content in the formats you chose |
wordCount, extractor | Words in the main content; which extractor produced it |
likelyJsRendered | True when the server sent almost no text (page builds itself with JavaScript) |
truncated | True if a field was cut at 1.5 M characters |
metadata | keywords, robots, modifiedAt, Open Graph, Twitter card, JSON-LD types, hreflang, content type, response time, size |
crawledAt | ISO time (UTC) |
Measured (2026-10-03, see TEST_REPORT.md)
- crawlee.dev guides: 20 pages in 22 s; with sitemaps + globs: 60 pages in 67 s (59 found through the sitemap).
- books.toscrape.com: 12 pages in 12 s;
/and/index.htmlsaved once (content de-duplication). - About 1 page per second per site with the default delay.
Limits
- No browser: pages that render their text with JavaScript come out (nearly) empty and are flagged
likelyJsRendered. - PDFs, images and other non-HTML files are skipped. No login or cookies.
- Default scope follows only URLs that start with the start URL — start from a section or the homepage, or pick
sameDomain. - Sites behind anti-bot challenges (Cloudflare, DataDome, ...) are logged as blocked and skipped; try a RESIDENTIAL proxy.
Example
{"startUrls": [{"url": "https://crawlee.dev/python/docs/guides"}], "maxPages": 50, "useSitemaps": true,"excludeUrlGlobs": ["**/docs/next/**"]}
Why use the Website Content Crawler
The Website Content Crawler is a Website scraper and Website data extractor for anyone who needs public Website content data as clean data. The Website Content Crawler handles web scraping, proxy rotation and retries for you, so you get a ready dataset instead of raw pages. Use the Website Content Crawler for market research, competitor analysis, lead generation, analytics or monitoring, and export the results to JSON, CSV, Excel or Google Sheets.
- No login and no code: the Website Content Crawler runs on Apify with one click or one API call.
- Pay per result: the Website Content Crawler charges only for rows delivered - no monthly Website API fee.
- Complete fields: the Website Content Crawler returns every public field, with a stable schema for automation.
- Integrations: connect the Website Content Crawler to Zapier, Make, n8n, Google Sheets, a webhook or your CRM.
Website Content Crawler use cases
- Market research: the Website Content Crawler collects Website content data at scale, so you can compare trends across weeks and niches.
- Competitor analysis: run the Website Content Crawler on a schedule and export the dataset to Google Sheets or Excel to watch what competitors publish.
- Lead generation and B2B prospecting: the Website Content Crawler gives a sales or agency team a clean list to load into a CRM.
- Content and social media monitoring: the Website Content Crawler keeps a fresh copy of Website data through an Apify schedule and a webhook.
- Analytics and dashboards: send the Website Content Crawler output as JSON through the Apify API into Python, Node.js or a BI tool.
- Academic and data science projects: the Website Content Crawler returns a consistent schema, ready for pandas, R or a notebook.
- Price, product and brand tracking: the Website Content Crawler turns public Website data into a dataset you can diff run by run.
- Automation workflows: connect the Website Content Crawler to Zapier, Make or n8n so every new row starts your next step.
How to run it with the API, Python and Node.js
Start the Website Content Crawler from the Apify Console, or call it from code. The Website Content Crawler returns its dataset as JSON, CSV or Excel. With the Apify API you can run the Website Content Crawler from Python or Node.js, wait for it to finish and read the rows, which makes the Website Content Crawler easy to plug into an automation, a data pipeline or a scheduled export.
- Open the Website Content Crawler on Apify and fill in the input (or send the same input as JSON to the API).
- Start the Website Content Crawler. Follow the run log while the Website Content Crawler collects public Website data.
- Download the dataset as CSV, JSON, Excel or XLSX, or let a webhook send it on.
- Schedule the Website Content Crawler to keep your Website data fresh for monitoring and analytics.
Tips for better results
- Start small: run the Website Content Crawler with a low maximum first, check the fields, then scale up the same input.
- Split big jobs: several smaller Website Content Crawler runs (by keyword, URL list or date) finish faster and are easier to resume.
- Keep a history: name a dataset or use a schedule so each Website Content Crawler run adds to your Website data over time.
- Filter early: use the input filters of the Website Content Crawler so you only pay for the Website content data you really need.
- Automate the hand-off: a webhook or the Zapier and Make integrations can move every Website Content Crawler result straight to Google Sheets, a CRM or a database.
- Watch the run log: the Website Content Crawler writes what it is doing and why an item was skipped, so a surprising result is easy to explain.
Website Content Crawler FAQ
Can it replace the official Website API?
For public data, yes. The Website Content Crawler gives you Website data through the Apify API without an official Website API key, quota approval or developer account.
Which formats can I download?
The Website Content Crawler dataset downloads as JSON, CSV, Excel (XLSX), XML, RSS or an HTML table, and the Apify API returns the same Website Content Crawler data page by page.
What is the best Website scraper for content data?
The Website Content Crawler is built for exactly this job: public Website content data, complete fields, no login, pay per result and a clean export to JSON, CSV or Excel.
Can I schedule it?
Yes. Apify schedules run the Website Content Crawler hourly, daily or weekly, and a webhook can notify your system when new Website data is ready.
How much does it cost?
The Website Content Crawler is pay per result: you pay only for the rows you receive, with lower prices on higher Apify plans. You can cap the cost of every run with the maximum cost setting.
How is this different from other Website scrapers?
The Website Content Crawler combines complete public fields, no login, pay-per-result pricing and a stable output schema, so the Website Content Crawler fits both quick one-off exports and long-running automation.
Can I run it for many inputs at once?
Yes. Give the Website Content Crawler a list of inputs in one run; each row says which input it came from, so one Website Content Crawler run can cover a whole list.
Does it work with proxies?
Yes. The Website Content Crawler uses Apify Proxy by default and you can pick the proxy group or country in the input.
Is it legal to scrape Website?
The Website Content Crawler collects only publicly available data. If results contain personal data, process it under GDPR or the rules that apply to you, and only for a legitimate purpose.
How do I get more results?
Raise the maximum items, add more inputs (keywords, URLs or profiles), or run the Website Content Crawler several times on a schedule. Results are de-duplicated.
How fast is it?
The Website Content Crawler runs requests in parallel and rotates proxy sessions automatically, so a typical run of a few hundred items finishes in minutes.
Do you offer custom scrapers?
Yes. If you need a field or a filter the Website Content Crawler does not have yet, contact Neuro Scraper through the Apify Store page and we can extend the Website Content Crawler for you.
What happens if a run is interrupted?
The Website Content Crawler saves its progress, so a resurrected or restarted run continues where it stopped instead of paying for the same rows again.
Can I use it through the API?
Yes. Start the Website Content Crawler and read its dataset through the Apify API from Python, Node.js or any HTTP client, or call it from Zapier, Make and n8n.
Can I send the data to my CRM?
Yes. Push Website Content Crawler rows to HubSpot, Salesforce, Pipedrive or any CRM through Zapier, Make or a webhook - a common setup for an agency or B2B sales team.
Do I need a Website account or login?
No. The Website Content Crawler works with no login and no cookies. It reads only public data, the same data any visitor can see.
Why is a field empty in my results?
The Website Content Crawler fills every field Website publishes for that item. When Website does not show a value - for example a figure the owner hid - the field stays empty rather than being guessed.
What does this Website scraper do?
The Website Content Crawler collects public Website content data and returns it as a structured dataset. It is web scraping without code: you set the input, start the Website Content Crawler and download JSON, CSV, Excel or XLSX.
Can I export the results to Excel, CSV or Google Sheets?
Yes. Every Website Content Crawler run stores a dataset you can export as CSV, JSON, Excel (XLSX), XML or HTML, or send to Google Sheets with the Apify integration.
More scrapers from Neuro Scraper
| Actor | What it extracts |
|---|---|
| ⭐ Facebook Reels Scraper | Facebook Reels with exact views, plays, every reaction type, comments and shares |
| Website Images Scraper | Website images with all public fields, exported to JSON, CSV or Excel |
| Website SEO Audit Scraper | Website seo audit data with all public fields, exported to JSON, CSV or Excel |
| Snapchat Profile Scraper | Snapchat profile data with all public fields, exported to JSON, CSV or Excel |
| Amazon Scraper | Amazon data with all public fields, exported to JSON, CSV or Excel |
| Instagram Reels Scraper | Instagram reels with all public fields, exported to JSON, CSV or Excel |
| Facebook Video Details Scraper | Facebook video details with all public fields, exported to JSON, CSV or Excel |
| Facebook Reels Scraper Advanced | Facebook reels advanced data with all public fields, exported to JSON, CSV or Excel |
| Facebook Reels Downloader | Facebook reels with all public fields, exported to JSON, CSV or Excel |
| Instagram Search Scraper | Instagram search data with all public fields, exported to JSON, CSV or Excel |
| TikTok Video Details Scraper | TikTok video details with all public fields, exported to JSON, CSV or Excel |
| Instagram Hashtag Scraper | Instagram hashtag data with all public fields, exported to JSON, CSV or Excel |
| TikTok Profile Scraper | TikTok profile data with all public fields, exported to JSON, CSV or Excel |