Website Crawler – Whole-Site Text, Emails & Contacts

Give it a website. Get back every page as clean Markdown, and on every page the business contacts it holds: emails, Facebook / Instagram / LinkedIn / X / YouTube / TikTok, WhatsApp, contact page, tech stack.

• 🕸️ Follows the site itself – internal links plus the sitemap, so pages that the homepage never links to are found too. Same host only (subdomains optional).
• 🤝 Polite by default – honours robots.txt, one site at a time, capped by pages and depth, a few parallel fetches. No hammering.
• ⚡ No browser – plain HTTP, hundreds of pages a minute, a fraction of the price of browser-based crawlers.
• 🧾 Honest statuses – ok, thin (client-side rendered), parked, http_error, not_html, unreachable, blocked, refused. Every URL comes back with one.
• 💵 Flat price per page. Nothing else.

---

🎯 What can you do with it?

• Feed a RAG / LLM pipeline with a whole documentation site or knowledge base.
• Find every contact point a business publishes across its site, not just the homepage.
• Audit a site – thin pages, broken links (http_error), parked domains, tech stack.
• Monitor content – schedule it weekly and diff the dataset.

Pair with Website Email & Contact Scraper + Page Content Extractor when you already have the exact URLs, and with the Google Maps actors to go from "dentists in Lisbon" to every clinic's full website.

---

▶️ How to use it

1. Paste the homepage(s) into Websites to crawl.
2. Set Max pages per site and Max link depth (defaults: 50 and 3).
3. Optionally restrict with Only crawl URLs containing (e.g. /docs/) or Skip URLs containing (e.g. /tag/).
4. Run. Download JSON / CSV / Excel, or push to your pipeline via the API.

⚙️ Input

Field	Type	Default	Notes
startUrls	list	–	Site homepages, up to 50 per run
maxPagesPerSite	integer	50	1–2,000
maxDepth	integer	3	0 = start page only
includeSubdomains	boolean	false	Follow blog.example.com etc.
includeUrlPatterns	string[]	–	Keep only URLs containing one of these
excludeUrlPatterns	string[]	–	Skip URLs containing any of these
useSitemap	boolean	true	Seed from sitemap.xml
respectRobots	boolean	true	Skip disallowed paths
outputFormat	markdown / text	markdown
maxChars	integer	50,000	Cut and marked truncated
includeContacts	boolean	true	Emails, socials, WhatsApp, contact page, tech stack
includeLinks	boolean	false	Outgoing links per page (up to 500)
concurrency	integer	5	Parallel fetches per site (1–10)

🧾 Output

One row per page – the same fields as the single-page extractor plus where it sits in the site:

{
  "site": "acme.example",
  "url": "https://acme.example/services/repair",
  "depth": 2,
  "found_on": "https://acme.example/services",
  "status": "ok",
  "http_status": 200,
  "title": "Widget repair – Acme",
  "description": "Same-day widget repair since 1999.",
  "language": "en",
  "headings": ["Widget repair", "Pricing"],
  "content": "# Widget repair\n\nSame-day repair…",
  "word_count": 512,
  "truncated": false,
  "emails": ["repairs@acme.example"],
  "socials": {"instagram": "https://www.instagram.com/acme"},
  "whatsapp": "+971501234567",
  "contact_form_url": "https://acme.example/contact",
  "tech_stack": ["wordpress", "google-analytics"],
  "fetched_at": "2026-09-06T12:00:00+00:00"
}

---

🛡️ What it deliberately does not do

• No personal data. Only role-based business mailboxes (info@, sales@…); person-named mailboxes are dropped. Nothing is collected from social profiles.
• No JavaScript rendering. Client-side apps come back thin rather than faked.
• No bypassing. A challenge or block is reported as blocked and we back off; robots.txt is honoured; private addresses are refused.
• No cross-site wandering. Only the site you gave (and its subdomains if you ask).

---

❓ FAQ

How many pages will it find? Everything reachable by links within the depth limit, plus the sitemap, up to your page cap. depth and found_on show how each page was reached.

Can I crawl only a section? Yes – includeUrlPatterns: ["/docs/"].

Why is a page thin? Fewer than ~40 words of server-rendered text – usually a React/Vue app. Contacts may still be found in its HTML.

---

🧑‍💻 Support

Issues tab, or hello@locomint.io. Part of the Locomint local business data toolkit.