Site Crawler for Links and Tables

Give a start address and get every page of the site back as a structured row: internal and external links, file download links, every HTML table as rows, and every list. No code to write, no AI, no browser, no login. The same page gives the same output on every run. You pay per page read, and Mamba Labs pays the platform costs.

Features

• Whole site crawl. Follows links breadth first and also reads /sitemap.xml, which finds pages no link reaches. Stops at Maximum pages and Maximum depth.
• Download links, external ones included. Every link to a file type you list, as a field on the page row and, with download_links in extract, as one row per file (a small per row price), once per start address, with the first page it was found on and its link text. Files are listed, never downloaded.
• Tables as rows. Each HTML table comes back with its column names and rows as objects. Merged cells are filled in. The header rule is fixed: the rows in thead; otherwise the leading rows of all th cells; otherwise columns named column_1, column_2. Several header rows join per column. Tables with under 2 rows or 2 columns are treated as layout and left out.
• Lists as arrays. Each list with 2 or more items, outside nav, header, and footer.
• robots.txt obeyed by default. Disallowed pages are skipped and counted, and a crawl delay is honored up to 5 seconds.
• One free summary row per start address. Pages read, failed, refused, skipped, and left in the queue, so you can see what the crawl did not reach.
• Restart aware. If Apify moves the run to another host, the crawl resumes from its saved state instead of starting over. A restart with no warning can repeat up to the last 10 pages read.
• Cost guards. Maximum pages and a maximum charge per run.

How the data is read

Plain HTTP requests through the Apify proxy pool, parsed from the HTML the server returns. No login, no key, no AI, and no other developer's actor. With proxy_mode auto or residential, refused pages can use residential addresses at a higher page price.

Input

Field	Type	Default	What it does
start_urls	array		Web addresses to start from, up to 100. A bare domain such as example.com is read as https.
max_pages	string	100	Pages to read across the whole run, up to 10,000. Default 100.
max_depth	string	10	How many links to follow from a start address. 0 reads only the start address. Default 10.
same_domain_only	string	yes	yes (default): follow links only on the start address's host and its subdomains. no: also follow links to other sites, up to Maximum pages. Links to other sites are listed either way.
extract	array		Any of links, tables, lists, download_links. Default links, tables, lists. download_links adds one separate row per file link, charged as download-link-row; the download_links field on page rows is always filled at no extra charge.
download_extensions	array		File extensions that count as download links. Default pdf, zip, xlsx, xls, docx, doc, pptx, csv, mp3, mp4, exe, dmg, msi, pkg, gz, tar, tgz, 7z, rar, iso, apk. Files are listed, never downloaded.
use_sitemap	string	yes	yes (default): also queue the addresses in /sitemap.xml, which finds pages no link reaches. no: follow links only.
obey_robots	string	yes	yes (default): skip pages robots.txt disallows and honor its crawl delay up to 5 seconds. no: read them, only if you have the right to.
max_charge_usd	string		Optional. The run stops before a charge that would pass this amount. Summary rows are always returned.
proxy_mode	string	pool	pool (default): the Apify proxy pool only, charged as page-row; a page that refuses it gets a free blocked row. auto: a refused page is retried on residential addresses, and a page read that way is charged as residential-page-row. residential: every page uses residential addresses and is charged as residential-page-row.

Output

Field	Type	What it holds
row_type	string	One of: page, download_link, crawl_summary, run_summary. run_summary is the one free row written when the input is refused.
start_url	string	The start address this row came from.
url	string	The page address that was requested.
final_url	string	The address after redirects.
http_status	integer	The status the page answered with.
depth	integer	Links followed from the start address. Pages found in the sitemap read 1.
found_from	string	The page that linked here, or sitemap.
title	string	The page title.
internal_links	array	Links to the same site, with duplicates removed, up to 5,000.
external_links	array	Links to other sites, with duplicates removed, up to 5,000.
download_links	array	Links whose file extension is in download_extensions, internal and external.
tables	array	Each HTML table: caption, header_detected, columns, row_count, and rows as objects keyed by column name, up to 2,000 rows a table.
lists	array	Each list outside nav, header, and footer with 2 or more items, as an array of strings.
likely_needs_javascript	boolean	True when the page has under 200 characters of text and has scripts. This actor does not run a browser.
link_url	string	download_link rows: the file address.
link_text	string	download_link rows: the text of the link.
file_extension	string	download_link rows: the file extension.
link_scope	string	download_link rows: internal or external to the site.
pages_requested	integer	crawl_summary: pages requested for this start address.
pages_charged	integer	crawl_summary: page rows charged.
pages_failed	integer	crawl_summary: free rows for an HTTP error or a response that is not HTML.
pages_blocked	integer	crawl_summary: free rows for pages the site refused.
pages_skipped_robots	integer	crawl_summary: pages not requested because robots.txt disallows them.
pages_from_sitemap	integer	crawl_summary: addresses added from the sitemap.
pages_left_in_queue	integer	crawl_summary: pages found but not read when the run stopped.
download_links_found	integer	crawl_summary: download_link rows written.
row_status	string	ok, or why this row carries no data: no_results, no_change, partial, blocked, or error.
error_reason	string	Why the row is not ok. A refused input reads input_invalid: followed by what to fix.
billable_events_json	string	What this row was billed as, for example {"page-row":1}. {} means free.
scraped_at	string	When the row was written, UTC.

Pricing

Pay per event. You pay for pages read, separate download link rows if you ask for them, and the run start. Pages that failed or refused, summary rows, pages skipped by robots.txt, and a refused input are free.

Event	Free plan	Bronze	Silver	Gold
page-row, one page read on the proxy pool	$0.006	$0.0057	$0.0054	$0.0051
residential-page-row, one page read on residential addresses, only with proxy_mode auto or residential	$0.02	$0.019	$0.018	$0.017
download-link-row, one separate file link row, only when extract includes download_links	$0.0001	$0.000095	$0.00009	$0.000085
apify-actor-start, once per run (one per GB of memory; the default 256 MB is one)	$0.0055	$0.0055	$0.0055	$0.0055

Per 1,000 rows of page-row: $6.00 on the free plan, $5.10 on Gold.

Free plan cap: 1.50 USD of evaluation a month, about 250 rows. Any paid Apify plan removes the cap, and it resets on the 1st. The run stops before a charge that passes your maximum charge per run in Apify or your max_charge_usd input.

Mamba Labs pays the platform costs of your run, Apify proxy traffic included.

Usage examples

All external download links on a site.

{
  "start_urls": ["https://www.python.org/downloads/"],
  "extract": ["download_links"],
  "max_pages": "200"
}

Open the Downloads view of the dataset, and filter link_scope to external if you want only files on other hosts.

Tables and lists from a few pages.

{
  "start_urls": ["https://webscraper.io/test-sites/tables"],
  "extract": ["tables", "lists"],
  "max_depth": "1",
  "max_pages": "10"
}

Read the rows with GET https://api.apify.com/v2/datasets/{datasetId}/items.

Clay. Number and on or off inputs are text fields, which is what Clay sends. A list input takes a JSON list, so put the column inside a one item list.

MCP. The Mamba Labs MCP server @mambalabsdev/mcp-site-crawler-links-tables exposes this actor to Claude and other MCP clients.

Error handling

Case	What you get
Missing or unreadable input	One free row, row_type run_summary, error_reason starting input_invalid: with what to fix. The run ends successfully and charges nothing but the run start
An HTTP error or a response that is not HTML	A free page row with row_status error or no_results
A page refuses the request	Retried on a new address, then a free page row with row_status blocked. After 10 refused pages in a row the crawl of that site stops
robots.txt disallows the start address	A free summary row with row_status blocked that says so
The run is close to its timeout	The actor stops starting new work 90 seconds before the timeout and delivers what it has

Limitations

• No browser. A page built by JavaScript in the visitor's browser comes back with little or nothing, marked likely_needs_javascript true. Use a browser based crawler for those sites.
• Sites behind an anti bot wall (for example a Cloudflare challenge) can refuse the proxy pool. They return free blocked rows; proxy_mode auto may get through at the residential price, and may not.
• Forms, logins, and pages reached only through a site's search box are not crawled.
• Limits per page row: 5,000 links in each link field, 50 tables, 2,000 rows a table (row_count gives the real count), and 100 lists of 500 items. A row near the Apify 9 MB limit drops lists first, then tables, then links, and says so in error_reason.
• A run queues up to 5 times Maximum pages plus 1,000 addresses, at most 50,000.
• Subdomains of the start host count as the same site.
• A link is called a download by its file extension. A file served from an address with no extension is not listed as a download.

Mamba Labs actor suite

Actor	Actor ID
Account Hiring Watch	gRAZA5WBi1r6L0gy7
Agent Accessibility Auditor	anxbRv0lKrpQ1pnua
Agent to Brokerage Resolver	ePRRGLxOGyaVHO4dI
AI Answer Visibility Tracker	RJZvzKniYK4nswcMg
AI Tooling Detector	EwkHhmqiuJgRoVEbE
API Deprecation and Breaking Change Monitor	pHsbS3xXUK8Jqutz5
App Store and Google Play Review Monitor	MaCMEhGrbchxQZ92G
B2B Reddit Conversation Monitor	SoDryzzsjmY3SsiyG
Bluesky Brand Presence Mapper	eLpxzP4IuXznlFVND
Brand Sponsor Finder	PPgK6xrgOmvL7Nrys
Bulk WHOIS and RDAP Domain Age Lookup	MTrkG5FLU6SjqLdJY
Career Change Signal Engine	vX2Yga1a6wGaXaP14
Company Careers Page Jobs, 17 ATS Boards	izP0rfIgf5YaRZfIh
Company Change Event Feed	oX44rS0fkEJ3rXLWe
Company Discovery List Builder	jch0HdaZnpbMqlcGS
Company Firmographic Enricher	YlUtLWjfPpqykmB8g
Company Funding Record	OFS4Mt1gyYNtGfUbE
Company Headcount Estimator	aHC4GROnNt6kfZe5r
Company Identity Resolver	lr8fTRAmZCBZmuwwh
Company Role Email, Phone and Address Finder	4mMncKaJykiq94grz
Company Social Presence Mapper	4k6CCemkgBDz18m2h
Competitor Price, Stock, and Changelog Monitor	yxdgCbUQ4PYQWwRNU
Contact Classifier	0lGSeYJmniXhGANnO
Contractor License Search, CA, TX, WA, OR, LA	WBYK3CPI2kdeq4bP1
CRM Contact Job Change Checker	L8UbEmMJC1XYPSZOb
Domain Deliverability Checker	0tVgxI7A6o9jMlxmc
Domain to LinkedIn URL Resolver	3HtnSaqPHOg1Qg5gx
Ecommerce Platform Profiler	5G2NeYjBbQTjaTn9L
Ecommerce Store Discovery	PeHu4sLyFvQHXUUrX
Email List Verifier, Bring Your Own Key	CzVJ4NlaCKk80xpgE
Event Presence Index	WLhMy8fMDgsxdYxv5
Expert Witness Finder, US and Australia	LIpwlgfE3LaQBaXd4
Funding and Press Signal Scanner	FS13X6dhQVgX3XOM6
GDPR Cookie Banner and Consent Checker	TrKyOS0KBoGC35nbr
GitHub Organization Signal Scanner	ahF8RRpv5mzPao5CN
Google Ads Transparency Center Scraper by Domain	1eOwxhBVxzvOXAm4s
GTM Hiring Signal Scraper	D7O1SA2EqwHGsGr1P
GTM Signals Aggregator	xKdRfnfFNkdMpFuNs
ICP Fit Scorer	W161DT8W4kW55dMFh
Influencer Change Monitor	d2VVgahNL6UmcLkhg
Influencer Lead List Builder	KnmByszcv135yM30G
Influencer Profile Scraper	EqEnklDVMIcB1V499
Influencer Search Across TikTok, Instagram and YouTube	tpkedmloIIWtXx6sg
Influencer Talent Agency Lookup	zCuX4Mgyg6JvXgGzd
Instagram Facebook Brand Presence Mapper	HXh65I0JBjPsunbY8
Integration and Marketplace Presence Mapper	V0guNuvM85Q61f2gj
Job Board Keyword Signal Scanner	4DvqpvhMR74NLcDDY
Job Posting Monitor	QCfICD9kOPiOdj5iI
Journalist and Media List Builder	mAfizsZC8qRoR0ecS
Launch Directory List Builder	cRq3bmvEZxZtioQ7U
Lead File Import Checker	RwlGVXTf46LqZdUgt
Legal Entity Resolver	KHFyPCDIx7CyqULYm
Link in Bio Scraper and Newsletter Detector	OorucdheTIgu7RFzK
LinkedIn Company Page Mapper	zOPm6A9JEhUGcIdFW
LinkedIn Jobs Feed	SH6N0d8cz46BhCC8B
LinkedIn Post Tracker	oiGLNPuaf5BRaz9K5
Local Business Lead Builder	qMN97aoHhsAS5bznV
Local Rank and Map Pack Tracker	gv0kyHGDIg8XscclB
Lookalike Companies from Seed Domains	e2O426WBAbLLQL4Z2
Manufacturer and Supplier Finder	Qcsdmd0lUfPzVkonh
Meta Ad Library Ads by Company, Graph API	J1GWlSXfSGU3u8Wng
Mobile Phone Waterfall	6OOgPktf7RIa5XCOP
New Hire Finder	cKuc38p4nYHNF3IW8
Newly Registered Companies, UK, Norway, Finland, France	g2WfZhWvCtc57g2Jo
News and Media Mention Monitor	gol1PPUKrsIk4QLBd
Newsletter Sponsor Tracker	qSwpqxD1TiPZpLQHv
Open Tender and RFP Finder	kPTGz6t6TFoUUyfaO
Outbound Stack Detector	v43UJC8r7qW7cBSTG
Page Finder and Extractor	TpurgcOZbVnknlaiC
Partner and Member Directory Extractor	ZX0XJPPUvnFJxeC1a
People Finder & Email Verifier	LEgloG7tWgQl4o6TD
Person LinkedIn URL Resolver	ssuyPV8LnEsL8qbnb
Person Social Footprint Mapper	jfQXa2zW9FHCJot9K
Personal Email to Company Resolver	AefT5Ye6YNPFQwJBf
Pinterest Brand Presence Mapper	d1HvjZfKhgvWsoEwn
Podcast Guest and Host Finder	o2MGSvgbSLeeXOsIo
Product Hunt Daily Launches with Maker Domains	O4VX63drWK9Y9x641
Prospect Engine	q9rT6v64nUJdPexOQ
Public Feature Board and Roadmap Monitor	rcHF5gj5r2Vqnry35
Publishing Frequency Tracker	TbLwaUUATdYb6wp4N
Recent Funding Rounds Feed	KueVp9MOz7iOoAgDp
Reddit Brand Presence Mapper	pKQZmDaT70SEb54bZ
Reddit Search and Thread Scraper	4Gsth5cP2Xt9V2xDN
Reporting Window Finder	uINxR7a1IW8qUTRUX
Sequencer Event Poller	S6s8Rjd61tGdrMRk4
Sequencer Lead Push	0Jv27VeWM5tSZQs9x
Shopify App Store and Chrome Web Store Listings	gaY0BzkLyN77mD3wI
Substack Newsletter and Posts Scraper	UEFsdwiPLmPNVy9xd
Team Page People Extractor	j8a9zX0k4mNbvDbWK
Tech Stack Signal Detector	qyd7nNyqFPelQViBx
Threads Posts by Keyword, Exact Match	2OT8K8RiEEWtaj2MW
TikTok Brand Presence Mapper	fco8FvoJcSafbeyd1
Trustpilot Reputation Enricher	3lIqfCt61j4hqE1xz
US and UK Contract Award Winners by Company	zhEtllASykOcx9hJ8
VC and Accelerator Portfolio List Builder	chH8hrfuoqqolW06m
Website Accessibility Checker, WCAG 2.2 AA	P3jBj3FRghKEhX52z
Website Phrase and Business Model Classifier	CBuZ9k7sRwBeiKiCS
Website Traffic Rank Estimator	GnOFUzlNVIoXgeJjH
Work Email Waterfall	OT6xqTFThC0Rjf7Zj
Workplace Program Detector	Li6pPDhqbdzFz3h7G
X Brand Presence Mapper	oLOgadhHUkNAyDA3w
YouTube Channel from Company Domain	6hSpD3kd2s2IQ4VSs

Built by Mamba Labs.