# Website Crawl Map & Sitemap Diff (`harvestlab/website-crawl-map`) Actor Crawl websites, map every discovered URL and parent link, compare XML sitemaps, and export status, redirects, canonicals, depth, indexability, CSVs, and orphan candidates. HTTP and browser rendering. Pay per result from $0.005; diagnostics are free; x402-ready. - **URL**: https://apify.com/harvestlab/website-crawl-map.md - **Developed by:** [Nick](https://apify.com/harvestlab) (community) - **Categories:** SEO tools, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 url records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README Map the public URLs on one or more websites, preserve the internal links between them, and compare the crawl with public XML sitemaps. **Website Crawl Map & Sitemap Diff** returns a migration-ready URL inventory with HTTP status, redirects, canonicals, indexability, depth, every observed parent page, anchor text, and honest sitemap-only “orphan candidate” labels. For a useful first run, enter one website, keep **Auto** rendering, set **Max URL records** to 100 and **Max pages crawled** to 50, then inspect the URL Inventory and Crawl Summary outputs before increasing the limits. ### What does Website Crawl Map & Sitemap Diff do? The Actor starts from public HTTP or HTTPS pages and follows links within the selected website scope. At the same time, it looks for public XML sitemaps, including sitemap indexes declared in `robots.txt`. It merges both sources into one clean URL inventory. Each result explains not only which URL was found, but also: - whether it came from links, a sitemap, or both; - the minimum observed click depth from the submitted start page; - every observed source page and anchor text, subject to your evidence-list limit; - whether the URL was fetched, redirected, failed, or remained sitemap-only; - its HTTP status, final URL, response type, canonical, and robots directives; - whether it looks indexable, canonicalized elsewhere, blocked by `noindex`, or not crawled within the selected page budget; - the sitemap file and `lastmod`, `changefreq`, and `priority` values when supplied; - stable issue codes for filtering and downstream automation. The Actor does not extract page-body content, log in, submit forms, or claim access to search-engine indexes. It maps public URL and link evidence. It can be called from the Apify API, run on a schedule, connected to webhooks, or used as the first step in a larger scraping or website-migration pipeline. ### Why use this website URL crawler? #### Build a defensible URL inventory A normal link crawler misses URLs that are declared only in sitemaps. A sitemap parser misses reachable pages that were never added to the sitemap. This Actor combines both and reports the evidence source for every URL. #### Preserve the internal-link graph Many URL tools keep one parent URL and discard the rest. Website Crawl Map records all unique observed source-target-anchor-rel link edges up to an explicit graph limit. The URL row contains convenient parent and anchor arrays, while `LINK_GRAPH.csv` contains one row per observed edge. #### Find honest orphan candidates A URL is labeled `orphan_candidate` only when it appears in a sitemap but no internal HTML link to it was observed during this bounded crawl. That is useful evidence, but it is not proof that the page has no inbound link anywhere. Links behind login pages, forms, unvisited pages, scripts, or a tighter crawl budget may still exist. #### Control cost and crawl size before starting The input exposes separate limits for stored URL records, crawled pages, sitemap URLs, sitemap files, graph edges, depth, links per page, redirects, retries, response size, browser fallbacks, concurrency, and wall-clock work time. You can sample a website without accidentally requesting an unbounded crawl. #### Export operational files, not only dataset rows Alongside the standard Apify dataset, the Actor can create a flat URL inventory, link edge list, sitemap-diff file, run summary, and a conservative generated XML sitemap. These files are designed for spreadsheets, SQL imports, migration workbooks, QA tools, and downstream Actors. ### What data can this website crawler extract? | Field | Type | What it tells you | | --- | --- | --- | | `url` | string | Normalized URL. This is the portfolio-standard item URL field. | | `source_classification` | string | `both`, `crawl_only`, `sitemap_only`, or `external_link`. | | `orphan_candidate` | boolean | True for internal sitemap-only URLs other than the start URL. | | `crawl_status` | string | Whether the URL was crawled, discovered, or failed. | | `depth` | integer | Minimum observed hyperlink distance from the start URL. | | `parent_urls` | array | Unique pages that linked to this URL, capped by input. | | `anchor_texts` | array | Unique observed anchor text values, capped by input. | | `inbound_link_count` | integer | Count of unique source-target-anchor-rel observations. | | `unique_parent_count` | integer | Count of distinct parent pages. | | `status_code` | integer | Final HTTP status when the URL was fetched. | | `final_url` | string | Final normalized URL after redirects. | | `redirect_chain` | array | Requested and redirected URLs observed during navigation. | | `content_type` | string | Returned response type, such as HTML or PDF. | | `response_time_ms` | integer | End-to-end response time for the navigation. | | `title` | string | Page title when HTML was parsed. | | `meta_description` | string | Public meta description when present. | | `canonical_url` | string | Resolved canonical link when present. | | `robots_meta` | string | Page-level robots directives. | | `x_robots_tag` | string | HTTP-header robots directives. | | `indexability` | string | Evidence-based status such as `indexable`, `noindex`, or `redirect`. | | `sitemap_url` | string | First sitemap file that declared the URL. | | `sitemap_lastmod` | string | Last-modified value supplied by the sitemap. | | `issue_codes` | array | Stable flags for filtering and downstream automation. | Issue codes include `orphan_candidate`, `missing_from_sitemap`, `deep_page`, `http_4xx`, `http_5xx`, `redirect`, `redirect_chain`, `redirect_off_scope`, `fetch_failed`, `response_too_large`, `non_html`, `noindex`, `canonical_missing`, `canonicalized_elsewhere`, `nofollow_only`, and `sitemap_url_not_crawled`. ### How to crawl a website and compare its sitemap 1. Open the Actor and select the **Input** tab. 2. Add one public website, page URL, or bare domain under **Website URLs**. 3. Keep **Crawl scope** set to **Same registrable domain** when the site uses useful subdomains. Choose **Same host** for a stricter inventory. 4. For a first run, set **Max URL records** to 100, **Max pages crawled per website** to 50, and **Max crawl depth** to 3. 5. Leave **Discover XML sitemaps** enabled. 6. Use **Auto** page rendering. Choose HTTP only for predictable server-rendered sites, or browser only when you know navigation links appear after scripts run. 7. Start the Actor and open **URL Records**. 8. Review `source_classification`, `indexability`, and `issue_codes` before broadening the limits. 9. Download `URL_INVENTORY.csv`, `LINK_GRAPH.csv`, or `SITEMAP_DIFF.csv` from the output links when you need a spreadsheet or migration artifact. #### Minimal API input ```json { "startUrls": [{"url": "https://example.com"}], "maxResults": 100, "maxPagesPerStartUrl": 50, "maxDepth": 3 } ``` #### Multi-site migration input ```json { "startUrls": [ {"url": "https://www.example.com"}, {"url": "https://docs.example.org"} ], "crawlScope": "same-domain", "includeUrlPatterns": ["*/products/*", "*/docs/*", "*/blog/*"], "excludeUrlPatterns": ["*/logout*", "*/cart*", "*?replytocom=*"], "queryParameterPolicy": "drop-tracking", "assetPolicy": "pages-and-documents", "discoverSitemaps": true, "renderMode": "auto", "maxResults": 5000, "maxPagesPerStartUrl": 2000, "maxDepth": 8, "generateXmlSitemap": true } ``` ### Input options See the **Input** tab for the complete form and current bounds. #### Website and URL controls | Input | Default | Guidance | | --- | --- | --- | | `startUrls` | `https://example.com` | Add 1–20 public HTTP/HTTPS seeds. | | `crawlScope` | `same-domain` | Includes subdomains that share the registrable domain. | | `includeUrlPatterns` | empty | Restricts saved URL records but allows traversal through intermediate internal pages. | | `excludeUrlPatterns` | empty | Matching URLs are neither saved nor followed. Exclusions win over inclusions. | | `queryParameterPolicy` | `drop-tracking` | Removes common analytics parameters and sorts the rest for stable deduplication. | | `assetPolicy` | `pages-and-documents` | Saves HTML-style pages and common document links, but not images/scripts/media. | Patterns use shell-style matching. `*/blog/*` matches a blog section. `*?preview=*` matches URLs containing that query form. Test filters with a small result limit before running a large crawl. #### Sitemap and rendering controls | Input | Default | Guidance | | --- | --- | --- | | `discoverSitemaps` | true | Combines declared sitemap URLs with crawl discoveries. | | `maxSitemapsPerStartUrl` | 50 | Bounds nested sitemap-index work. | | `maxSitemapUrlsPerStartUrl` | 5,000 | Bounds URL entries merged from sitemaps. | | `renderMode` | `auto` | Uses fast page requests first, then a small browser fallback for likely script shells. | | `browserFallbackMaxPages` | 20 | Prevents auto mode from silently rendering a large site. | | `followNofollow` | true | Nofollow links are always recorded; this controls whether internal ones are followed. | #### Page, graph, and time budgets `maxResults` caps stored URL records across the run; failed-fetch rows inside that cap remain uncharged. `maxPagesPerStartUrl` caps unique internal pages navigated for each seed. They are not the same: a sitemap can add discovered records without those pages being fetched, and a narrow include pattern can require traversal through pages that are not saved. `maxEdges` caps link-graph observations. Parent and anchor arrays on each URL are separately capped by `maxParentUrlsPerRecord`, while aggregate counts remain complete up to the graph-edge cap. `maxRunSeconds` is a work budget, not the container timeout. The Actor stops crawl and sitemap work before this budget ends so it can store partial useful output and create the requested files. Set the Apify run timeout higher than `maxRunSeconds`. ### Output examples You can download the dataset in formats such as JSON, HTML, CSV, XML, or Excel through Apify’s standard dataset export controls. #### URL found through both navigation and sitemap ```json { "result_type": "url_record", "diagnostic_item": false, "start_url": "https://example.com/", "url": "https://example.com/docs", "relationship": "internal", "link_type": "page", "discovery_sources": ["link", "sitemap"], "source_classification": "both", "orphan_candidate": false, "crawl_status": "crawled", "fetch_mode": "http", "depth": 1, "parent_url": "https://example.com/", "parent_urls": ["https://example.com/", "https://example.com/help"], "anchor_texts": ["Documentation", "Read the docs"], "inbound_link_count": 2, "unique_parent_count": 2, "status_code": 200, "final_url": "https://example.com/docs", "redirect_chain": ["https://example.com/docs"], "content_type": "text/html; charset=utf-8", "response_time_ms": 84, "title": "Documentation", "canonical_url": "https://example.com/docs", "indexability": "indexable", "sitemap_url": "https://example.com/sitemap.xml", "sitemap_lastmod": "2026-08-31", "issue_codes": [], "scraped_at": "2026-09-02T12:00:00Z" } ``` #### Sitemap-only orphan candidate ```json { "result_type": "url_record", "diagnostic_item": false, "start_url": "https://example.com/", "url": "https://example.com/old-campaign", "discovery_sources": ["sitemap"], "source_classification": "sitemap_only", "orphan_candidate": true, "crawl_status": "discovered", "depth": null, "inbound_link_count": 0, "indexability": "not_crawled", "issue_codes": ["orphan_candidate", "sitemap_url_not_crawled"] } ``` #### No-charge diagnostic ```json { "result_type": "diagnostic", "diagnostic_item": true, "diagnostic_type": "source_unavailable", "start_url": "https://example.invalid/", "reason": "no_chargeable_urls", "message": "No chargeable URL records were found for this start URL.", "recommended_action": "Check filters, increase limits, try browser mode, or enable an appropriate proxy." } ``` Diagnostic rows are operational evidence. They do not create `url-record` charges. ### Downloadable crawl artifacts #### `SUMMARY` JSON with duration, selected rendering mode, stored record count, charge-eligible and platform-confirmed charged events, diagnostics, classification totals, issue totals, artifact errors, and per-site page, edge, sitemap, failure, warning, and limit information. #### `URL_INVENTORY.csv` A flat version of the accepted URL dataset suitable for a migration workbook, content inventory, SQL import, or spreadsheet review. #### `LINK_GRAPH.csv` One row per observed unique link edge with start URL, source URL, target URL, anchor text, rel values, internal/external relationship, and target depth. This is the most complete output for internal-link analysis. #### `SITEMAP_DIFF.csv` A focused internal-URL export showing whether each URL appeared in the crawl, a sitemap, or both. Filter `source_classification=sitemap_only` for orphan candidates and `crawl_only` for pages that may be missing from the sitemap. #### `GENERATED_SITEMAP.xml` Optional conservative sitemap containing deduplicated internal pages that finished with HTTP 200, returned HTML, and were classified as indexable. Review it before publishing: the Actor cannot decide your site’s complete editorial or SEO policy. ### How much does it cost to crawl a website? The repository pricing source proposes pay-per-event pricing: - **URL record:** `$0.005` after one useful unique, non-failed URL row is stored— equivalent to `$5.00` per 1,000 chargeable records before any future plan-tier changes. - **Browser-rendered page:** `$0.003` after a successful non-empty rendered page is processed and represented by a stored URL row. HTTP 4xx and 5xx rows are useful audit data and can be billable URL records because they answer whether a discovered URL works. Invalid-input, blocked/empty, unsafe- destination, and other diagnostic-only rows are not charged. Failed-fetch URL rows may preserve discovery/error evidence, but create neither a `url-record` nor a browser render charge. The live Store Pricing tab is the source of truth for current prices. Set a maximum charge when starting a run if you need a hard billing ceiling. ### Tips for accurate and efficient URL maps #### Start with a sample Use 50 crawled pages, 100 URL records, depth 3, and HTTP or Auto mode. Inspect the summary for sitemap counts, browser fallbacks, failures, and limit flags. Increase one budget at a time. #### Prefer HTTP mode when it works HTTP mode is normally faster and cheaper. Auto mode is a safe default because it only escalates pages that resemble empty script-driven shells, and browser fallback has its own explicit cap. Browser-only mode is best reserved for sites whose navigation truly requires rendering. #### Separate URL inventory size from page-audit size If you need a large sitemap inventory but only a small technical sample, use a high `maxSitemapUrlsPerStartUrl` and a lower `maxPagesPerStartUrl`. Sitemap-only records will have `indexability=not_crawled` because no HTTP claim was manufactured. #### Use include patterns for focused exports An include pattern restricts saved records without necessarily blocking traversal. This allows the Actor to cross category or navigation pages on the way to matching product, blog, or documentation URLs. Use exclusions for URLs that must not be visited. #### Be deliberate with query parameters The default removes common tracking parameters while retaining functional parameters. Choose `strip-all` only when parameters never identify distinct pages on the target. Choose `keep-all` when product variants, filters, pagination, or application state in the query string must remain distinct. A poor query policy can either duplicate pages or merge legitimately different URLs. #### Understand partial output The Actor finalizes partial useful results when a page, sitemap, edge, response-size, browser, result, or run-time limit is reached. Check `SUMMARY.limits` before treating an inventory as complete. “No issue found” within a bounded crawl is not proof that no issue exists elsewhere. ### FAQ #### Can it find every URL on a website? No crawler can guarantee every URL. Pages may be private, unlinked, generated only after form submissions, blocked, outside the selected scope, or beyond a configured limit. The Actor reports its sources and truncation state so you can judge coverage. #### What is an orphan candidate? It is an internal URL declared in a parsed sitemap for which this bounded crawl did not observe an HTML link. It is a review candidate, not proof of a globally orphaned page. #### Does it check broken links? It records status and redirect evidence for URLs that were actually fetched. Sitemap- only or budget-limited discovered URLs may not have HTTP facts. Use `crawl_status` and `indexability` to distinguish audited pages from discoveries. #### Why is a reachable URL marked missing from sitemap? The Actor parsed at least one sitemap, found the URL through the start page or an HTML link, and did not see it in those parsed sitemaps. The site may publish another sitemap outside the discovery paths or beyond your sitemap limits. #### Does it obey `robots.txt`? The Actor reads public sitemap declarations from `robots.txt`; it does not use the file as an automatic crawl blocker. You are responsible for choosing lawful targets and settings and for complying with website terms and applicable rules. #### Can it crawl external websites linked from my site? No. The include-external scope saves outbound URLs as discovered evidence but does not follow them. This prevents one submitted site from expanding into an open-ended web crawl. #### Can it crawl private networks or localhost? No. Seeds, sitemap documents, queued links, and redirects are restricted to publicly resolvable Internet destinations. This safety boundary is not configurable. #### Why did Auto mode not render a page? Auto mode uses a conservative heuristic to control cost. Select browser-only mode when you know that navigation links require rendering, and set an appropriate page budget. #### Where can I report a problem? Open the Actor’s **Issues** tab and include the run ID, start URL, selected mode, and the relevant diagnostic or summary warning. Do not include credentials, cookies, or private URLs. Use the **API** tab for current code examples in Python, JavaScript, and direct HTTP requests. ### Legal and compliance This Actor collects public URLs and technical page metadata. It does not intentionally extract private user data, bypass logins, or submit forms. You are responsible for reviewing the target website’s terms, crawl policies, intellectual-property rights, and applicable laws before running it. Use conservative concurrency and delay settings and avoid disrupting target services. Public pages can still contain personal data in URLs, titles, descriptions, or anchor text. Personal data is protected by the GDPR in the European Union and by other laws around the world. Do not collect or process personal data unless you have a legitimate purpose and an appropriate legal basis. Apply retention, access-control, deletion, and data-minimization practices to exported datasets and files. If you are unsure whether your use is lawful, consult qualified legal counsel. Website Crawl Map provides technical evidence, not legal advice, an SEO guarantee, or proof of global site structure. Review output before using it to publish sitemaps, remove pages, change canonicals, or deploy redirects. # Actor input Schema ## `startUrls` (type: `array`): One or more public website, page, or bare-domain URLs. Start with one site and a small page limit. ## `startUrl` (type: `string`): Hidden single-URL alias for startUrls. ## `url` (type: `string`): Hidden single-URL alias for startUrls. ## `crawlScope` (type: `string`): Same host follows only the exact hostname. Same domain includes subdomains. Include external saves outbound links but never follows them. ## `includeUrlPatterns` (type: `array`): Optional shell-style patterns for saved records, for example */blog/*. Intermediate internal pages may still be traversed. ## `excludeUrlPatterns` (type: `array`): Shell-style patterns that are neither saved nor followed, for example */logout* or *?replytocom=*. ## `queryParameterPolicy` (type: `string`): Keep all parameters, drop common analytics/tracking parameters, or strip every query parameter before deduplication. ## `assetPolicy` (type: `string`): Choose whether the inventory contains only web pages, common documents too, or every linked asset. ## `discoverSitemaps` (type: `boolean`): Read sitemap declarations from robots.txt and conventional sitemap paths, including nested sitemap indexes. ## `maxSitemapsPerStartUrl` (type: `integer`): Maximum sitemap and sitemap-index documents fetched for one start URL. ## `maxSitemapUrlsPerStartUrl` (type: `integer`): Maximum in-scope URL entries merged from sitemaps for one start URL. ## `renderMode` (type: `string`): Auto uses fast HTTP first and renders only likely JavaScript shells. Browser renders every crawled page and costs more. ## `browserFallbackMaxPages` (type: `integer`): Maximum pages rendered after HTTP detects likely empty JavaScript shells. Ignored in browser-only mode. ## `followNofollow` (type: `boolean`): Record all nofollow links and also follow internal ones when enabled. ## `maxResults` (type: `integer`): Global maximum useful URL rows stored and charged across the run. ## `maxPagesPerStartUrl` (type: `integer`): Maximum unique internal pages navigated for each start URL. ## `maxDepth` (type: `integer`): Maximum hyperlink distance from each start URL. Set 0 to fetch only the submitted page; sitemap discovery remains independent. ## `maxLinksPerPage` (type: `integer`): Maximum distinct hyperlinks parsed from one HTML page. ## `maxEdges` (type: `integer`): Maximum unique source-target-anchor-rel observations retained for one website. ## `maxParentUrlsPerRecord` (type: `integer`): Caps parent and anchor evidence arrays. Aggregate inbound counts remain complete up to maxEdges. ## `maxConcurrency` (type: `integer`): Concurrent HTTP page requests. Browser concurrency is automatically capped at 3. ## `requestTimeoutSecs` (type: `integer`): Maximum time allowed for one HTTP or browser navigation attempt. ## `maxRedirects` (type: `integer`): Maximum manually validated redirect hops per HTTP request. ## `maxResponseBytes` (type: `integer`): Maximum page or decompressed sitemap body read into memory. ## `requestDelayMillis` (type: `integer`): Polite delay before each navigation. Concurrency can still overlap requests. ## `maxRunSeconds` (type: `integer`): Wall-clock crawl budget. The Actor stops work early enough to preserve artifact finalization time. ## `retryCount` (type: `integer`): Retries for timeouts, HTTP 429, and selected 5xx responses. ## `saveUrlInventoryCsv` (type: `boolean`): Create URL\_INVENTORY.csv in the run key-value store. ## `saveLinkGraphCsv` (type: `boolean`): Create LINK\_GRAPH.csv with one row per observed hyperlink edge. ## `saveSitemapDiffCsv` (type: `boolean`): Create SITEMAP\_DIFF.csv with sitemap-only, crawl-only, and both classifications. ## `generateXmlSitemap` (type: `boolean`): Create GENERATED\_SITEMAP.xml from internal, final-200, indexable HTML pages. ## `proxyConfiguration` (type: `object`): Optional Apify proxy settings. Start without a proxy for public sites; enable one when a site rate-limits your run. ## Actor input object example ```json { "startUrls": [ { "url": "https://example.com" } ], "crawlScope": "same-domain", "includeUrlPatterns": [], "excludeUrlPatterns": [], "queryParameterPolicy": "drop-tracking", "assetPolicy": "pages-and-documents", "discoverSitemaps": true, "maxSitemapsPerStartUrl": 50, "maxSitemapUrlsPerStartUrl": 5000, "renderMode": "auto", "browserFallbackMaxPages": 10, "followNofollow": true, "maxResults": 100, "maxPagesPerStartUrl": 50, "maxDepth": 3, "maxLinksPerPage": 500, "maxEdges": 100000, "maxParentUrlsPerRecord": 50, "maxConcurrency": 10, "requestTimeoutSecs": 30, "maxRedirects": 10, "maxResponseBytes": 5000000, "requestDelayMillis": 1000, "maxRunSeconds": 600, "retryCount": 2, "saveUrlInventoryCsv": true, "saveLinkGraphCsv": true, "saveSitemapDiffCsv": true, "generateXmlSitemap": false, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `urlRecords` (type: `string`): Unique URL audit records and no-charge diagnostics. ## `summary` (type: `string`): Global and per-site counts, limits, warnings, classifications, issues, and timing. ## `urlInventoryCsv` (type: `string`): No description ## `linkGraphCsv` (type: `string`): No description ## `sitemapDiffCsv` (type: `string`): No description ## `generatedSitemap` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://example.com" } ], "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("harvestlab/website-crawl-map").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://example.com" }], "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("harvestlab/website-crawl-map").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://example.com" } ], "proxyConfiguration": { "useApifyProxy": false } }' | apify call harvestlab/website-crawl-map --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,harvestlab/website-crawl-map" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/95MiAFXHovNYr6Unx/builds/z6CrhvDJMl8vchXKT/openapi.json