# Vee Map: Sitemap URL Extractor (`veelabs/vee-map`) Actor

Give a domain, get every URL in its sitemaps with lastmod, priority and image counts. Finds sitemaps through robots.txt, follows indexes and .gz files, filters by regex. Respects robots.txt. Not a Google Maps tool. $0.40 per 1,000 URLs.

- **URL**: https://apify.com/veelabs/vee-map.md
- **Developed by:** [Hoxei R\&D](https://apify.com/veelabs) (community)
- **Categories:** SEO tools, Developer tools, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Vee Map

**Every URL a website publishes in its sitemaps, from just a domain name.**

Vee Map is a sitemap URL extractor. It is not a maps or Google Maps tool. Give it `example.com` and it locates the sitemaps, walks through all of them, and returns a de-duplicated list of URLs with `lastmod`, `changefreq`, `priority` and image counts.

### Who it is for

- SEO teams running content inventories and audits
- Anyone checking what changed on a site (`lastmod`) or what a migration missed
- Developers feeding crawlers, search indexes and RAG pipelines with a clean URL list
- Teams monitoring competitor content

### What you get: every sitemap URL with lastmod and priority (CSV, JSON, Excel)

One record per URL:

```json
{
  "url": "https://apify.com/store",
  "lastmod": "2026-10-01",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "sitemapUrl": "https://apify.com/sitemap/pages.xml",
  "domain": "apify.com"
}
```

Fields the site does not publish are `null`. A `SUMMARY` record in the key-value store gives, for each site, the number of sitemaps fetched, URLs returned, errors, and sitemaps skipped because of robots.txt (`blockedByRobots`).

### How it works

1. Reads `robots.txt` and collects its `Sitemap:` entries. If there are none, it tries `/sitemap.xml` and `/sitemap_index.xml`.
2. Follows sitemap indexes recursively. Handles plain XML, gzip (`.xml.gz`) and plain-text sitemaps.
3. Applies your regex filter, removes duplicates and returns the URLs.

It uses plain HTTP, with no browser and no proxies, and never fetches page content.

### How it is different

- **robots.txt is honoured.** `Disallow` rules for a sitemap URL are applied, and skipped files are counted in `SUMMARY`. `Crawl-delay` is respected, up to 30 seconds. The Actor identifies itself as `VeeMap`. If robots.txt cannot be fetched because of a server or network error, the site is skipped and reported, because permission cannot be verified. A missing robots.txt (404) counts as allowed.
- **No need to know where the sitemap is.** A domain is enough. You can also pass a direct sitemap URL, and if it returns nothing the Actor falls back to robots.txt discovery.
- **Stays on the target site.** Sitemaps are followed only when they are on the same site as your start URL (same host, `www.` variant or subdomain). Entries pointing to other domains are ignored and logged.
- **Safe regex filter.** `urlFilter` accepts ordinary patterns such as `/blog/` or `(en|fr)/`. Patterns that hang, like `(a+)+$`, are rejected with a clear message.
- **Failures do not stop the run.** A missing or broken sitemap is logged and skipped, and the other sites continue.
- **Size-safe.** Each file is capped at 50 MB on the wire and 50 MB after decompression, which is the sitemaps.org limit.
- **Cost control.** `maxUrlsPerSite`, `maxUrls` and `maxSitemaps` cap the run, and charging stops if you set a maximum cost.

### If you get 0 URLs

The `SUMMARY` record and the run's status message explain why, for example: the site has no sitemap (robots.txt lists none and `/sitemap.xml` does not exist; pass a direct sitemap URL), the site blocks access (HTTP 403), `robots.txt` disallows the sitemap, or your `urlFilter` matched nothing. If `robots.txt` cannot be fetched (server error, timeout, DNS), the site is skipped and the reason says which. A page that appears in several sitemaps or several inputs is returned and billed once.

### Input examples

One domain:

```json
{ "startUrls": ["example.com"] }
```

Only blog URLs from two sites, at most 5,000 each:

```json
{
  "startUrls": ["https://example.com", "https://example.org"],
  "urlFilter": "/blog/",
  "maxUrlsPerSite": 5000
}
```

A direct sitemap, with a contact in the User-Agent:

```json
{
  "startUrls": ["https://www.example.com/sitemap.xml"],
  "contact": "https://example.com/contact"
}
```

| Field | What it does |
|---|---|
| `startUrls` | Domains or direct sitemap URLs (max 100) |
| `maxUrlsPerSite` | Cap per site (default 10,000) |
| `maxUrls` | Cap for the whole run (default 100,000) |
| `urlFilter` | Optional regex a URL must match (max 200 characters; back-references such as `\1` are not allowed) |
| `maxSitemaps` | Max sitemap files per site (default 500) |
| `maxRuntimeSecs` | Stop after this many seconds (default 3,600) |
| `contact` | Optional URL or email added to the User-Agent |

### What it costs

Pay per event. You pay only for results, and the price is the same on every Apify plan.

| Event | Price |
|---|---|
| URL extracted | $0.40 per 1,000 URLs ($0.0004 each) |
| Run start | $0.005 per run |

Platform usage (compute) is included in these prices, so there is no separate bill for it. The pricing table at the top of the Actor page is Apify's standard layout and lists every plan (Free, Starter, Scale, Business). Our price is the same in every column, so you never pay more or less because of your plan.

Examples, each for one run:

| You get | Results | Run start | Total |
|---|---|---|---|
| 1,000 URLs | $0.40 | $0.405 |
| 10,000 URLs | $4.00 | $4.005 |
| 100,000 URLs | $40.00 | $40.005 |

Filters and caps (limits such as the per-run result caps and your input filters) only reduce the number of results you pay for. Set a maximum cost on the run in Apify if you want a hard spending limit.

### FAQ

**Is this for Google Maps?** No. It extracts URLs from website sitemaps.

**What if the site has no sitemap?** The site is reported in `SUMMARY` with zero URLs. Vee Map does not crawl pages to invent one.

**Does it fetch the pages themselves?** No. It reads only the sitemap and robots files that sites publish for crawlers.

**Can it read compressed sitemaps?** Yes, `.xml.gz` and gzip-encoded responses.

**Does it follow sitemap indexes?** Yes, recursively, up to `maxSitemaps` files per site.

**Why is `lastmod` empty?** The site does not publish it in that sitemap.

**Why was a site skipped?** Check `SUMMARY`: robots.txt could not be fetched, the start URL is invalid or not allowed, or robots.txt disallowed the sitemap.

**Are URLs de-duplicated?** Yes, within each site.

### Limits

- Up to 100 start URLs. `maxUrls` and `maxUrlsPerSite` up to 1,000,000. `maxSitemaps` up to 1,000.
- Only public http(s) hosts on ports 80 and 443. Private, loopback, link-local and cloud-metadata addresses are blocked, and redirects must stay on the same site.
- Fetches time out after 30 seconds, with a 250 ms delay between fetches.
- URLs longer than 2,048 characters are dropped, and only http(s) URLs are returned.
- Treat output URLs as untrusted data. `lastmod` is limited to 64 characters (a leading `=`, `@` or `+`/`-` that is not followed by a digit gets an apostrophe so spreadsheets cannot run it as a formula), and `changefreq` is one of the standard values (`always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, `never`) or `null`.
- Notes in `SUMMARY` are cut to 200 characters and contain no control characters.

### Responsible use, legal and data protection

This Actor retrieves publicly available data only. It is not affiliated with, endorsed by or sponsored by the websites you scan. All trademarks belong to their owners and are used only to describe compatibility.

Vee provides tools only; you are fully responsible for how you use them and the results. The tool is provided as is, without warranty, and Vee gives no legal advice. You are fully responsible for your use of the data: comply with the terms of service, robots.txt, copyright and database rights of the sites you target, and with all applicable law, including data protection laws (GDPR, UK GDPR and similar). Output URLs contain personal data whenever a site puts it in a URL, for example profile slugs, user names or email addresses in URLs from forums or social sites. Have a lawful basis before you store or reuse any personal data, minimise it and delete it when no longer needed, and do not use it for unsolicited contact. Do not republish copyrighted text or images without permission. Vee does not collect or keep your inputs or results; they stay in your Apify account. Vee reserves the right to change or withdraw the Actor if a site owner objects. To raise a concern, open an issue on this Actor's Issues tab on Apify.

### Support

Questions or a feature request? Open an issue on this Actor's Issues tab on Apify and we will reply there.

# Actor input Schema

## `startUrls` (type: `array`):

Domains (example.com) or direct sitemap URLs. Sitemaps are discovered via robots.txt, then /sitemap.xml. The same page is never returned twice, even if you list a site twice.

## `maxUrls` (type: `integer`):

Stop the whole run after this many URLs.

## `maxUrlsPerSite` (type: `integer`):

Per-website cap.

## `urlFilter` (type: `string`):

Only keep URLs matching this regular expression (max 200 characters), e.g. /blog/. Invalid or catastrophic patterns, and back-references such as \1, are rejected.

## `maxSitemaps` (type: `integer`):

Safety limit for huge sitemap indexes.

## `maxRuntimeSecs` (type: `integer`):

Stop gracefully after this many seconds (also stops 60 s before the platform timeout).

## `contact` (type: `string`):

Optional. Added to the User-Agent so site owners can reach you.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrls": 100,
  "maxUrlsPerSite": 100,
  "maxSitemaps": 500,
  "maxRuntimeSecs": 3600
}
```

# Actor output Schema

## `results` (type: `string`):

Every URL found in the website sitemaps, with lastmod, changefreq, priority and image counts, in the default dataset, ready to export as CSV, JSON or Excel.

## `summary` (type: `string`):

SUMMARY record with run statistics and any inputs that were skipped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxUrls": 100,
    "maxUrlsPerSite": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("veelabs/vee-map").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxUrls": 100,
    "maxUrlsPerSite": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("veelabs/vee-map").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrls": 100,
  "maxUrlsPerSite": 100
}' |
apify call veelabs/vee-map --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,veelabs/vee-map"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ehlQ3gUJDCerlKjQu/builds/Ugcb17PFB6s7U2gF0/openapi.json