# PendingDelete.Domains Crawler (`accessible_agenda/crawl-pendingdelete-domains-actor`) Actor Crawl Expired Domains From PendingDelete.Domains - **URL**: https://apify.com/accessible\_agenda/crawl-pendingdelete-domains-actor.md - **Developed by:** [sprro](https://apify.com/accessible_agenda) (community) - **Categories:** Automation, SEO tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## PendingDelete.Domains Crawler **Scrape expiring domain metrics from [PendingDelete.Domains](https://pendingdelete.domains)** — a platform that curates **pending-delete and expiring domains** with high traffic, strong domain authority, quality backlink profiles, and valuable keyword rankings. This Actor discovers the domains published each day and extracts their public SEO metrics — **global rank, monthly visits, domain authority, backlinks, and ranking keywords** — without requiring a paid subscription or login. Run it on demand or [schedule it](https://docs.apify.com/platform/schedules) to capture the daily drop list automatically. As an Apify Actor you also get [API access](https://docs.apify.com/api/v2), [integrations](https://apify.com/integrations) (Make, Zapier, Google Drive, Slack), [proxy rotation](https://docs.apify.com/platform/proxy), and run monitoring out of the box. ### Why use the PendingDelete.Domains Crawler? Expiring domains are a goldmine for **SEO practitioners, domain investors, and content site builders**. While some carry high direct traffic, many are valuable primarily for their **existing backlink profiles and keyword rankings** — making them ideal for building authority sites, 301 redirects, or PBNs. - **Daily discovery** — automatically finds every domain report published today via the site's sitemap. - **No subscription needed** — retrieves publicly accessible metrics that are normally bundled behind a paywall. - **Structured output** — clean JSON ready for spreadsheets, databases, or downstream automation. - **Hands-off automation** — schedule daily runs and pipe results into your stack. ### How it works 1. **Discover today's domains via sitemap** — the Actor fetches `https://pendingdelete.domains/sitemap.xml` and extracts the domain report URLs published that day (e.g. `https://pendingdelete.domains/domains/minicruzer.com`). 2. **Scrape each domain report page** — for every discovered URL, the Actor visits the page and parses the domain metrics: global rank, monthly visits, domain authority, backlink count, and keyword rankings. 3. **Googlebot user-agent required** — requests are sent with a `User-Agent` string containing `googlebot` to receive the full page content. Without it, the server returns restricted or empty results: ```` Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ```` ### How to use the PendingDelete.Domains Crawler 1. Click **Try for free** (or open the Actor in your Apify Console). 2. Optionally adjust the input — by default the Actor reads the daily sitemap; you can also provide specific domain report URLs. 3. Click **Start** and wait for the run to finish. 4. Open the **Output** / **Storage** tab and download the dataset as **JSON, CSV, Excel, or HTML**, or pull it via the [API](https://docs.apify.com/api/v2). ### Input Configure the run from the **Input** tab in the Apify Console or via the API. | Field | Type | Description | | --- | --- | --- | | `startUrls` | array | Domain report URLs or sitemap URL(s) to crawl. Defaults to the daily sitemap discovery. | | `maxRequestsPerCrawl` | integer | Maximum number of pages to scrape in a single run (`0` = unlimited). | | `proxyConfiguration` | object | Proxy settings used for requests. | Example input: ```json { "startUrls": [ { "url": "https://pendingdelete.domains/sitemap.xml" } ], "maxRequestsPerCrawl": 1000 } ```` ### Output Results are stored in a [dataset](https://docs.apify.com/platform/storage/dataset). You can download the dataset in various formats such as **JSON, HTML, CSV, or Excel**. Example output item: ```json { "domain": "minicruzer.com", "global_rank": 482915, "monthly_visits": 12400, "domain_authority": 34, "backlinks": 1820, "keywords": 256, "crawled_at": "2026-06-09T10:15:00.000Z" } ``` #### Data table | Field | Description | | --- | --- | | `domain` | Domain name | | `global_rank` | Global traffic rank | | `monthly_visits` | Estimated monthly visits | | `domain_authority` | Domain authority score | | `backlinks` | Number of backlinks | | `keywords` | Number of ranking keywords | | `crawled_at` | Timestamp of the crawl (ISO 8601) | ### Pricing / Cost estimation This Actor runs on the Apify platform and consumes [compute units](https://docs.apify.com/platform/actors/running/usage-and-resources) (CU) based on runtime and memory. Because it uses the lightweight [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler) (HTTP-only, no browser), a typical daily run scraping a few hundred domain pages costs only a small fraction of a compute unit. The [Apify Free tier](https://apify.com/pricing) includes monthly usage credits that comfortably cover regular runs. ### Tips & advanced options - **Schedule daily runs** to capture each day's drop list automatically — the freshest expiring domains are only listed for a short window. - **Limit `maxRequestsPerCrawl`** to cap compute usage during testing. - **Use Apify Proxy** if you encounter rate limiting. - **Be a good citizen** — keep concurrency reasonable and respect the target site's resources and Terms of Service. ### FAQ, disclaimers, and support **Is web scraping legal?** Scraping publicly available data is generally legal in many jurisdictions, but you are responsible for how you use the collected data and for complying with the target site's Terms of Service and applicable laws (including GDPR/CCPA where relevant). This Actor only retrieves publicly accessible metrics. **Why do I get empty results?** The target server only returns full content when the request includes a `googlebot` user-agent. This Actor sets it automatically; if the site changes its behavior, results may vary. **Found a bug or have a feature request?** Open an issue on the Actor's **Issues** tab. Custom or extended solutions can be arranged on request. # Actor input Schema ## `domains` (type: `array`): List of domain names to look up on PendingDelete.Domains (e.g. "minicruzer.com"). Each domain's metrics will be scraped and saved to the dataset. ## `maxRequestsPerCrawl` (type: `integer`): Maximum number of domain pages to scrape in a single run (0 = unlimited). ## Actor input object example ```json { "domains": [ "minicruzer.com" ], "maxRequestsPerCrawl": 100 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "minicruzer.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("accessible_agenda/crawl-pendingdelete-domains-actor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["minicruzer.com"] } # Run the Actor and wait for it to finish run = client.actor("accessible_agenda/crawl-pendingdelete-domains-actor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "minicruzer.com" ] }' | apify call accessible_agenda/crawl-pendingdelete-domains-actor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=accessible_agenda/crawl-pendingdelete-domains-actor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "PendingDelete.Domains Crawler", "description": "Crawl Expired Domains From PendingDelete.Domains", "version": "0.0", "x-build-id": "1IcHM53VBOGPdqxrr" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/accessible_agenda~crawl-pendingdelete-domains-actor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-accessible_agenda-crawl-pendingdelete-domains-actor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/accessible_agenda~crawl-pendingdelete-domains-actor/runs": { "post": { "operationId": "runs-sync-accessible_agenda-crawl-pendingdelete-domains-actor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/accessible_agenda~crawl-pendingdelete-domains-actor/run-sync": { "post": { "operationId": "run-sync-accessible_agenda-crawl-pendingdelete-domains-actor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "domains" ], "properties": { "domains": { "title": "Domains", "type": "array", "description": "List of domain names to look up on PendingDelete.Domains (e.g. \"minicruzer.com\"). Each domain's metrics will be scraped and saved to the dataset.", "default": [], "items": { "type": "string" } }, "maxRequestsPerCrawl": { "title": "Max Requests per Crawl", "minimum": 0, "type": "integer", "description": "Maximum number of domain pages to scrape in a single run (0 = unlimited).", "default": 100 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```