# Dodle Spider (`dodle/dodle-spider`) Actor

Collects and analyzes data from multiple web sources.

- **URL**: https://apify.com/dodle/dodle-spider.md
- **Developed by:** [dodle](https://apify.com/dodle) (community)
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Multi-Source Web Intelligence

Climb Web crawls a website once to report SEO issues, extract image links, and find Wikipedia pageview trends related to the site's page titles, descriptions, and headings. Each fetched page is written as its own `crawl_result` dataset item. Site summaries and Wikipedia trends are emitted once as separate dataset items.

| Mode | What it does |
| --- | --- |
| `climb_web` | Crawl pages on the same hostname, report SEO metadata and issues, extract image links, and include Wikipedia trends whose article titles appear in crawled page titles, descriptions, or headings. |

This is a lightweight audit of server-returned HTML, not a JavaScript-rendering crawler or a full technical SEO platform. It does not measure rankings, backlinks, Core Web Vitals, or browser-rendered content.

### Quick start

#### Crawl a website

```bash
apify run --input '{"source_mode":"climb_web","start_urls":[{"url":"https://example.com/"}],"max_pages":25}'
```

The crawl follows links on the same hostname, subject to the site's `robots.txt` and configured URL filters. It returns unique absolute HTTP(S) image URLs found in common image and lazy-load attributes, `srcset`, and Open Graph or Twitter image metadata; it does not download image files. Image records include source attributes, available alt/title/dimension attributes, and whether an `img` element lacks an `alt` attribute. An empty alt attribute is treated as present, since it may mark a decorative image.

Each successful page result includes its image links and one `image_summary`, including when the page has no images. The summary reports the unique image count, counts images with or missing an `img` alt attribute, identifies metadata or `<source>` image URLs without an applicable alt attribute, and lists the URLs missing alt attributes. Optional image attributes that were not found are omitted.

### Website SEO audit

The Actor uses Crawlee and Beautiful Soup to fetch server-rendered HTML and follow links on the same hostname. Each start URL gets an independent crawl of up to 25 pages by default (maximum 1,000). Pages are processed one at a time by default, with a limit of 10 crawler tasks per minute. SEO and image records are produced during this same crawl; external links are not followed.

The page limit counts pages processed, not all HTTP requests. Retries and `robots.txt` fetches may add requests. JavaScript is not run, so content added only by client-side scripts may be missing.

Each `crawl_result` contains exactly one fetched page. Each `site_summary` contains run-level crawl totals, HTTP errors, missing metadata counts, and duplicate title and description groups for one start URL. Duplicate checks include successful pages only and ignore case and surrounding whitespace. Matching Wikipedia trends are emitted as individual `wikipedia_trend` records.

Set `include_archive_snapshot` to look up the latest successful Internet Archive snapshot for each page. This adds one request per page. The lookup is disabled by default; `archive_snapshot_status` is `found`, `not_found`, `error`, or `disabled`. Lookup errors do not stop the crawl.

### Wikipedia trends

The Actor checks Wikimedia's `robots.txt` before requesting each REST API endpoint and skips endpoints that are disallowed. It retrieves rankings for previous complete UTC days, then keeps only articles whose normalized title phrase appears in a crawled page title, meta description, or H1-H3 heading. Matching ignores case and punctuation. It does not infer broader semantic relationships, so related articles without a phrase match are omitted.

- `worldwide` returns the top five articles separately from English (`en`), Japanese (`ja`), Chinese (`zh`), Thai (`th`), Korean (`ko`), French (`fr`), and German (`de`). This is not one combined global ranking: each rank and pageview count belongs to its language edition. Each record includes `language_code`; at most 35 trend records are returned per day.
- `country` returns up to `max_rank` articles for the selected reader country. Each record includes its ISO 3166-1 alpha-2 `country_code`; Wikimedia rounds country pageview counts up.

The Actor fetches one extra, earlier UTC day as a comparison baseline but does not emit records for it. Therefore a one-day run can include day-over-day changes. Worldwide mode makes one sequential request per language edition per date, including the baseline. Requests have a two-second minimum pause; rate-limit and transient failures are retried, using Wikimedia's `Retry-After` delay when provided. Editions or countries that return 404 are skipped with a warning. A positive `rank_change` means the article climbed.

Each trend entry includes `comparison_date` and `comparison_status`. `compared` means the article was ranked on both dates and change values are populated. `entered_top_n` means prior data was available, but the article was outside the previous day's returned ranking; it does not imply the article is new. `prior_data_unavailable` means the previous date could not be fetched, for example because it was disallowed by `robots.txt` or returned 404. Change fields remain `null` unless the status is `compared`.

### Input reference

The defaults below apply when a field is omitted.

| Field | Default / limits | Used for | Description |
| --- | --- | --- | --- |
| `source_mode` | `climb_web` | All | The only supported mode; runs the site crawl, SEO audit, image extraction, and related Wikipedia trend check. |
| `start_urls` | `[]` | All | Full URLs to crawl. At least one URL is required. |
| `max_pages` | `25`; 1–1,000 | Crawl | Maximum pages processed from each start URL. |
| `include_url_patterns` | `[]` | Crawl | Optional Python regular expressions. If set, a discovered absolute URL must match at least one pattern to be followed. |
| `exclude_url_patterns` | `[]` | Crawl | Optional Python regular expressions; matching discovered URLs are skipped. |
| `max_concurrency` | `1`; 1–5 | Crawl | Maximum number of pages fetched simultaneously. |
| `max_requests_per_minute` | `10`; 10–60 | Crawl | Crawlee task-rate limit for page fetches, not a strict limit on all HTTP requests. |
| `include_archive_snapshot` | `false` | Crawl | Look up the latest successful Internet Archive snapshot for every audited page; adds one request per page. |
| `trend_region` | `worldwide` | Wikipedia | `worldwide` returns separate top-five rankings for seven language editions, not one combined ranking; `country` returns rankings for one reader country. |
| `country_code` | `US` | Wikipedia | Country selected by ISO 3166-1 alpha-2 code; used only with `trend_region: "country"`. |
| `max_rank` | `10`; 1–1,000 | Wikipedia | Maximum country ranking. Worldwide mode always returns ranks 1–5 per edition. |
| `lookback_days` | `1`; 1–30 | Wikipedia | Number of complete UTC days to return. One additional earlier day is fetched as the comparison baseline. |

### Dataset records

Each fetched page is emitted as a separate dataset item with `record_type: "crawl_result"`. It includes the current `start_url` and `page`, plus the submitted `start_urls` and a one-item `pages` array for compatibility. Run-level results are emitted separately and only once:

| Record type | Main fields |
| --- | --- |
| `crawl_result` | `start_url`, `page`, `start_urls`, `pages`. `page` is the current page result, containing an `audit` object and, when applicable, `images` and `image_summary`. |
| `site_summary` | Crawl totals and duplicate metadata groups for one start URL. |
| `wikipedia_trend` | One matching Wikipedia article's pageview trend and day-over-day comparison. |

For each page audit, `issues` lists the codes in `findings`: `missing_title`, `missing_meta_description`, `missing_h1`, `multiple_h1`, `heading_level_skipped`, and `http_error`. Each finding includes `issue`, `severity` (`error`, `warning`, or `info`), `evidence`, and `recommendation`. A heading-level issue is reported when a heading skips a level, such as an H1 followed by an H3.

`word_count` counts whitespace-separated words in parsed text, excluding text inside `script`, `style`, `noscript`, and `template` elements. CSS-hidden text may still be counted. Wayback date and URL fields are `null` when no snapshot is found, the lookup fails, or the lookup is disabled.

For Wikipedia records, `rank_change` is previous rank minus current rank, so positive values indicate a climb. `pageviews_change` and `pageviews_change_percent` compare with `comparison_date`. Country pageview counts are rounded up by Wikimedia, so country changes are approximate.

Discovered URLs returning HTTP 403 or 404 are recorded with their status and an `http_error` issue; their response pages are not used to discover more links. Other HTTP errors follow Crawlee's retry behavior, and a request that still fails is not guaranteed to produce a record. The Actor does not attempt to bypass access restrictions.

### Run locally

To run with an input file, save your JSON as `input.json` and use:

```bash
apify run --input-file ./input.json
```

You can also pass JSON inline with `apify run --input '{...}'` or `apify run -i '{...}'`. To use the stored input in `storage/key_value_stores/default/INPUT.json`, run `apify run` without an input option. Local dataset records are written to `storage/datasets/default/`.

Use `--purge` to clear the default dataset, request queue, and key-value store before a run; this is already the default for Crawlee projects. Use `--resurrect` to keep and reuse them. See `apify run --help` for other options.

### Standby API

The Actor also supports Apify Standby mode. `GET /` and `GET /health` return `{"status":"ok"}` when the server is ready. Send a JSON input matching the fields above to `POST /run`; the request stays open until the workflow finishes. The response groups all run results into one envelope:

```json
{
	"data": {
		"start_urls": ["https://example.com/"],
		"pages": [{"audit": {"url": "https://example.com/"}, "images": [], "image_summary": {}}],
		"site_summaries": [{"record_type": "site_summary", "start_url": "https://example.com/"}],
		"wikipedia_trends": []
	},
	"meta": {
		"source_mode": "climb_web",
		"page_count": 1,
		"site_summary_count": 1,
		"wikipedia_trend_count": 0
	}
}
```

The Actor also writes the existing page, site summary, and Wikipedia trend records to the run dataset. API errors use `{"error":{"code":"...","message":"..."}}`. The request body is limited to 1 MiB, and Standby requests must finish within five minutes. Climb Web requests require at least one `start_urls` entry. Invalid input returns HTTP 400; unsupported content types return 415.

Respect each crawled website's terms of service and crawling policies.

# Actor input Schema

## `source_mode` (type: `string`):

Crawl a website for SEO issues and image links, then find related Wikipedia trends.

## `start_urls` (type: `array`):

Website pages to crawl for SEO data, image links, and topics related to Wikipedia trends.

## `max_pages` (type: `integer`):

Maximum pages to crawl starting from each start URL.

## `max_rank` (type: `integer`):

Maximum country rank to check for related topics. Worldwide mode checks the top five separately for each of seven selected language editions.

## `lookback_days` (type: `integer`):

Number of complete UTC days to check for related topics. One additional earlier day is fetched as a comparison baseline.

## `trend_region` (type: `string`):

Choose top-five rankings separately for seven selected language editions, or rankings for one selected reader country. Only articles matching crawled page topics are included.

## `country_code` (type: `string`):

Choose a country by name. Its ISO 3166-1 alpha-2 code is sent to Wikimedia. Used only when Wikipedia trend region is Specific country.

## `include_url_patterns` (type: `array`):

Optional Python regular expressions for discovered URLs. When provided, a URL must match at least one pattern to be crawled.

## `exclude_url_patterns` (type: `array`):

Optional Python regular expressions; matching discovered URLs are skipped.

## `max_concurrency` (type: `integer`):

Maximum pages fetched simultaneously; lower values are gentler on websites.

## `max_requests_per_minute` (type: `integer`):

Rate limit for page requests during the crawl.

## `include_archive_snapshot` (type: `boolean`):

For each audited page, look up its latest successful HTML snapshot in the Internet Archive. This makes one additional request per page and is disabled by default.

## Actor input object example

```json
{
  "source_mode": "climb_web",
  "start_urls": [],
  "max_pages": 25,
  "max_rank": 10,
  "lookback_days": 1,
  "trend_region": "worldwide",
  "country_code": "US",
  "include_url_patterns": [],
  "exclude_url_patterns": [],
  "max_concurrency": 1,
  "max_requests_per_minute": 10,
  "include_archive_snapshot": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "start_urls": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("dodle/dodle-spider").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "start_urls": [] }

# Run the Actor and wait for it to finish
run = client.actor("dodle/dodle-spider").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "start_urls": []
}' |
apify call dodle/dodle-spider --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dodle/dodle-spider"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sIiBHbHELHiAor9b2/builds/4JlXyUgWJRbGg8Yq1/openapi.json