Multi-Source Web Intelligence

Climb Web crawls a website once to report SEO issues, extract image links, and find Wikipedia pageview trends related to the site's page titles, descriptions, and headings. Each fetched page is written as its own crawl_result dataset item. Site summaries and Wikipedia trends are emitted once as separate dataset items.

Mode	What it does
climb_web	Crawl pages on the same hostname, report SEO metadata and issues, extract image links, and include Wikipedia trends whose article titles appear in crawled page titles, descriptions, or headings.

This is a lightweight audit of server-returned HTML, not a JavaScript-rendering crawler or a full technical SEO platform. It does not measure rankings, backlinks, Core Web Vitals, or browser-rendered content.

Quick start

Crawl a website

$apify run --input '{"source_mode":"climb_web","start_urls":[{"url":"https://example.com/"}],"max_pages":25}'

The crawl follows links on the same hostname, subject to the site's robots.txt and configured URL filters. It returns unique absolute HTTP(S) image URLs found in common image and lazy-load attributes, srcset, and Open Graph or Twitter image metadata; it does not download image files. Image records include source attributes, available alt/title/dimension attributes, and whether an img element lacks an alt attribute. An empty alt attribute is treated as present, since it may mark a decorative image.

Each successful page result includes its image links and one image_summary, including when the page has no images. The summary reports the unique image count, counts images with or missing an img alt attribute, identifies metadata or <source> image URLs without an applicable alt attribute, and lists the URLs missing alt attributes. Optional image attributes that were not found are omitted.

Website SEO audit

The Actor uses Crawlee and Beautiful Soup to fetch server-rendered HTML and follow links on the same hostname. Each start URL gets an independent crawl of up to 25 pages by default (maximum 1,000). Pages are processed one at a time by default, with a limit of 10 crawler tasks per minute. SEO and image records are produced during this same crawl; external links are not followed.

The page limit counts pages processed, not all HTTP requests. Retries and robots.txt fetches may add requests. JavaScript is not run, so content added only by client-side scripts may be missing.

Each crawl_result contains exactly one fetched page. Each site_summary contains run-level crawl totals, HTTP errors, missing metadata counts, and duplicate title and description groups for one start URL. Duplicate checks include successful pages only and ignore case and surrounding whitespace. Matching Wikipedia trends are emitted as individual wikipedia_trend records.

Set include_archive_snapshot to look up the latest successful Internet Archive snapshot for each page. This adds one request per page. The lookup is disabled by default; archive_snapshot_status is found, not_found, error, or disabled. Lookup errors do not stop the crawl.

Wikipedia trends

The Actor checks Wikimedia's robots.txt before requesting each REST API endpoint and skips endpoints that are disallowed. It retrieves rankings for previous complete UTC days, then keeps only articles whose normalized title phrase appears in a crawled page title, meta description, or H1-H3 heading. Matching ignores case and punctuation. It does not infer broader semantic relationships, so related articles without a phrase match are omitted.

• worldwide returns the top five articles separately from English (en), Japanese (ja), Chinese (zh), Thai (th), Korean (ko), French (fr), and German (de). This is not one combined global ranking: each rank and pageview count belongs to its language edition. Each record includes language_code; at most 35 trend records are returned per day.
• country returns up to max_rank articles for the selected reader country. Each record includes its ISO 3166-1 alpha-2 country_code; Wikimedia rounds country pageview counts up.

The Actor fetches one extra, earlier UTC day as a comparison baseline but does not emit records for it. Therefore a one-day run can include day-over-day changes. Worldwide mode makes one sequential request per language edition per date, including the baseline. Requests have a two-second minimum pause; rate-limit and transient failures are retried, using Wikimedia's Retry-After delay when provided. Editions or countries that return 404 are skipped with a warning. A positive rank_change means the article climbed.

Each trend entry includes comparison_date and comparison_status. compared means the article was ranked on both dates and change values are populated. entered_top_n means prior data was available, but the article was outside the previous day's returned ranking; it does not imply the article is new. prior_data_unavailable means the previous date could not be fetched, for example because it was disallowed by robots.txt or returned 404. Change fields remain null unless the status is compared.

Input reference

The defaults below apply when a field is omitted.

Field	Default / limits	Used for	Description
source_mode	climb_web	All	The only supported mode; runs the site crawl, SEO audit, image extraction, and related Wikipedia trend check.
start_urls	[]	All	Full URLs to crawl. At least one URL is required.
max_pages	25; 1–1,000	Crawl	Maximum pages processed from each start URL.
include_url_patterns	[]	Crawl	Optional Python regular expressions. If set, a discovered absolute URL must match at least one pattern to be followed.
exclude_url_patterns	[]	Crawl	Optional Python regular expressions; matching discovered URLs are skipped.
max_concurrency	1; 1–5	Crawl	Maximum number of pages fetched simultaneously.
max_requests_per_minute	10; 10–60	Crawl	Crawlee task-rate limit for page fetches, not a strict limit on all HTTP requests.
include_archive_snapshot	false	Crawl	Look up the latest successful Internet Archive snapshot for every audited page; adds one request per page.
trend_region	worldwide	Wikipedia	worldwide returns separate top-five rankings for seven language editions, not one combined ranking; country returns rankings for one reader country.
country_code	US	Wikipedia	Country selected by ISO 3166-1 alpha-2 code; used only with trend_region: "country".
max_rank	10; 1–1,000	Wikipedia	Maximum country ranking. Worldwide mode always returns ranks 1–5 per edition.
lookback_days	1; 1–30	Wikipedia	Number of complete UTC days to return. One additional earlier day is fetched as the comparison baseline.

Dataset records

Each fetched page is emitted as a separate dataset item with record_type: "crawl_result". It includes the current start_url and page, plus the submitted start_urls and a one-item pages array for compatibility. Run-level results are emitted separately and only once:

Record type	Main fields
crawl_result	start_url, page, start_urls, pages. page is the current page result, containing an audit object and, when applicable, images and image_summary.
site_summary	Crawl totals and duplicate metadata groups for one start URL.
wikipedia_trend	One matching Wikipedia article's pageview trend and day-over-day comparison.

For each page audit, issues lists the codes in findings: missing_title, missing_meta_description, missing_h1, multiple_h1, heading_level_skipped, and http_error. Each finding includes issue, severity (error, warning, or info), evidence, and recommendation. A heading-level issue is reported when a heading skips a level, such as an H1 followed by an H3.

word_count counts whitespace-separated words in parsed text, excluding text inside script, style, noscript, and template elements. CSS-hidden text may still be counted. Wayback date and URL fields are null when no snapshot is found, the lookup fails, or the lookup is disabled.

For Wikipedia records, rank_change is previous rank minus current rank, so positive values indicate a climb. pageviews_change and pageviews_change_percent compare with comparison_date. Country pageview counts are rounded up by Wikimedia, so country changes are approximate.

Discovered URLs returning HTTP 403 or 404 are recorded with their status and an http_error issue; their response pages are not used to discover more links. Other HTTP errors follow Crawlee's retry behavior, and a request that still fails is not guaranteed to produce a record. The Actor does not attempt to bypass access restrictions.

Run locally

To run with an input file, save your JSON as input.json and use:

$apify run --input-file ./input.json

You can also pass JSON inline with apify run --input '{...}' or apify run -i '{...}'. To use the stored input in storage/key_value_stores/default/INPUT.json, run apify run without an input option. Local dataset records are written to storage/datasets/default/.

Use --purge to clear the default dataset, request queue, and key-value store before a run; this is already the default for Crawlee projects. Use --resurrect to keep and reuse them. See apify run --help for other options.

Standby API

The Actor also supports Apify Standby mode. GET / and GET /health return {"status":"ok"} when the server is ready. Send a JSON input matching the fields above to POST /run; the request stays open until the workflow finishes. The response groups all run results into one envelope:

{
	"data": {
		"start_urls": ["https://example.com/"],
		"pages": [{"audit": {"url": "https://example.com/"}, "images": [], "image_summary": {}}],
		"site_summaries": [{"record_type": "site_summary", "start_url": "https://example.com/"}],
		"wikipedia_trends": []
	},
	"meta": {
		"source_mode": "climb_web",
		"page_count": 1,
		"site_summary_count": 1,
		"wikipedia_trend_count": 0
	}
}

The Actor also writes the existing page, site summary, and Wikipedia trend records to the run dataset. API errors use {"error":{"code":"...","message":"..."}}. The request body is limited to 1 MiB, and Standby requests must finish within five minutes. Climb Web requests require at least one start_urls entry. Invalid input returns HTTP 400; unsupported content types return 415.

Respect each crawled website's terms of service and crawling policies.