Website Metadata Scraper

Inventory website metadata across recurring public URL lists. Extract declared SEO tags, Open Graph tags, Twitter cards and evidence-backed technology hints into one record per successfully fetched page. Use the results for content audits, migration comparisons and social-preview inventories without downloading images or running a browser.

What does it extract?

This Actor reads the HTML returned by each supplied URL. It does not generate descriptions, infer missing tags, score SEO, crawl linked pages or render JavaScript.

Field	Meaning
url, fetchedUrl	Normalized input and response URL
status, httpStatus	Successful HTML fetch status
title	First HTML title
description, keywords, robots	First declared named meta tag
canonicalUrl	First canonical link resolved against the page URL
language, viewport	Declared language and viewport
openGraph	All og:* tags; arrays preserve repeated images/properties
twitterCard	All twitter:* tags as arrays
metaTags	Named, property and http-equiv meta declarations
technologyHints	Observed technology markers with bounded evidence
scrapedAt	UTC extraction time
error	Reason on an uncharged failed-URL record

Missing scalar metadata is null; missing social metadata is {} and no observed technology hints is []. Declared values are not rewritten into SEO recommendations.

Who is it for?

• SEO analysts comparing titles, descriptions and canonicals across landing pages.
• Content teams checking whether social-sharing metadata survived a migration.
• Developers building a factual URL inventory for spreadsheets or data pipelines.
• Agencies repeating the same audit against client-owned public pages.

Why use this Actor?

It keeps the audit evidence together. Repeated social tags are preserved rather than collapsed to one image. Each technology hint includes the actual asset path, asset host, generator declaration or server marker that triggered it. Missing tags remain missing rather than becoming an AI-generated summary.

It also makes failed URLs visible. Mixed runs retain useful results alongside uncharged errors; an all-failed run fails instead of reporting a successful empty audit.

Getting started

1. Prepare final public HTML page URLs, not login pages, API endpoints or redirected aliases.
2. Enter them in Public HTML page URLs.
3. Set Maximum distinct URLs to attempt if you want a smaller batch.
4. Run the Actor and inspect the metadata inventory dataset and run summary.
5. Export JSON for complete nested tag arrays or CSV for scalar fields.

{
  "startUrls": [
    {"url": "https://apify.com/"},
    {"url": "https://www.iana.org/domains/reserved"}
  ],
  "maxItems": 10
}

Input parameters

Parameter	Default	Behavior
startUrls	Required	1–1000 objects with a public HTTP(S) url. Only supplied pages are fetched.
maxItems	100	Global count of distinct URLs attempted, including errors; integer 1–1000.

URL fragments are removed and identical normalized URLs are deduplicated before applying the global limit. Query strings are retained because they can select different pages. No zero/unlimited setting is supported. URLs are processed in supplied order.

Each URL entry accepts only url and an optional method: "GET". Request payload, custom headers and userData are rejected, including when a hosted MCP editor exposes those generic fields. Only standard HTTP(S) ports are supported. Embedded credentials, private/reserved IP addresses and private/reserved DNS resolutions are rejected. Redirects are not followed; supply the final URL. There is no proxy configuration or automatic browser/residential fallback.

Output example

This representative local result came from the prefilled example.com page:

{
  "url": "https://example.com/",
  "fetchedUrl": "https://example.com/",
  "status": "ok",
  "httpStatus": 200,
  "title": "Example Domain",
  "description": null,
  "keywords": null,
  "robots": null,
  "canonicalUrl": null,
  "language": "en",
  "viewport": "width=device-width,initial-scale=1",
  "openGraph": {},
  "twitterCard": {},
  "metaTags": {"viewport": ["width=device-width,initial-scale=1"]},
  "technologyHints": [],
  "scrapedAt": "2026-10-10T20:04:58.064Z"
}

The SUMMARY key-value record contains requested, successful and failed URL counts. Error rows include url, status: "error", error and scrapedAt, not invented metadata.

Technology hints and limitations

Recognized markers include WordPress asset paths, Next.js asset/state markers, Shopify asset hosts, Google tag/Analytics, Meta Pixel SDK, Hotjar, Stripe and Webflow assets. A declared generator or x-powered-by header is reported as declared software, not independently verified identity.

Hints can be missing, stale or ambiguous. An included asset does not prove an active paid subscription, exact version, complete stack or ownership. The Actor does not claim 50+ technology detection. Text mentioning a framework is not treated as evidence.

How much does it cost to extract website metadata?

Pay per event: a one-time $0.00005 start fee, plus a fee for each successful HTML metadata record. Failed URLs and duplicate URLs have no item event charge. A page with a real title but no social tags is still a successful metadata inventory record.

Plan tier	Per successful page
FREE	$0.002346
BRONZE	$0.00204
SILVER	$0.0015912
GOLD / PLATINUM / DIAMOND	$0.001224

At BRONZE, a batch of 1 successful page totals $0.00209, 10 total $0.02045, and 100 total $0.20405. These are Actor event prices; consult your Apify plan for any other applicable platform charges. No AI or separate technology-hint event is charged. Always check the current Store pricing before starting.

Reliability and troubleshooting

• A request has a 20-second deadline. Transient network errors and selected 429/5xx responses receive at most two retries. Permanent HTTP errors are not repeatedly retried.
• HTML responses are limited to 2 MiB. Images, scripts and fonts are not downloaded.
• Non-HTML responses, known challenge pages and redirects become explicit errors.
• Mixed success/error runs succeed with partial results; inspect SUMMARY before treating a batch as complete.
• If every URL fails, the run fails but keeps its diagnostic error rows.
• The default run timeout is five minutes. Large or slow lists may require a longer run timeout or smaller batches; timeouts can leave partial results.

If a description is null, check the original server-returned HTML. Tags added only after JavaScript execution are outside this Actor's static-HTML contract. If a URL redirects, replace it with its final public page URL. Protected, authenticated and private pages are unsupported.

Integrations

Export scalar fields to Google Sheets or Excel for title/description inventories. Retain JSON in S3, a database or your data pipeline when repeated Open Graph tags matter. Schedule runs in Apify against the same URL list and compare exported rows downstream for migration regressions.

Use Make, Zapier, n8n or an Apify webhook to pass a completed dataset to your existing workflow. Scheduling, change comparison and alerts are downstream integrations, not built-in monitoring features.

API usage

Start a run and retain its ID. Poll that same run to completion, then read its defaultDatasetId; run metadata is not the extracted dataset.

cURL

curl -X POST 'https://api.apify.com/v2/acts/automation-lab~website-metadata-bulk-extractor/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"startUrls":[{"url":"https://example.com/"}],"maxItems":10}'

JavaScript

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-metadata-bulk-extractor').call({
  startUrls: [{ url: 'https://example.com/' }], maxItems: 10,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems({ limit: 10 });
console.log(items);

Python

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/website-metadata-bulk-extractor').call(
    run_input={'startUrls': [{'url': 'https://example.com/'}], 'maxItems': 10})
print(client.dataset(run['defaultDatasetId']).list_items(limit=10).items)

MCP setup and bounded results

Select this Actor explicitly:

claude mcp add --transport http apify \
  'https://mcp.apify.com?tools=automation-lab/website-metadata-bulk-extractor'

Equivalent Claude Desktop, Cursor, and VS Code client configuration (use an HTTP-capable MCP adapter where required):

{"mcpServers":{"apify":{"url":"https://mcp.apify.com?tools=automation-lab/website-metadata-bulk-extractor"}}}

Authenticate through your client's supported Apify token configuration. Discover actual tool names and arguments with read-only tools/list. Selecting an Actor also adds run/dataset/key-value/abort helpers, not exactly one tool.

Example prompt: “Extract declared website metadata from these public URLs, then show titles, canonical URLs and any missing social tags.”

Start once when authorized and retain the run/storage IDs. call-actor waits at most 45 seconds (default 30); configure the client timeout above the selected wait plus transport margin. For a nonterminal run, use get-actor-run for the same run with 2, 4, 8-second backoff capped at 10 seconds and a 120-second total deadline. At the deadline report pending with the run ID. Recover a known run after client timeout; do not restart just to poll. Report failures and partial results honestly.

After success, use get-dataset-items with explicit fields, limit and offset. A reasonable consumer budget is pages of 20 source rows, at most 100 rows and 64 KiB of serialized UTF-8 content admitted to model context, whichever comes first. Enforce the byte ceiling host-side, disclose omitted oversized rows, and track source offsets. If the client cannot intercept oversized responses, do not claim a hard byte guarantee. Full exports stay in storage outside model context. These consumer context choices do not reduce Actor collection limits. No universal token-savings claim is made.

Legality and diagnostic privacy

Use public pages you are authorized to inspect and respect site terms, rate limits and applicable law. Do not submit credentials or private customer data. The Actor reads declared metadata, not permission to republish copyrighted content.

Failed operations send sanitized diagnostic input, exceptions and actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL queries are removed from diagnostic reports, which are retained for 30 days. Dataset output intentionally retains the submitted URL and declared public metadata; review exports before sharing them. No AI provider receives input or output. Apify stores datasets, logs and key-value records according to your account retention settings; delete them in Console or via the API when no longer needed. No cross-run cache or private session store is created. For questions, use the Actor's Apify Store Issues tab.

FAQ

Does it crawl the whole website? No. It fetches only the supplied URL list, up to the global attempt limit.

Can it render JavaScript? No. It inventories server-returned HTML only.

Why are tags arrays? Multiple Open Graph images and repeated metadata declarations are legitimate. Arrays preserve them in document order.

Does a technology hint prove a site's full stack? No. It reports an observed marker only, with its evidence.

Can I compare changes over time? Schedule recurring runs and compare exports downstream. No history database or automatic change alerts are included.

Can I use CSV? Yes for scalar columns; nested social tags and evidence are best retained as JSON.

Related workflows

This is a standalone per-URL declared metadata inventory. It does not replace a full crawler, contact-enrichment product, JavaScript renderer or security audit. Combine its dataset with your existing site-crawl URL list or migration checklist rather than assuming missing coverage.