# Website Metadata Scraper (`automation-lab/website-metadata-bulk-extractor`) Actor

Extract declared SEO metadata, Open Graph tags, Twitter cards and evidence-backed technology hints from public URL lists for recurring site audits.

- **URL**: https://apify.com/automation-lab/website-metadata-bulk-extractor.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** SEO tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.22 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Metadata Scraper

Inventory website metadata across recurring public URL lists. Extract declared SEO tags, Open Graph tags, Twitter cards and evidence-backed technology hints into one record per successfully fetched page. Use the results for content audits, migration comparisons and social-preview inventories without downloading images or running a browser.

### What does it extract?

This Actor reads the HTML returned by each supplied URL. It does not generate descriptions, infer missing tags, score SEO, crawl linked pages or render JavaScript.

| Field | Meaning |
| --- | --- |
| `url`, `fetchedUrl` | Normalized input and response URL |
| `status`, `httpStatus` | Successful HTML fetch status |
| `title` | First HTML title |
| `description`, `keywords`, `robots` | First declared named meta tag |
| `canonicalUrl` | First canonical link resolved against the page URL |
| `language`, `viewport` | Declared language and viewport |
| `openGraph` | All `og:*` tags; arrays preserve repeated images/properties |
| `twitterCard` | All `twitter:*` tags as arrays |
| `metaTags` | Named, property and http-equiv meta declarations |
| `technologyHints` | Observed technology markers with bounded evidence |
| `scrapedAt` | UTC extraction time |
| `error` | Reason on an uncharged failed-URL record |

Missing scalar metadata is `null`; missing social metadata is `{}` and no observed technology hints is `[]`. Declared values are not rewritten into SEO recommendations.

### Who is it for?

- SEO analysts comparing titles, descriptions and canonicals across landing pages.
- Content teams checking whether social-sharing metadata survived a migration.
- Developers building a factual URL inventory for spreadsheets or data pipelines.
- Agencies repeating the same audit against client-owned public pages.

### Why use this Actor?

It keeps the audit evidence together. Repeated social tags are preserved rather than collapsed to one image. Each technology hint includes the actual asset path, asset host, generator declaration or server marker that triggered it. Missing tags remain missing rather than becoming an AI-generated summary.

It also makes failed URLs visible. Mixed runs retain useful results alongside uncharged errors; an all-failed run fails instead of reporting a successful empty audit.

### Getting started

1. Prepare final public HTML page URLs, not login pages, API endpoints or redirected aliases.
2. Enter them in **Public HTML page URLs**.
3. Set **Maximum distinct URLs to attempt** if you want a smaller batch.
4. Run the Actor and inspect the metadata inventory dataset and run summary.
5. Export JSON for complete nested tag arrays or CSV for scalar fields.

```json
{
  "startUrls": [
    {"url": "https://apify.com/"},
    {"url": "https://www.iana.org/domains/reserved"}
  ],
  "maxItems": 10
}
```

### Input parameters

| Parameter | Default | Behavior |
| --- | --- | --- |
| `startUrls` | Required | 1–1000 objects with a public HTTP(S) `url`. Only supplied pages are fetched. |
| `maxItems` | 100 | Global count of distinct URLs attempted, including errors; integer 1–1000. |

URL fragments are removed and identical normalized URLs are deduplicated before applying the global limit. Query strings are retained because they can select different pages. No zero/unlimited setting is supported. URLs are processed in supplied order.

Each URL entry accepts only `url` and an optional `method: "GET"`. Request payload, custom headers and `userData` are rejected, including when a hosted MCP editor exposes those generic fields. Only standard HTTP(S) ports are supported. Embedded credentials, private/reserved IP addresses and private/reserved DNS resolutions are rejected. Redirects are not followed; supply the final URL. There is no proxy configuration or automatic browser/residential fallback.

### Output example

This representative local result came from the prefilled example.com page:

```json
{
  "url": "https://example.com/",
  "fetchedUrl": "https://example.com/",
  "status": "ok",
  "httpStatus": 200,
  "title": "Example Domain",
  "description": null,
  "keywords": null,
  "robots": null,
  "canonicalUrl": null,
  "language": "en",
  "viewport": "width=device-width,initial-scale=1",
  "openGraph": {},
  "twitterCard": {},
  "metaTags": {"viewport": ["width=device-width,initial-scale=1"]},
  "technologyHints": [],
  "scrapedAt": "2026-10-10T20:04:58.064Z"
}
```

The `SUMMARY` key-value record contains requested, successful and failed URL counts. Error rows include `url`, `status: "error"`, `error` and `scrapedAt`, not invented metadata.

### Technology hints and limitations

Recognized markers include WordPress asset paths, Next.js asset/state markers, Shopify asset hosts, Google tag/Analytics, Meta Pixel SDK, Hotjar, Stripe and Webflow assets. A declared generator or `x-powered-by` header is reported as declared software, not independently verified identity.

Hints can be missing, stale or ambiguous. An included asset does not prove an active paid subscription, exact version, complete stack or ownership. The Actor does not claim 50+ technology detection. Text mentioning a framework is not treated as evidence.

### How much does it cost to extract website metadata?

Pay per event: a one-time **$0.00005 start fee**, plus a fee for each successful HTML metadata record. Failed URLs and duplicate URLs have no item event charge. A page with a real title but no social tags is still a successful metadata inventory record.

| Plan tier | Per successful page |
| --- | ---: |
| FREE | $0.002346 |
| BRONZE | $0.00204 |
| SILVER | $0.0015912 |
| GOLD / PLATINUM / DIAMOND | $0.001224 |

At BRONZE, a batch of 1 successful page totals $0.00209, 10 total $0.02045, and 100 total $0.20405. These are Actor event prices; consult your Apify plan for any other applicable platform charges. No AI or separate technology-hint event is charged. Always check the current Store pricing before starting.

### Reliability and troubleshooting

- A request has a 20-second deadline. Transient network errors and selected 429/5xx responses receive at most two retries. Permanent HTTP errors are not repeatedly retried.
- HTML responses are limited to 2 MiB. Images, scripts and fonts are not downloaded.
- Non-HTML responses, known challenge pages and redirects become explicit errors.
- Mixed success/error runs succeed with partial results; inspect `SUMMARY` before treating a batch as complete.
- If every URL fails, the run fails but keeps its diagnostic error rows.
- The default run timeout is five minutes. Large or slow lists may require a longer run timeout or smaller batches; timeouts can leave partial results.

If a description is null, check the original server-returned HTML. Tags added only after JavaScript execution are outside this Actor's static-HTML contract. If a URL redirects, replace it with its final public page URL. Protected, authenticated and private pages are unsupported.

### Integrations

Export scalar fields to Google Sheets or Excel for title/description inventories. Retain JSON in S3, a database or your data pipeline when repeated Open Graph tags matter. Schedule runs in Apify against the same URL list and compare exported rows downstream for migration regressions.

Use Make, Zapier, n8n or an Apify webhook to pass a completed dataset to your existing workflow. Scheduling, change comparison and alerts are downstream integrations, not built-in monitoring features.

### API usage

Start a run and retain its ID. Poll that same run to completion, then read its `defaultDatasetId`; run metadata is not the extracted dataset.

#### cURL

```bash
curl -X POST 'https://api.apify.com/v2/acts/automation-lab~website-metadata-bulk-extractor/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"startUrls":[{"url":"https://example.com/"}],"maxItems":10}'
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-metadata-bulk-extractor').call({
  startUrls: [{ url: 'https://example.com/' }], maxItems: 10,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems({ limit: 10 });
console.log(items);
```

#### Python

```python
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/website-metadata-bulk-extractor').call(
    run_input={'startUrls': [{'url': 'https://example.com/'}], 'maxItems': 10})
print(client.dataset(run['defaultDatasetId']).list_items(limit=10).items)
```

### MCP setup and bounded results

Select this Actor explicitly:

```bash
claude mcp add --transport http apify \
  'https://mcp.apify.com?tools=automation-lab/website-metadata-bulk-extractor'
```

Equivalent **Claude Desktop**, **Cursor**, and **VS Code** client configuration (use an HTTP-capable MCP adapter where required):

```json
{"mcpServers":{"apify":{"url":"https://mcp.apify.com?tools=automation-lab/website-metadata-bulk-extractor"}}}
```

Authenticate through your client's supported Apify token configuration. Discover actual tool names and arguments with read-only `tools/list`. Selecting an Actor also adds run/dataset/key-value/abort helpers, not exactly one tool.

Example prompt: “Extract declared website metadata from these public URLs, then show titles, canonical URLs and any missing social tags.”

Start once when authorized and retain the run/storage IDs. `call-actor` waits at most 45 seconds (default 30); configure the client timeout above the selected wait plus transport margin. For a nonterminal run, use `get-actor-run` for the same run with 2, 4, 8-second backoff capped at 10 seconds and a 120-second total deadline. At the deadline report pending with the run ID. Recover a known run after client timeout; do not restart just to poll. Report failures and partial results honestly.

After success, use `get-dataset-items` with explicit fields, limit and offset. A reasonable consumer budget is pages of 20 source rows, at most 100 rows and 64 KiB of serialized UTF-8 content admitted to model context, whichever comes first. Enforce the byte ceiling host-side, disclose omitted oversized rows, and track source offsets. If the client cannot intercept oversized responses, do not claim a hard byte guarantee. Full exports stay in storage outside model context. These consumer context choices do not reduce Actor collection limits. No universal token-savings claim is made.

### Legality and diagnostic privacy

Use public pages you are authorized to inspect and respect site terms, rate limits and applicable law. Do not submit credentials or private customer data. The Actor reads declared metadata, not permission to republish copyrighted content.

Failed operations send sanitized diagnostic input, exceptions and actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL queries are removed from diagnostic reports, which are retained for 30 days. Dataset output intentionally retains the submitted URL and declared public metadata; review exports before sharing them. No AI provider receives input or output. Apify stores datasets, logs and key-value records according to your account retention settings; delete them in Console or via the API when no longer needed. No cross-run cache or private session store is created. For questions, use the Actor's Apify Store Issues tab.

### FAQ

**Does it crawl the whole website?** No. It fetches only the supplied URL list, up to the global attempt limit.

**Can it render JavaScript?** No. It inventories server-returned HTML only.

**Why are tags arrays?** Multiple Open Graph images and repeated metadata declarations are legitimate. Arrays preserve them in document order.

**Does a technology hint prove a site's full stack?** No. It reports an observed marker only, with its evidence.

**Can I compare changes over time?** Schedule recurring runs and compare exports downstream. No history database or automatic change alerts are included.

**Can I use CSV?** Yes for scalar columns; nested social tags and evidence are best retained as JSON.

### Related workflows

This is a standalone per-URL declared metadata inventory. It does not replace a full crawler, contact-enrichment product, JavaScript renderer or security audit. Combine its dataset with your existing site-crawl URL list or migration checklist rather than assuming missing coverage.

# Changelog

This Actor's version history is a separate document: https://apify.com/automation-lab/website-metadata-bulk-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Supply 1–1000 public HTTP(S) page URLs. Each entry accepts only url and an optional GET method. Payload, headers and userData are unsupported and rejected. Only these pages are fetched; no links are crawled and no JavaScript is rendered. Duplicate normalized URLs and URL fragments are removed before the global limit. Redirects are reported as errors: supply the final page URL. Private/reserved networks, credentials and nonstandard ports are rejected.

## `maxItems` (type: `integer`):

Global limit on distinct URLs attempted in supplied order after deduplication, including failures. Default 100; range 1–1000. Zero/unlimited is not supported. Successful HTML records are charged; error records are not.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/"
    },
    {
      "url": "https://www.iana.org/domains/reserved"
    }
  ],
  "maxItems": 10
}
```

# Actor output Schema

## `dataset` (type: `string`):

Declared SEO/social metadata with technology evidence and uncharged error rows.

## `summary` (type: `string`):

Requested, successful and failed URL counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com/"
        },
        {
            "url": "https://www.iana.org/domains/reserved"
        }
    ],
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/website-metadata-bulk-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://apify.com/" },
        { "url": "https://www.iana.org/domains/reserved" },
    ],
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/website-metadata-bulk-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com/"
    },
    {
      "url": "https://www.iana.org/domains/reserved"
    }
  ],
  "maxItems": 10
}' |
apify call automation-lab/website-metadata-bulk-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/website-metadata-bulk-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oJfSkvBdeqYXAs3eV/builds/XYglBOfNwS5ylSHdg/openapi.json