# Page Metadata Extractor - SEO & RAG JSON Feed (`stefano_seggio/page-metadata-extractor`) Actor

Extract clean, structured metadata from any site: title, meta description, canonical URL, Open Graph tags, H1 and word count. Built-in retries, session rotation and optional pagination keep every crawl resilient. Built for SEO audits and LLM/RAG pipelines - pay only per result, never per wasted run.

- **URL**: https://apify.com/stefano\_seggio/page-metadata-extractor.md
- **Developed by:** [Stefano Seggio](https://apify.com/stefano_seggio) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 extracted results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Page Metadata Extractor - SEO & LLM/RAG Data Pipeline Crawler (CleanMeta)

### Executive Value Proposition

Auditing page metadata by hand means opening each page, reading the source, and copying title, description, canonical, Open Graph and H1 values into a spreadsheet one URL at a time - and doing it again every time you need a refresh. CleanMeta Crawler does the same extraction across an entire site in a single run: point it at one or more start URLs and it follows same-hostname links (and pagination, if you configure a selector) on its own, handing back clean, typed metadata instead of raw HTML you'd otherwise have to fetch and parse yourself. Built-in retries and session rotation mean a handful of flaky pages don't stall or skew the audit, and a permanently failed page is recorded with its error rather than silently dropped. The output is the same seven-field metadata snapshot a manual audit produces - title, description, canonical, Open Graph, language, H1, word count - just without the manual part.

### Use Cases

- **SEO agencies auditing client sites at scale.** Crawl a client's domain in one run and get every page's title, meta description and canonical URL back as structured data, so missing or duplicated tags surface in a spreadsheet instead of a manual page-by-page check.
- **RAG / LLM pipeline builders needing clean structured page metadata.** Feed a model `{title, metaDescription, h1}` per URL instead of raw HTML - fewer tokens spent on markup, no HTML parsing on your side, and a `wordCount` field to gauge page substance before ingestion.
- **Content teams checking Open Graph tags before publish.** Run the crawler against a staging or production section of the site and check `ogTitle` / `ogImage` per URL to catch a broken or stale social preview before it ships.

### Input

```json
{
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "maxRequestsPerCrawl": 20
}
```

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `startUrls` | array | yes | `[{"url": "https://apify.com"}]` | URLs to start crawling from. At least one is required. |
| `maxRequestsPerCrawl` | integer | no | `100` | Hard cap on pages fetched this run, across start URLs, same-hostname link discovery and pagination. |
| `paginationSelector` | string | no | - | CSS selector for a "next page" link, e.g. `a[rel=next]` or `.pagination .next`. When set, the crawler follows it up to `maxPaginationDepth` pages per start URL, on top of normal same-hostname link discovery. |
| `maxPaginationDepth` | integer | no | `3` | Max paginated pages to follow per start URL. Ignored when `paginationSelector` is not set. |
| `proxyConfiguration` | object | no | Apify Proxy (datacenter) | Standard Apify proxy configuration object. |
| `onlyChanged` | boolean | no | `false` | When `true`, only deliver (and get charged for) pages that are new or whose extracted metadata changed since the last time this Actor scraped that URL. Every visited page is still crawled and its links still followed either way. |

### Output

One dataset item per crawled page:

```json
{
  "url": "https://example.com/blog/post",
  "title": "How We Cut Page Load Time by 40%",
  "metaDescription": "A breakdown of the changes that moved the needle.",
  "canonicalUrl": "https://example.com/blog/post",
  "ogTitle": "How We Cut Page Load Time by 40%",
  "ogImage": "https://example.com/og/post.png",
  "language": "en",
  "h1": "How We Cut Page Load Time by 40%",
  "wordCount": 1284,
  "statusCode": 200,
  "crawlDepth": 1,
  "scrapedAt": "2026-09-04T11:04:27.177Z",
  "eventType": "NEW_URL",
  "contentHash": "3f9a1c2b8e7d4f0a1b2c3d4e5f60718293a4b5c",
  "previousScrapedAt": null
}
```

| Field | Type | Description |
|---|---|---|
| `url` | string | The page's final (loaded) URL. |
| `title` | string | Page `<title>`. |
| `metaDescription` | string or null | `meta[name=description]` content. |
| `canonicalUrl` | string or null | `link[rel=canonical]` href. |
| `ogTitle` | string or null | `og:title` meta content. |
| `ogImage` | string or null | `og:image` meta content. |
| `language` | string or null | `html[lang]` attribute. |
| `h1` | string or null | Text of the first `<h1>`, if any. |
| `wordCount` | integer | Approximate visible body word count. |
| `statusCode` | integer or null | HTTP status code of the response. |
| `crawlDepth` | integer | Link-hops from the nearest start URL (0 for a start URL itself). |
| `scrapedAt` | string | ISO timestamp of extraction. |
| `eventType` | string | `NEW_URL`, `CONTENT_CHANGED` or `UNCHANGED` - whether this URL's metadata is new, differs from the last scrape, or matches it. |
| `contentHash` | string | Fingerprint of the page's extracted content, used to detect `CONTENT_CHANGED` on later runs. |
| `previousScrapedAt` | string or null | Timestamp of the last scrape of this URL, or `null` if new. |

A request that permanently fails after retries is still recorded, as a dataset item with `url`, `error` and `failedAtRetry` fields instead of being silently dropped.

### Reliability

Every request goes through Crawlee's `CheerioCrawler` configured with `maxRequestRetries: 4` and `retryOnBlocked: true`, with `useSessionPool` and `persistCookiesPerSession` enabled so a suspected block rotates to a fresh session instead of hammering the same one. A page that still fails after its retries is not dropped silently - it's pushed to the dataset as an error record (`url`, `error`, `failedAtRetry`) so you can see exactly what didn't come back and why. `maxRequestsPerCrawl` is a hard ceiling checked across start URLs, same-hostname link discovery and pagination combined, so a run can't run away past what you asked for.

Pagination is opt-in and explicit: set `paginationSelector` to a CSS selector for a "next page" link (e.g. `a[rel=next]`), and the crawler follows it up to `maxPaginationDepth` pages per start URL, independently of normal same-hostname link discovery. Leave it unset and the crawler relies purely on same-hostname link discovery to find pages.

Change-detection state (one content fingerprint per URL) persists across runs in a dedicated key-value store, not the run's own temporary storage - so `onlyChanged` and the `eventType` field work correctly on a scheduled Actor task without any extra setup on your part.

### Pricing

CleanMeta Crawler runs on Apify's pay-per-event pricing: **$0.0005 per extracted result - $0.50 per 1,000 results** - plus a flat $0.00005 fee charged once when a run starts, regardless of how many pages it processes. You are not charged for requests that fail after retries; those are recorded as error items but never trigger a charge, so a flaky target site costs you nothing beyond the pages that actually came back. Set input `onlyChanged: true` to go further: pages are still crawled and their links still followed, but a page classified `UNCHANGED` since the last run is neither added to the dataset nor charged - the cheapest way to re-run a scheduled SEO audit or social-preview check and only pay for what's actually different this time. Beyond the once-per-run start fee, there's no idle-server cost, no proxy bill, and no charge tied to how many requests the crawler had to make to get there - pay only per result, never per wasted run.

### Support & Enterprise SLA

CleanMeta Crawler is built and maintained by an independent developer, not a vendor team with a formal enterprise support contract. Bug reports and feature requests go through the Issues tab on this Actor's Apify Store page and are typically triaged within about 48 hours. There's no guaranteed uptime commitment or dedicated account manager attached to this listing - if your use case needs a contractual SLA, custom extraction fields, or a Playwright-based variant for JavaScript-heavy sites, open an issue on the Store page to discuss what's realistic before relying on it in production.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start crawling from. At least one is required - the Actor charges an actor-start event just for running, so an empty list would bill the caller for zero output.

## `maxRequestsPerCrawl` (type: `integer`):

Hard limit on how many pages this run will fetch, across start URLs, discovered same-site links and pagination.

## `paginationSelector` (type: `string`):

Optional CSS selector for a next-page link, e.g. a\[rel=next] or .pagination .next. When set, the crawler follows it up to Max pagination depth pages per start URL, on top of normal same-site link discovery.

## `maxPaginationDepth` (type: `integer`):

Maximum number of paginated pages to follow per start URL when Pagination selector is set. Ignored otherwise.

## `proxyConfiguration` (type: `object`):

Proxies used to fetch pages. Apify Proxy (datacenter) is recommended as the default.

## `onlyChanged` (type: `boolean`):

When enabled, a page is still crawled (its links are still followed) but is only added to the dataset - and charged - if it's the first time this exact URL has been scraped, or its extracted metadata differs from the last time this Actor scraped it. Pages with unchanged metadata are skipped. Useful for scheduled re-runs where you only want to pay for what's new or different since the last run.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxRequestsPerCrawl": 100,
  "maxPaginationDepth": 3,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "onlyChanged": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("stefano_seggio/page-metadata-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("stefano_seggio/page-metadata-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call stefano_seggio/page-metadata-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,stefano_seggio/page-metadata-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/U9fUBHDngX6IyjzzF/builds/NuSVBrsKMdem8kTYq/openapi.json