# AI-Ready Webpage Extractor (`s3nafps/ai-ready-webpage-extractor`) Actor

Convert public web pages into clean Markdown, text, links, tables, images, metadata, and JSON-LD for AI workflows.

- **URL**: https://apify.com/s3nafps/ai-ready-webpage-extractor.md
- **Developed by:** [mohamed senator](https://apify.com/s3nafps) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## AI-Ready Webpage Extractor

Convert public web pages into clean Markdown, text, links, tables, images, metadata, and schema.org JSON-LD for AI workflows.

### What this Actor does

AI-Ready Webpage Extractor fetches public web pages, extracts the main content area, and saves structured content records to the Apify Dataset.

It is useful when you need clean page content for LLM prompts, RAG ingestion, search indexes, or content analysis.

### Use cases

- Prepare public pages for LLM prompts.
- Build RAG ingestion pipelines.
- Extract docs, blog, pricing, and marketing content.
- Collect links, headings, tables, images, metadata, and JSON-LD from public pages.

### Who this is for

AI app builders, RAG pipeline builders, researchers, data teams, documentation teams, and agencies that need clean public page content.

### Input

- `urls` - Public pages to extract. Required, 1-100 items.
- `outputFormat` - Output content format: `markdown`, `text`, or `both`. Default: `markdown`.
- `includeLinks` - Return extracted links with anchor text. Default: `true`.
- `includeTables` - Return HTML tables as structured row arrays. Default: `true`.
- `includeMetadata` - Return metadata and schema.org JSON-LD when present. Default: `true`.
- `maxPages` - Maximum pages to extract in one run. Default: `100`, maximum: `100`.

### Example input

```json
{
  "urls": ["https://docs.apify.com/platform"],
  "outputFormat": "markdown",
  "includeLinks": true,
  "includeTables": true,
  "includeMetadata": true,
  "maxPages": 10
}
````

### Output

The Actor saves one Dataset item per page.

- `sourceUrl` - Final fetched page URL.
- `finalUrl` - Same as `sourceUrl` in this version.
- `title` - Page title.
- `description` - Meta description or Open Graph description.
- `author` - Meta author when present.
- `publishedDate`, `modifiedDate` - Article or time metadata when present.
- `mainContentMarkdown` - Markdown content unless `outputFormat` is `text`.
- `mainContentText` - Plain text content when `outputFormat` is `text` or `both`.
- `headings` - Heading levels and text.
- `links` - Extracted links, or an empty array when disabled.
- `tables` - Extracted table row arrays, or an empty array when disabled.
- `images` - Image URLs and alt text from the main content.
- `schemaOrgData` - Parsed JSON-LD objects, or an empty array when disabled.
- `wordCount` - Approximate word count.
- `estimatedTokens` - Rough token estimate based on text length.
- `language` - HTML language attribute when present.
- `scrapedAt` - ISO timestamp for the run.
- `extractionStatus` - `succeeded` or `failed`.
- `errorMessage` - Failure reason when extraction fails.

### Example output

```json
{
  "sourceUrl": "https://docs.apify.com/platform",
  "finalUrl": "https://docs.apify.com/platform",
  "title": "Apify platform",
  "description": "Apify platform documentation",
  "author": null,
  "publishedDate": null,
  "modifiedDate": null,
  "mainContentMarkdown": "# Platform\n\nApify platform content...",
  "mainContentText": null,
  "headings": [{ "level": 1, "text": "Platform" }],
  "links": [{ "text": "Actors", "url": "https://docs.apify.com/platform/actors" }],
  "tables": [],
  "images": [],
  "schemaOrgData": [],
  "wordCount": 900,
  "estimatedTokens": 1200,
  "language": "en",
  "scrapedAt": "2026-07-04T00:00:00.000Z",
  "extractionStatus": "succeeded",
  "errorMessage": null
}
```

### How it works

- Reads the Actor input.
- Normalizes and deduplicates public URLs.
- Fetches each page as public HTML.
- Selects `main`, `article`, or `body` as the content root.
- Extracts Markdown, text, headings, links, tables, images, metadata, and JSON-LD.
- Saves one structured record per page to the Apify Dataset.

### Limitations

- Public pages only.
- No login, private data, cookies, CAPTCHA bypass, or restricted content.
- Static HTML extraction is used first.
- JavaScript-only sites may return limited or failed content.
- Main-content detection is heuristic and depends on page structure.
- Token estimates are approximate.

### Error handling

Invalid or empty `urls` input fails run validation. Request failures return a failed Dataset item with `extractionStatus: "failed"` and `errorMessage`. Pages with too little extracted main content are marked as failed with `Not enough main content extracted`.

### Troubleshooting

- If extracted content is too short, the page may rely on JavaScript or hide the main article content from the initial HTML.
- If you only need plain text, set `outputFormat` to `text`.
- If the Dataset is larger than expected, disable links, tables, or metadata fields you do not need.
- If metadata is missing, check whether the source page exposes meta tags or JSON-LD in public HTML.

### API usage

```bash
curl -X POST "https://api.apify.com/v2/acts/USERNAME~ai-ready-webpage-extractor/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://docs.apify.com/platform"],
    "outputFormat": "both",
    "includeLinks": true,
    "includeTables": true,
    "includeMetadata": true,
    "maxPages": 10
  }'
```

### Integration ideas

- Send Markdown into an LLM prompt or RAG pipeline.
- Export text records to a search index.
- Use links as crawl targets for follow-up runs.
- Store JSON-LD metadata for content enrichment.

### SEO keywords

AI webpage extractor, Markdown extractor, RAG scraper, clean text API, webpage to markdown, content extraction, docs extractor, blog extractor, Apify AI actor, public web data

### Ethical use

This Actor is designed for public data and user-provided URLs only. Do not use it to access private, login-protected, or restricted content.

# Actor input Schema

## `urls` (type: `array`):

Public pages to extract.

## `outputFormat` (type: `string`):

Choose whether to return Markdown, plain text, or both.

## `includeLinks` (type: `boolean`):

Return extracted links with anchor text.

## `includeTables` (type: `boolean`):

Return HTML tables as structured row arrays.

## `includeMetadata` (type: `boolean`):

Return metadata and schema.org JSON-LD where present.

## `maxPages` (type: `integer`):

Maximum public pages to extract in one run.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/platform"
  ],
  "outputFormat": "markdown",
  "includeLinks": true,
  "includeTables": true,
  "includeMetadata": true,
  "maxPages": 100
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/platform"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("s3nafps/ai-ready-webpage-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com/platform"] }

# Run the Actor and wait for it to finish
run = client.actor("s3nafps/ai-ready-webpage-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/platform"
  ]
}' |
apify call s3nafps/ai-ready-webpage-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=s3nafps/ai-ready-webpage-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "AI-Ready Webpage Extractor",
        "description": "Convert public web pages into clean Markdown, text, links, tables, images, metadata, and JSON-LD for AI workflows.",
        "version": "0.1",
        "x-build-id": "ZFvtPqbR4ZjQcCNP7"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/s3nafps~ai-ready-webpage-extractor/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-s3nafps-ai-ready-webpage-extractor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/s3nafps~ai-ready-webpage-extractor/runs": {
            "post": {
                "operationId": "runs-sync-s3nafps-ai-ready-webpage-extractor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/s3nafps~ai-ready-webpage-extractor/run-sync": {
            "post": {
                "operationId": "run-sync-s3nafps-ai-ready-webpage-extractor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "urls"
                ],
                "properties": {
                    "urls": {
                        "title": "Public URLs",
                        "minItems": 1,
                        "maxItems": 100,
                        "type": "array",
                        "description": "Public pages to extract.",
                        "items": {
                            "type": "string"
                        }
                    },
                    "outputFormat": {
                        "title": "Output format",
                        "enum": [
                            "markdown",
                            "text",
                            "both"
                        ],
                        "type": "string",
                        "description": "Choose whether to return Markdown, plain text, or both.",
                        "default": "markdown"
                    },
                    "includeLinks": {
                        "title": "Include links",
                        "type": "boolean",
                        "description": "Return extracted links with anchor text.",
                        "default": true
                    },
                    "includeTables": {
                        "title": "Include tables",
                        "type": "boolean",
                        "description": "Return HTML tables as structured row arrays.",
                        "default": true
                    },
                    "includeMetadata": {
                        "title": "Include metadata",
                        "type": "boolean",
                        "description": "Return metadata and schema.org JSON-LD where present.",
                        "default": true
                    },
                    "maxPages": {
                        "title": "Maximum pages",
                        "minimum": 1,
                        "maximum": 100,
                        "type": "integer",
                        "description": "Maximum public pages to extract in one run.",
                        "default": 100
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```