AI-Ready Webpage Extractor

Convert public web pages into clean Markdown, text, links, tables, images, metadata, and schema.org JSON-LD for AI workflows.

What this Actor does

AI-Ready Webpage Extractor fetches public web pages, extracts the main content area, and saves structured content records to the Apify Dataset.

It is useful when you need clean page content for LLM prompts, RAG ingestion, search indexes, or content analysis.

Use cases

• Prepare public pages for LLM prompts.
• Build RAG ingestion pipelines.
• Extract docs, blog, pricing, and marketing content.
• Collect links, headings, tables, images, metadata, and JSON-LD from public pages.

Who this is for

AI app builders, RAG pipeline builders, researchers, data teams, documentation teams, and agencies that need clean public page content.

Input

• urls - Public pages to extract. Required, 1-100 items.
• outputFormat - Output content format: markdown, text, or both. Default: markdown.
• includeLinks - Return extracted links with anchor text. Default: true.
• includeTables - Return HTML tables as structured row arrays. Default: true.
• includeMetadata - Return metadata and schema.org JSON-LD when present. Default: true.
• maxPages - Maximum pages to extract in one run. Default: 100, maximum: 100.

Example input

{
  "urls": ["https://docs.apify.com/platform"],
  "outputFormat": "markdown",
  "includeLinks": true,
  "includeTables": true,
  "includeMetadata": true,
  "maxPages": 10
}

Output

The Actor saves one Dataset item per page.

• sourceUrl - Final fetched page URL.
• finalUrl - Same as sourceUrl in this version.
• title - Page title.
• description - Meta description or Open Graph description.
• author - Meta author when present.
• publishedDate, modifiedDate - Article or time metadata when present.
• mainContentMarkdown - Markdown content unless outputFormat is text.
• mainContentText - Plain text content when outputFormat is text or both.
• headings - Heading levels and text.
• links - Extracted links, or an empty array when disabled.
• tables - Extracted table row arrays, or an empty array when disabled.
• images - Image URLs and alt text from the main content.
• schemaOrgData - Parsed JSON-LD objects, or an empty array when disabled.
• wordCount - Approximate word count.
• estimatedTokens - Rough token estimate based on text length.
• language - HTML language attribute when present.
• scrapedAt - ISO timestamp for the run.
• extractionStatus - succeeded or failed.
• errorMessage - Failure reason when extraction fails.

Example output

{
  "sourceUrl": "https://docs.apify.com/platform",
  "finalUrl": "https://docs.apify.com/platform",
  "title": "Apify platform",
  "description": "Apify platform documentation",
  "author": null,
  "publishedDate": null,
  "modifiedDate": null,
  "mainContentMarkdown": "# Platform\n\nApify platform content...",
  "mainContentText": null,
  "headings": [{ "level": 1, "text": "Platform" }],
  "links": [{ "text": "Actors", "url": "https://docs.apify.com/platform/actors" }],
  "tables": [],
  "images": [],
  "schemaOrgData": [],
  "wordCount": 900,
  "estimatedTokens": 1200,
  "language": "en",
  "scrapedAt": "2026-07-04T00:00:00.000Z",
  "extractionStatus": "succeeded",
  "errorMessage": null
}

How it works

• Reads the Actor input.
• Normalizes and deduplicates public URLs.
• Fetches each page as public HTML.
• Selects main, article, or body as the content root.
• Extracts Markdown, text, headings, links, tables, images, metadata, and JSON-LD.
• Saves one structured record per page to the Apify Dataset.

Limitations

• Public pages only.
• No login, private data, cookies, CAPTCHA bypass, or restricted content.
• Static HTML extraction is used first.
• JavaScript-only sites may return limited or failed content.
• Main-content detection is heuristic and depends on page structure.
• Token estimates are approximate.

Error handling

Invalid or empty urls input fails run validation. Request failures return a failed Dataset item with extractionStatus: "failed" and errorMessage. Pages with too little extracted main content are marked as failed with Not enough main content extracted.

Troubleshooting

• If extracted content is too short, the page may rely on JavaScript or hide the main article content from the initial HTML.
• If you only need plain text, set outputFormat to text.
• If the Dataset is larger than expected, disable links, tables, or metadata fields you do not need.
• If metadata is missing, check whether the source page exposes meta tags or JSON-LD in public HTML.

API usage

curl -X POST "https://api.apify.com/v2/acts/USERNAME~ai-ready-webpage-extractor/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://docs.apify.com/platform"],
    "outputFormat": "both",
    "includeLinks": true,
    "includeTables": true,
    "includeMetadata": true,
    "maxPages": 10
  }'

Integration ideas

• Send Markdown into an LLM prompt or RAG pipeline.
• Export text records to a search index.
• Use links as crawl targets for follow-up runs.
• Store JSON-LD metadata for content enrichment.

SEO keywords

AI webpage extractor, Markdown extractor, RAG scraper, clean text API, webpage to markdown, content extraction, docs extractor, blog extractor, Apify AI actor, public web data

Ethical use

This Actor is designed for public data and user-provided URLs only. Do not use it to access private, login-protected, or restricted content.