# AI-Ready Content Extractor — Structured Web Data for LLM & MCP (`yuchiaoniu/ai-content-extractor`) Actor

Extract structured JSON from any URL for LLM, RAG, and MCP integration. Outputs title, sections, contact info, links, structured data, and clean plain text.

- **URL**: https://apify.com/yuchiaoniu/ai-content-extractor.md
- **Developed by:** [Niu Yuchiao](https://apify.com/yuchiaoniu) (community)
- **Categories:** AI, Developer tools
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## AI-Ready Content Extractor — Structured Web Data for LLM & MCP

Extract fully structured JSON content from any URL — optimized for LLM consumption, RAG pipelines, and **MCP (Model Context Protocol)** tool integration. One Actor call returns everything you need: title, sections, headings, contact info, internal/external links, JSON-LD structured data, and clean plain text.

**Works seamlessly with Claude, GPT-4, LangChain, LlamaIndex, and any AI framework.**

### Why Use This vs. a Regular Scraper?

| Feature | Regular Scraper | This Actor |
|---------|----------------|------------|
| Output format | Raw HTML | Clean JSON |
| LLM-ready | ❌ needs post-processing | ✅ ready to paste into prompt |
| Contact extraction | ❌ manual regex | ✅ built-in |
| Structured data | ❌ ignored | ✅ JSON-LD parsed |
| Section splitting | ❌ flat blob | ✅ heading hierarchy |
| MCP compatible | ❌ | ✅ |

### Use Cases

- **AI agents**: Give your Claude/GPT agent the ability to read and understand any webpage
- **RAG pipelines**: Feed structured sections directly into vector databases
- **Lead enrichment**: Extract emails, phones, and addresses from company websites
- **Content auditing**: Audit page structure, headings, and link patterns at scale
- **Competitive research**: Extract structured data from competitor pages
- **MCP tool integration**: Use as a tool in your MCP server to give AI access to web content

### Input

```json
{
  "urls": [
    "https://example.com",
    "https://another-site.com/about"
  ],
  "extractSections": true,
  "extractContactInfo": true,
  "extractLinks": true,
  "extractStructuredData": true,
  "includeRawText": true,
  "maxContentLength": 10000
}
````

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `urls` | Array | (required) | List of URLs to extract |
| `extractSections` | Boolean | `true` | Split content into heading-based sections |
| `extractContactInfo` | Boolean | `true` | Find emails, phones in page text |
| `extractLinks` | Boolean | `true` | Collect internal + external links |
| `extractStructuredData` | Boolean | `true` | Parse JSON-LD (schema.org) if present |
| `includeRawText` | Boolean | `true` | Include full plain text for LLM prompt injection |
| `maxContentLength` | Integer | `0` | Truncate `textForLlm` at N chars (0 = unlimited) |

### Output

Each dataset record:

```json
{
  "url": "https://example.com",
  "canonical": "https://example.com/",
  "title": "Example Domain",
  "lang": "en",
  "meta": {
    "description": "This domain is for illustrative examples.",
    "keywords": null
  },
  "wordCount": 312,
  "hasEmail": true,
  "hasPhone": false,
  "contactInfo": {
    "emails": ["contact@example.com"],
    "phones": []
  },
  "sections": [
    {
      "heading": "About Us",
      "level": 2,
      "content": ["We are a company that...", "Founded in 2020..."]
    }
  ],
  "links": {
    "internal": [
      { "url": "https://example.com/about", "text": "About" }
    ],
    "external": [
      { "url": "https://twitter.com/example", "text": "Follow us" }
    ]
  },
  "structuredData": [
    { "@context": "https://schema.org", "@type": "Organization", "name": "Example" }
  ],
  "textForLlm": "Example Domain This domain is for illustrative examples...",
  "extractedAt": "2026-07-04T10:00:00.000Z"
}
```

### MCP Integration

This Actor is designed to be called from MCP tool definitions. Example MCP tool wrapper:

```python
## In your MCP server
@server.tool()
async def read_webpage(url: str) -> str:
    """Read and extract structured content from a webpage for AI analysis."""
    from apify_client import ApifyClient
    client = ApifyClient("YOUR_TOKEN")
    run = client.actor("yuchiaoniu/ai-content-extractor").call(run_input={
        "urls": [url],
        "includeRawText": True,
        "maxContentLength": 8000,
        "extractContactInfo": True,
    })
    items = client.dataset(run["defaultDatasetId"]).list_items().items
    return items[0]["textForLlm"] if items else "Could not extract content."
```

Once wrapped, your AI agent can call `read_webpage("https://...")` and get clean, LLM-ready text back instantly.

### Use with LangChain

```python
from langchain.tools import tool
from apify_client import ApifyClient

@tool
def extract_web_content(url: str) -> dict:
    """Extract structured content from a URL for analysis."""
    client = ApifyClient("YOUR_TOKEN")
    run = client.actor("yuchiaoniu/ai-content-extractor").call(run_input={
        "urls": [url],
        "extractSections": True,
        "maxContentLength": 6000,
    })
    items = client.dataset(run["defaultDatasetId"]).list_items().items
    return items[0] if items else {}
```

### Use with JavaScript / Claude API

```javascript
import Anthropic from '@anthropic-ai/sdk';
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const anthropic = new Anthropic();

// Extract page content
const run = await client.actor('yuchiaoniu/ai-content-extractor').call({
    urls: ['https://competitor.com/pricing'],
    includeRawText: true,
    maxContentLength: 8000,
});
const [page] = (await client.dataset(run.defaultDatasetId).listItems()).items;

// Feed directly into Claude
const response = await anthropic.messages.create({
    model: 'claude-opus-4-5',
    max_tokens: 1024,
    messages: [{
        role: 'user',
        content: `Analyze this pricing page and summarize the plans:\n\n${page.textForLlm}`,
    }],
});
```

### Pair with RAG Pipeline Scraper

For full RAG pipelines, pair this Actor with [RAG Pipeline Scraper](https://apify.com/yuchiaoniu/rag-pipeline-scraper):

- **This Actor** → structured JSON with contact info, sections, links (best for targeted pages)
- **RAG Pipeline Scraper** → multi-page crawl with Markdown + JSONL chunks (best for entire sites)

### FAQ

**Does it handle JavaScript-rendered pages?**
This Actor uses Cheerio (HTML parser) for maximum speed. For heavily JS-rendered sites (SPAs), consider pairing with a Playwright-based crawler first.

**How many URLs can I process per run?**
No hard limit — tested with 500+ URLs in a single run. Set `maxConcurrency` in the Actor settings if you want to throttle.

**What languages does it support?**
Any language — the extractor is language-agnostic. The `lang` field in output reflects the page's declared language.

**Can I use this to build a RAG chatbot?**
Yes! Extract content with `extractSections: true`, then embed each section individually in your vector database for fine-grained retrieval.

# Actor input Schema

## `urls` (type: `array`):

List of URLs to extract structured content from

## `extractSections` (type: `boolean`):

Split content into logical sections by headings

## `extractContactInfo` (type: `boolean`):

Find emails, phone numbers, and addresses in content

## `extractLinks` (type: `boolean`):

Include all internal and external links found

## `extractStructuredData` (type: `boolean`):

Parse JSON-LD structured data (schema.org) if present

## `includeRawText` (type: `boolean`):

Include concatenated plain text (ideal for feeding directly into LLM prompts)

## `maxContentLength` (type: `integer`):

Truncate plain text at this character count (0 = unlimited)

## Actor input object example

```json
{
  "extractSections": true,
  "extractContactInfo": true,
  "extractLinks": true,
  "extractStructuredData": true,
  "includeRawText": true,
  "maxContentLength": 0
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("yuchiaoniu/ai-content-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("yuchiaoniu/ai-content-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call yuchiaoniu/ai-content-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=yuchiaoniu/ai-content-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "AI-Ready Content Extractor — Structured Web Data for LLM & MCP",
        "description": "Extract structured JSON from any URL for LLM, RAG, and MCP integration. Outputs title, sections, contact info, links, structured data, and clean plain text.",
        "version": "1.0",
        "x-build-id": "Din3OkvwiEhWbWKmr"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/yuchiaoniu~ai-content-extractor/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-yuchiaoniu-ai-content-extractor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/yuchiaoniu~ai-content-extractor/runs": {
            "post": {
                "operationId": "runs-sync-yuchiaoniu-ai-content-extractor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/yuchiaoniu~ai-content-extractor/run-sync": {
            "post": {
                "operationId": "run-sync-yuchiaoniu-ai-content-extractor",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "urls"
                ],
                "properties": {
                    "urls": {
                        "title": "URLs to Extract",
                        "type": "array",
                        "description": "List of URLs to extract structured content from",
                        "items": {
                            "type": "string"
                        }
                    },
                    "extractSections": {
                        "title": "Extract Sections",
                        "type": "boolean",
                        "description": "Split content into logical sections by headings",
                        "default": true
                    },
                    "extractContactInfo": {
                        "title": "Extract Contact Info",
                        "type": "boolean",
                        "description": "Find emails, phone numbers, and addresses in content",
                        "default": true
                    },
                    "extractLinks": {
                        "title": "Extract Links",
                        "type": "boolean",
                        "description": "Include all internal and external links found",
                        "default": true
                    },
                    "extractStructuredData": {
                        "title": "Extract Structured Data (JSON-LD)",
                        "type": "boolean",
                        "description": "Parse JSON-LD structured data (schema.org) if present",
                        "default": true
                    },
                    "includeRawText": {
                        "title": "Include Plain Text",
                        "type": "boolean",
                        "description": "Include concatenated plain text (ideal for feeding directly into LLM prompts)",
                        "default": true
                    },
                    "maxContentLength": {
                        "title": "Max Text Length (chars)",
                        "minimum": 0,
                        "type": "integer",
                        "description": "Truncate plain text at this character count (0 = unlimited)",
                        "default": 0
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```