# Universal Web Scraper & Data Extractor (`motivational_nickel/universal-scraper`) Actor

Fast AI-ready webpage extraction with structured JSON output for automation workflows, AI agents, RAG pipelines, SEO analysis, and website intelligence.

- **URL**: https://apify.com/motivational\_nickel/universal-scraper.md
- **Developed by:** [Leoncio Jr Coronado](https://apify.com/motivational_nickel) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

Fast, AI-ready webpage extraction for automation workflows, AI agents, Retrieval-Augmented Generation (RAG), website intelligence, lead generation, SEO analysis, and structured data collection.

Universal Scraper extracts structured information from webpages using a lightweight HTTP-first engine. Instead of working with raw HTML, you receive organized JSON containing metadata, headings, text, links, images, and structured data that can be used directly in automation, analytics, and AI workflows.

Designed with a modular extraction pipeline, Universal Scraper provides consistent, production-ready output that is easy to integrate into business systems, APIs, AI agents, and downstream data pipelines.

***

### What does Universal Scraper do?

Universal Scraper combines multiple webpage extraction capabilities into a single Actor and returns the results using a predictable JSON structure.

It extracts:

- Webpage metadata
- Heading hierarchy (H1–H6)
- Visible webpage text
- Hyperlinks
- Images
- Structured data (JSON-LD)

Rather than parsing raw HTML yourself, you receive structured data that can be consumed immediately by automation workflows and AI applications.

***

### Common Use Cases

Universal Scraper is suitable for a wide range of web data extraction tasks.

#### AI Agents

Provide AI agents with structured webpage information instead of raw HTML.

#### Retrieval-Augmented Generation (RAG)

Extract webpage content for indexing into vector databases and knowledge bases used by LLM applications.

#### Website Intelligence

Analyze webpage structure, metadata, links, and structured information for research and competitive analysis.

#### Lead Generation

Collect public information from company websites as part of CRM enrichment and lead qualification workflows.

#### SEO Analysis

Extract titles, descriptions, heading structures, canonical URLs, and links for technical SEO audits.

#### Research

Build structured datasets from publicly accessible webpages for reporting, analytics, and content research.

#### Business Automation

Feed structured webpage data into APIs, Python applications, workflow automation platforms, and business systems.

***

### Getting Started

Using Universal Scraper requires only a few steps.

1. Open the Actor in Apify Console.
2. Enter one or more webpage URLs.
3. Choose which data you want to extract.
4. Start the Actor.
5. Download or integrate the generated dataset.

No browser automation is required for standard webpages.

***

### Input

Example input:

```json
{
  "start_urls": [
    {
      "url": "https://apify.com"
    }
  ],
  "extract_metadata": true,
  "extract_headings": true,
  "extract_text": true,
  "extract_links": true,
  "extract_images": true,
  "extract_structured_data": true,
  "follow_redirects": true,
  "verify_ssl": true
}
```

#### Main Input Options

| Option                  | Description                         |
|-------------------------|-------------------------------------|
| start\_urls              | List of webpages to scrape          |
| extract\_metadata        | Extract webpage metadata            |
| extract\_headings        | Extract H1–H6 headings              |
| extract\_text            | Extract visible webpage text        |
| extract\_links           | Extract hyperlinks                  |
| extract\_images          | Extract image information           |
| extract\_structured\_data | Extract JSON-LD and Schema.org data |
| follow\_redirects        | Follow HTTP redirects               |
| verify\_ssl              | Verify SSL certificates             |

### Output

Universal Scraper returns a consistent, AI-ready JSON structure designed for automation workflows, AI applications, analytics platforms, and business systems.

Example output:

```json
{
  "url": "https://apify.com",
  "resolved_url": "https://apify.com",
  "title": "Apify: Full-stack web scraping and data extraction platform",
  "description": "Build, deploy, and publish web scrapers, AI agents, and automation tools.",
  "canonical_url": "https://apify.com/",
  "language": "en",
  "metadata": {
    "og:type": "website",
    "og:site_name": "Apify"
  },
  "headings": [
    {
      "level": "h1",
      "text": "Web scraping meets AI"
    }
  ],
  "text_content": "Build reliable web scraping and automation workflows...",
  "links": [
    {
      "url": "https://docs.apify.com",
      "text": "Documentation",
      "link_type": "external"
    }
  ],
  "images": [
    {
      "src": "/images/logo.svg",
      "alt": "Apify",
      "title": ""
    }
  ],
  "structured_data": [
    {
      "format": "json-ld",
      "schema_type": "Organization",
      "content": "{...}"
    }
  ]
}
```

The exact output depends on the enabled extraction options and the structure of the target webpage.

***

### Key Features

Universal Scraper is designed to provide clean, structured webpage data through a lightweight and production-oriented extraction engine.

#### Metadata Extraction

Extract important webpage metadata including:

- Page title
- Meta description
- Canonical URL
- Document language
- Open Graph metadata
- Twitter Card metadata
- HTML meta tags

***

#### Heading Extraction

Build the complete heading hierarchy of a webpage.

Supported elements:

- H1
- H2
- H3
- H4
- H5
- H6

***

#### Text Extraction

Extract visible webpage text suitable for:

- AI preprocessing
- Knowledge bases
- Search indexing
- Content analysis
- Natural Language Processing (NLP)

***

#### Link Extraction

Collect hyperlinks and automatically classify them as:

- Internal
- External
- Email
- Telephone
- Anchor

***

#### Image Extraction

Extract image information including:

- Source URL
- Alternative text
- Title attribute

***

#### Structured Data Extraction

Extract structured information embedded in webpages, including:

- JSON-LD
- Schema.org data

***

### Why Choose Universal Scraper?

Universal Scraper is designed for reliability, maintainability, and easy integration into modern automation workflows.

#### Consistent JSON Output

Receive predictable, structured data instead of raw HTML, reducing the need for custom parsing.

#### AI-Ready

Use the extracted data directly in:

- AI Agents
- LLM applications
- Retrieval-Augmented Generation (RAG)
- Knowledge bases
- Automation workflows

#### Lightweight

The HTTP-first architecture provides fast extraction for standard webpages without requiring browser automation.

#### Modular

Independent extraction modules make it easy to configure the Actor based on the information you need.

#### Production-Oriented

Built with maintainability, extensibility, and predictable output as primary engineering goals.

### Limitations

Universal Scraper V1 focuses on fast and reliable HTTP-based webpage extraction. To keep the extraction engine lightweight and maintainable, the current version has the following limitations:

- JavaScript-rendered content is not supported.
- Login-protected pages are not supported.
- Browser automation is not included in V1.
- Recursive website crawling is not supported.
- Extraction quality depends on the HTML structure and accessibility of the target webpage.

These limitations are intentional design decisions that prioritize reliability, predictable output, and ease of maintenance. Future releases may expand the supported capabilities while preserving the existing output format.

***

### Best Fit

Universal Scraper is an excellent choice for:

- AI engineers building LLM applications
- Automation developers
- Data engineers
- SEO professionals
- Researchers and analysts
- Growth and marketing teams
- Business intelligence teams
- Developers integrating webpage data into automation systems

***

### Typical Workflows

Universal Scraper works well as a building block in larger automation pipelines.

Example workflow:

Website

↓

Universal Scraper

↓

Structured JSON

↓

AI Agent / RAG Pipeline / CRM / Analytics / API

Because the output structure is predictable, downstream systems require minimal custom parsing.

***

### Performance

Universal Scraper uses a lightweight HTTP-first architecture designed for efficient extraction of standard webpages.

Advantages include:

- Fast execution
- Low memory usage
- No browser startup overhead
- Configurable extraction modules
- Predictable JSON output

This design makes the Actor suitable for automation workflows that require structured webpage data without the additional resource cost of browser-based scraping.

***

### Version Roadmap

#### Current Release

- HTTP-first webpage extraction
- Metadata extraction
- Heading extraction
- Text extraction
- Link extraction
- Image extraction
- Structured data extraction

#### Future Enhancements

Future releases may introduce:

- JavaScript rendering
- Browser automation
- Multi-page crawling
- Additional extractors
- AI-assisted content extraction
- Enhanced content normalization

The current V1 output model is intended to remain stable as new capabilities are introduced.

***

### Support & Resources

Universal Scraper is actively maintained and designed for long-term evolution.

Resources include:

- GitHub repository
- Issue tracker
- Technical documentation
- Apify Actor page

Bug reports, feature requests, and community feedback are always welcome and help guide future development.

***

### Why Universal Scraper?

Many web scraping tools focus on extracting only one type of information or require developers to combine multiple libraries and custom scripts.

Universal Scraper provides a unified extraction engine that returns structured, AI-ready JSON through a single, consistent workflow. This simplifies automation, reduces development time, and provides a reliable foundation for integrating webpage data into business applications and AI systems.

Whether you're building an AI agent, enriching business data, analyzing websites, or creating automated research workflows, Universal Scraper helps you move from raw HTML to structured information with minimal effort.

***

### License

Universal Scraper is provided under the MIT License.

***

### Feedback

Your feedback helps improve Universal Scraper.

If you discover a bug, have suggestions for new features, or would like to contribute ideas for future releases, please open an issue in the project repository.

Community feedback plays an important role in guiding future improvements while maintaining a stable and reliable extraction engine.

***

### About This Project

Universal Scraper is developed following the LeoCor Engineering Principles:

- Business-first software.
- People-first design.

The project emphasizes:

- Production-grade architecture
- Modular engineering
- Predictable output
- Long-term maintainability
- AI-ready data extraction

Thank you for using Universal Scraper.

# Actor input Schema

## `start_urls` (type: `array`):

Public webpage URLs to scrape. Each URL produces one dataset result.

## `verify_ssl` (type: `boolean`):

Verify website SSL certificates. Disable only when scraping a trusted site with certificate problems.

## `follow_redirects` (type: `boolean`):

Follow HTTP redirects and report the final resolved URL.

## `timeout_seconds` (type: `integer`):

Maximum request duration in seconds for each URL.

## `retry_count` (type: `integer`):

Number of additional request attempts after a temporary failure.

## `user_agent` (type: `string`):

Optional User-Agent header. Leave empty to use the Actor default.

## `include_html` (type: `boolean`):

Include the downloaded HTML in each dataset item. Disabled by default to keep results compact.

## `extract_metadata` (type: `boolean`):

Extract title, description, canonical URL, language, robots directives, Open Graph, and social metadata.

## `extract_text` (type: `boolean`):

Extract readable plain-text page content.

## `extract_headings` (type: `boolean`):

Extract the page heading hierarchy from H1 through H6.

## `extract_links` (type: `boolean`):

Extract and categorize internal, external, email, and telephone links.

## `extract_images` (type: `boolean`):

Extract image URLs, alternative text, dimensions when available, and related metadata.

## `extract_structured_data` (type: `boolean`):

Extract JSON-LD and other supported machine-readable webpage data.

## `extract_statistics` (type: `boolean`):

Calculate content, heading, link, image, and response statistics.

## Actor input object example

```json
{
  "start_urls": [
    {
      "url": "https://apify.com"
    }
  ],
  "verify_ssl": true,
  "follow_redirects": true,
  "timeout_seconds": 30,
  "retry_count": 2,
  "user_agent": "",
  "include_html": false,
  "extract_metadata": true,
  "extract_text": true,
  "extract_headings": true,
  "extract_links": true,
  "extract_images": true,
  "extract_structured_data": true,
  "extract_statistics": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "start_urls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("motivational_nickel/universal-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "start_urls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("motivational_nickel/universal-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "start_urls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call motivational_nickel/universal-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=motivational_nickel/universal-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zSG5zcigxSa99tY7A/builds/FAsCG8WqEKhNPJ1NO/openapi.json