# website content crawler (`akash9078/website-content-crawler`)

Powerful website content crawler tool to extract, analyze, and index web pages automatically. Streamline data collection with fast, accurate web scraping technology.

- **URL**: https://apify.com/akash9078/website-content-crawler.md
- **Developed by:** [Akash Kumar Naik](https://apify.com/akash9078) (community)
- **Categories:** Developer tools, Automation, Other
- **Stats:** 4 total users, 4 monthly users, 100.0% runs succeeded, NaN bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

# README

## Website Content Crawler

Extract website content with this advanced web crawler. Features stealth browsing with Camoufox, proxy rotation, and intelligent data extraction for content analysis.

### Key Features

- **🌐 Universal Crawling**: Crawl any website with configurable data extraction
- **🔒 Stealth Browsing**: Camoufox integration for avoiding detection
- **🚀 Advanced Proxy Management**: Request-level proxy rotation with automatic failover
- **🔗 Intelligent Link Following**: Depth-limited crawling with domain restrictions
- **📈 Rich Metadata**: Extract titles, descriptions, images, and meta tags

### Quick Start

#### 1. Run on Apify Platform

```bash
apify login
apify push
```

Then run your actor in Apify Console with these settings:

- **Start URLs**: `https://yesintelligent.com`
- **Max Pages**: `100`
- **Use Apify Proxy**: `true`

#### 2. Local Development

```bash
cd website-content-crawler
npm install
apify run
```

### Configuration Options

#### Basic Settings

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrls` | array | `["https://yesintelligent.com"]` | URLs to start crawling from |
| `maxPages` | integer | `100` | Maximum pages to crawl (1-1000) |
| `crawlDepth` | integer | `2` | Maximum crawl depth (0 = current page only) |

#### Proxy Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `proxyConfig.useApifyProxy` | boolean | `true` | Enable Apify proxy for anonymous browsing |
| `proxyConfig.apifyProxyGroups` | array | `["RESIDENTIAL"]` | Proxy groups: RESIDENTIAL, DATACENTER, or custom |

### Usage Examples

#### Example 1: Basic Website Crawling

```json
{
  "startUrls": [{"url": "https://yesintelligent.com"}],
  "maxPages": 50,
  "proxyConfig": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

#### Example 2: Deep Content Crawling

```json
{
  "startUrls": [{"url": "https://example-blog.com"}],
  "maxPages": 100,
  "crawlDepth": 3,
  "followExternal": false,
  "proxyConfig": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

### Output Data Structure

Each crawled page produces the following data:

```json
{
  "url": "https://example.com/page",
  "title": "Page Title",
  "description": "Page meta description",
  "content": "Main page content text",
  "meta": {
    "keywords": "keyword1, keyword2",
    "author": "Author Name",
    "publishedTime": "2024-01-01T00:00:00Z",
    "wordCount": 1000,
    "charCount": 5000,
    "readingTime": 5
  },
  "images": [
    {
      "src": "/image1.jpg",
      "alt": "Image description",
      "url": "https://example.com/image1.jpg"
    }
  ],
  "scrapedAt": "2024-01-01T12:00:00Z",
  "statusCode": 200,
  "depth": 1
}
```

### Best Practices

#### 1. Respect Rate Limits

- Use appropriate delays between requests
- Start with conservative concurrency settings
- Monitor server response times

#### 2. Optimize Data Extraction

- The crawler automatically extracts content from common selectors
- Images are deduplicated and converted to absolute URLs
- Metadata is extracted from meta tags

#### 3. Handle Anti-Bot Measures

- Enable proxy rotation for large crawls
- Use residential proxies for sensitive sites
- Monitor for rate limiting responses

### Deployment

#### Apify Platform

```bash
apify login
apify push
```

#### Docker Deployment

```bash
docker build -t website-content-crawler .
docker run -e APIFY_INPUT='{"startUrls":[{"url":"https://yesintelligent.com"}]}' website-content-crawler
```

### Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request

### License

This project is licensed under the ISC License.

### Support

For issues and questions:

- Check the [Apify Documentation](https://docs.apify.com/)
- Visit the [Apify Community Forum](https://community.apify.com/)
- Create an issue in the project repository

***

Built with ❤️ using Apify, Playwright, and Camoufox

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start crawling from

## `maxPages` (type: `integer`):

Maximum number of pages to crawl

## `proxyConfig` (type: `object`):

Configure proxies to avoid IP blocking. Using residential proxies is highly recommended for crawling to prevent detection and rate limiting.

## `crawlDepth` (type: `integer`):

How deep to follow links (0 = current page only, 1 = follow links 1 level deep)

## `followExternal` (type: `boolean`):

Follow links to other websites (careful with this)

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://yesintelligent.com"
    }
  ],
  "maxPages": 1,
  "proxyConfig": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  },
  "crawlDepth": 0,
  "followExternal": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://yesintelligent.com"
        }
    ],
    "proxyConfig": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("akash9078/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://yesintelligent.com" }],
    "proxyConfig": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("akash9078/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://yesintelligent.com"
    }
  ],
  "proxyConfig": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call akash9078/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=akash9078/website-content-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "website content crawler",
        "description": "Powerful website content crawler tool to extract, analyze, and index web pages automatically. Streamline data collection with fast, accurate web scraping technology.",
        "version": "0.0",
        "x-build-id": "GcpzXZMheZX9B99Yj"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/akash9078~website-content-crawler/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-akash9078-website-content-crawler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/akash9078~website-content-crawler/runs": {
            "post": {
                "operationId": "runs-sync-akash9078-website-content-crawler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/akash9078~website-content-crawler/run-sync": {
            "post": {
                "operationId": "run-sync-akash9078-website-content-crawler",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "startUrls"
                ],
                "properties": {
                    "startUrls": {
                        "title": "Start URLs",
                        "type": "array",
                        "description": "URLs to start crawling from",
                        "items": {
                            "type": "object",
                            "required": [
                                "url"
                            ],
                            "properties": {
                                "url": {
                                    "type": "string",
                                    "title": "URL of a web page",
                                    "format": "uri"
                                }
                            }
                        }
                    },
                    "maxPages": {
                        "title": "Max Pages to Crawl",
                        "minimum": 1,
                        "maximum": 1000,
                        "type": "integer",
                        "description": "Maximum number of pages to crawl",
                        "default": 1
                    },
                    "proxyConfig": {
                        "title": "🌐 Proxy Configuration",
                        "type": "object",
                        "description": "Configure proxies to avoid IP blocking. Using residential proxies is highly recommended for crawling to prevent detection and rate limiting.",
                        "default": {
                            "useApifyProxy": true,
                            "apifyProxyGroups": [
                                "RESIDENTIAL"
                            ]
                        }
                    },
                    "crawlDepth": {
                        "title": "Crawl Depth",
                        "minimum": 0,
                        "maximum": 5,
                        "type": "integer",
                        "description": "How deep to follow links (0 = current page only, 1 = follow links 1 level deep)",
                        "default": 0
                    },
                    "followExternal": {
                        "title": "Follow External Links",
                        "type": "boolean",
                        "description": "Follow links to other websites (careful with this)",
                        "default": false
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```
