Pricing
$15.00/month + usage
This web crawler is designed to provide users with complete flexibility by allowing them to use their **own proxies**. The scraper collects all pages from the website and returns extracts the **MetaData**, **Title**, and **Content** of the page in MarkDown.
0.0 (0)
Pricing
$15.00/month + usage
0
38
7
Last modified
4 months ago
You can access the Web Crawler programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.
{ "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?actors=rigelbytes/webcrawler", "--header", "Authorization: Bearer <YOUR_API_TOKEN>" ] } }}You have a few options for interacting with the MCP server:
Use mcp.apify.com via mcp-remote from your local machine to connect and authenticate using OAuth or an API token (as shown in the JSON configuration above).
Set up the connection directly in your MCP client UI by providing the URL https://mcp.apify.com/?actors=rigelbytes/webcrawler along with an API token (or use OAuth).
Connect to mcp.apify.com via Server-Sent Events (SSE), as shown below:
{ "mcpServers": { "apify": { "type": "sse", "url": "https://mcp.apify.com/?actors=rigelbytes/webcrawler", "headers": { "Authorization": "Bearer <YOUR_API_TOKEN>" } } }}You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.
If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.