Web Crawler
Pricing
$15.00/month + usage
Web Crawler
This web crawler is designed to provide users with complete flexibility by allowing them to use their **own proxies**. The scraper collects all pages from the website and returns extracts the **MetaData**, **Title**, and **Content** of the page in MarkDown.
0.0 (0)
Pricing
$15.00/month + usage
0
Total users
29
Monthly users
9
Runs succeeded
97%
Last modified
2 months ago
You can access the Web Crawler programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.
{ "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/sse?actors=rigelbytes/webcrawler", "--header", "Authorization: Bearer <YOUR_API_TOKEN>" ] } }}Configure MCP server with Web Crawler
You have a few options for interacting with the MCP server:
Use
mcp.apify.comviamcp-remotefrom your local machine to connect and authenticate using OAuth or an API token (as shown in the JSON configuration above).Set up the connection directly in your MCP client UI by providing the URL
https://mcp.apify.com/sse?actors=rigelbytes/webcrawleralong with an API token (or use OAuth).Connect to
mcp.apify.comvia Server-Sent Events (SSE), as shown below:
{ "mcpServers": { "apify": { "type": "sse", "url": "https://mcp.apify.com/sse?actors=rigelbytes/webcrawler", "headers": { "Authorization": "Bearer <YOUR_API_TOKEN>" } } }}You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.
If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.
