# Scrapy Cloud Runner (`solutionssmart/scrapy-cloud-runner`) Actor Run Scrapy spiders on Apify with request queue, dataset export, proxy rotation, scheduling, and cloud-ready deployment. - **URL**: https://apify.com/solutionssmart/scrapy-cloud-runner.md - **Developed by:** [Solutions Smart](https://apify.com/solutionssmart) (community) - **Categories:** Developer tools, Integrations, Lead generation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $2.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Scrapy Cloud Runner Run Scrapy spiders on Apify with cloud scheduling, API access, dataset output, proxy support, and production-friendly crawl defaults. ### What this actor does Scrapy Cloud Runner is a Python Apify Actor that executes Scrapy spiders bundled with the actor codebase. It uses the official Apify Python SDK and Scrapy integration so you can run, schedule, and monitor Scrapy crawls in the Apify Console or through the API. The actor: - runs a selected Scrapy spider by `spiderName` - reads input from the Apify input form or API - pushes scraped items to the default dataset - stores a crawl summary in the `OUTPUT` key-value store record - supports Apify proxy configuration - exposes crawl controls for limits, retries, delays, cache, and robots.txt ### Included spider The actor includes one bundled example spider: - `page_meta`: crawls pages, extracts basic page metadata, and optionally follows links The example spider is designed to be a solid starting point, not a universal website crawler. By default it stays on the same hostname as the start URLs to avoid drifting into subdomains with different blocking or rate-limit behavior. ### Why use it on Apify Running Scrapy on Apify gives you: - scheduled runs - API-triggered runs - centralized logs - dataset export - proxy integration - managed cloud execution You keep the Scrapy spider model, but you do not need to manage servers, deployment plumbing, or result storage yourself. ### Quick start 1. Open the actor in Apify Console. 2. Set `spiderName` to `page_meta` or to your own bundled spider. 3. Add one or more `startUrls`. 4. Keep the default limits for the first run. 5. Run the actor. 6. Review results in the **Dataset** tab and the summary in the `OUTPUT` record. ### Example input ```json { "spiderName": "page_meta", "startUrls": [ { "url": "https://apify.com" } ], "followLinks": true, "sameHostnameOnly": true, "includeHtml": false, "maxRequestsPerCrawl": 20, "maxDepth": 1, "maxConcurrency": 16, "requestTimeoutSecs": 30, "downloadDelaySecs": 1, "retryTimes": 2, "useAutoThrottle": true, "autoThrottleTargetConcurrency": 1, "autoThrottleStartDelaySecs": 1, "autoThrottleMaxDelaySecs": 15, "respectRobotsTxt": true, "useHttpCache": true, "httpCacheExpirationSecs": 7200, "spiderArgs": [ { "key": "category", "value": "books" } ] } ```` ### Input settings | Input | Type | Description | |---|---|---| | `spiderName` | string | Name of the bundled Scrapy spider to run. | | `startUrls` | array | Starting URLs for the crawl. | | `allowedDomains` | array | Optional domain allowlist for Scrapy offsite filtering. | | `followLinks` | boolean | Follow links discovered on crawled pages. | | `sameHostnameOnly` | boolean | Restrict followed links to the exact hostnames from `startUrls`. Recommended for focused crawls. | | `includeHtml` | boolean | Include raw HTML in dataset items. | | `maxRequestsPerCrawl` | integer | Maximum number of scraped pages/items emitted by the bundled spider. | | `maxDepth` | integer | Maximum follow depth from the initial pages. | | `maxConcurrency` | integer | Maximum concurrent Scrapy requests. | | `requestTimeoutSecs` | integer | Download timeout per request. | | `downloadDelaySecs` | number | Base delay between requests to the same site. | | `retryTimes` | integer | Retry count for retryable failures. | | `useAutoThrottle` | boolean | Enable Scrapy AutoThrottle. | | `autoThrottleTargetConcurrency` | number | Target average concurrency per remote site. | | `autoThrottleStartDelaySecs` | number | Initial AutoThrottle delay. | | `autoThrottleMaxDelaySecs` | number | Maximum AutoThrottle delay. | | `respectRobotsTxt` | boolean | Respect robots.txt. | | `useHttpCache` | boolean | Enable HTTP cache. | | `httpCacheExpirationSecs` | integer | Cache expiration time in seconds. | | `proxyConfiguration` | object | Apify proxy configuration. | | `spiderArgs` | array | Spider arguments entered as schema-based key/value rows in Apify Console. | | `spiderArgsJson` | object | Structured spider arguments for API callers. Merged over `spiderArgs` on duplicate keys. | ### Output The default dataset contains one item per scraped page. For the bundled `page_meta` spider, each item includes fields such as: ```json { "url": "https://apify.com", "status": 200, "title": "Apify: Full-stack web scraping and data extraction platform", "metaDescription": "Cloud platform for web scraping, browser automation, AI agents, and data for AI.", "canonicalUrl": "https://apify.com", "h1": "Get real-time web data for your AI", "contentType": "text/html; charset=utf-8", "depth": 0, "referrer": null, "textLength": 125419, "crawledAt": "2026-05-16T11:21:11.435924+00:00", "html": null } ``` The actor also stores a summary record in `OUTPUT`: ```json { "availableSpiders": ["page_meta"], "finishedAt": "2026-05-16T11:21:15.000000+00:00", "itemCount": 5, "requestCount": 12, "spiderName": "page_meta", "startedAt": "2026-05-16T11:21:10.000000+00:00", "stats": {} } ``` ### Default crawl behavior The bundled actor defaults are tuned for focused website crawls: - same-host following is enabled by default - AutoThrottle is enabled by default - HTTP cache uses RFC2616 policy - common blocked/error responses such as `403` and `429` are not cached - cookies are disabled - robots.txt is respected by default These defaults are more conservative and more production-friendly than simply running Scrapy at high parallelism. ### Add your own spiders 1. Add a spider module under `src/spiders/`. 2. Give the spider a unique Scrapy `name`. 3. Read any custom runtime options from spider kwargs or `spiderArgsJson`. 4. Deploy the updated actor. 5. Run the actor with `spiderName` set to your spider's name. The actor uses Scrapy's spider loader, so bundled spiders are discovered automatically from `src.spiders`. ### Practical guidance - Start with one or two `startUrls`. - Keep `sameHostnameOnly` enabled unless you intentionally want cross-subdomain crawling. - Use proxy configuration for websites with blocking or rate limiting. - Keep `includeHtml` off unless you need full source in the dataset. - For broad or multi-domain crawling, create a dedicated spider with different settings instead of using the bundled example as-is. ### Legal and operational note You are responsible for using this actor in compliance with the target site's terms, applicable law, and reasonable load limits. Keep `respectRobotsTxt` enabled unless you have a clear reason not to. # Actor input Schema ## `spiderName` (type: `string`): The Scrapy spider name to run. The bundled default is page\_meta. ## `startUrls` (type: `array`): Entry URLs for the spider. For most spiders, each item only needs a url field. ## `allowedDomains` (type: `array`): Optional domain allowlist. If empty, the actor derives domains from startUrls for the bundled page\_meta spider. ## `followLinks` (type: `boolean`): For the bundled page\_meta spider, continue crawling internal links discovered on each page. ## `sameHostnameOnly` (type: `boolean`): Restrict link following to the exact host names from startUrls. This avoids drifting into subdomains with different rate limits or blocking rules. ## `includeHtml` (type: `boolean`): Include raw HTML in dataset items. Keep disabled unless you really need full page source. ## `maxRequestsPerCrawl` (type: `integer`): Hard cap for processed pages. ## `maxDepth` (type: `integer`): How far the spider may follow links from the initial pages. ## `maxConcurrency` (type: `integer`): Maximum concurrent Scrapy requests. ## `requestTimeoutSecs` (type: `integer`): Download timeout applied to each request. ## `downloadDelaySecs` (type: `number`): Optional delay between requests to the same site. ## `retryTimes` (type: `integer`): Maximum retries for failed requests. ## `useAutoThrottle` (type: `boolean`): Enable Scrapy AutoThrottle to adapt request rate based on observed site latency. ## `autoThrottleTargetConcurrency` (type: `number`): Average concurrent requests Scrapy should aim for per remote site. ## `autoThrottleStartDelaySecs` (type: `number`): Initial delay before AutoThrottle adapts. ## `autoThrottleMaxDelaySecs` (type: `number`): Maximum delay AutoThrottle may apply when latency rises. ## `respectRobotsTxt` (type: `boolean`): Enable Scrapy robots.txt handling. ## `useHttpCache` (type: `boolean`): Cache responses in the default Apify key-value store to speed up repeated runs. ## `httpCacheExpirationSecs` (type: `integer`): How long cached responses stay valid when HTTP cache is enabled. ## `proxyConfiguration` (type: `object`): Apify proxy configuration passed into the Scrapy integration. ## `spiderArgs` (type: `array`): Optional spider arguments entered as key/value rows in the Console form. ## `spiderArgsJson` (type: `object`): Optional structured spider arguments for API callers. Merged over Spider arguments on duplicate keys. ## Actor input object example ```json { "spiderName": "page_meta", "startUrls": [ { "url": "https://apify.com" } ], "followLinks": true, "sameHostnameOnly": true, "includeHtml": false, "maxRequestsPerCrawl": 100, "maxDepth": 1, "maxConcurrency": 16, "requestTimeoutSecs": 30, "downloadDelaySecs": 1, "retryTimes": 2, "useAutoThrottle": true, "autoThrottleTargetConcurrency": 1, "autoThrottleStartDelaySecs": 1, "autoThrottleMaxDelaySecs": 15, "respectRobotsTxt": true, "useHttpCache": false, "httpCacheExpirationSecs": 7200, "spiderArgs": [ { "key": "category", "value": "books" } ], "spiderArgsJson": {} } ``` # Actor output Schema ## `results` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "spiderName": "page_meta", "startUrls": [ { "url": "https://apify.com" } ], "spiderArgs": [ { "key": "category", "value": "books" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("solutionssmart/scrapy-cloud-runner").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "spiderName": "page_meta", "startUrls": [{ "url": "https://apify.com" }], "spiderArgs": [{ "key": "category", "value": "books", }], } # Run the Actor and wait for it to finish run = client.actor("solutionssmart/scrapy-cloud-runner").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "spiderName": "page_meta", "startUrls": [ { "url": "https://apify.com" } ], "spiderArgs": [ { "key": "category", "value": "books" } ] }' | apify call solutionssmart/scrapy-cloud-runner --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=solutionssmart/scrapy-cloud-runner", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Scrapy Cloud Runner", "description": "Run Scrapy spiders on Apify with request queue, dataset export, proxy rotation, scheduling, and cloud-ready deployment.", "version": "0.1", "x-build-id": "Doza1wgni36tg02D1" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/solutionssmart~scrapy-cloud-runner/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-solutionssmart-scrapy-cloud-runner", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/solutionssmart~scrapy-cloud-runner/runs": { "post": { "operationId": "runs-sync-solutionssmart-scrapy-cloud-runner", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/solutionssmart~scrapy-cloud-runner/run-sync": { "post": { "operationId": "run-sync-solutionssmart-scrapy-cloud-runner", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "spiderName": { "title": "Spider name", "type": "string", "description": "The Scrapy spider name to run. The bundled default is page_meta.", "default": "page_meta" }, "startUrls": { "title": "Start URLs", "type": "array", "description": "Entry URLs for the spider. For most spiders, each item only needs a url field.", "items": { "type": "object", "additionalProperties": false, "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL", "description": "Starting URL for the crawl." } } } }, "allowedDomains": { "title": "Allowed domains", "type": "array", "description": "Optional domain allowlist. If empty, the actor derives domains from startUrls for the bundled page_meta spider.", "items": { "type": "string" } }, "followLinks": { "title": "Follow links", "type": "boolean", "description": "For the bundled page_meta spider, continue crawling internal links discovered on each page.", "default": true }, "sameHostnameOnly": { "title": "Same hostname only", "type": "boolean", "description": "Restrict link following to the exact host names from startUrls. This avoids drifting into subdomains with different rate limits or blocking rules.", "default": true }, "includeHtml": { "title": "Include HTML", "type": "boolean", "description": "Include raw HTML in dataset items. Keep disabled unless you really need full page source.", "default": false }, "maxRequestsPerCrawl": { "title": "Max requests per crawl", "minimum": 1, "type": "integer", "description": "Hard cap for processed pages.", "default": 100 }, "maxDepth": { "title": "Max crawl depth", "minimum": 0, "type": "integer", "description": "How far the spider may follow links from the initial pages.", "default": 1 }, "maxConcurrency": { "title": "Max concurrency", "minimum": 1, "maximum": 256, "type": "integer", "description": "Maximum concurrent Scrapy requests.", "default": 16 }, "requestTimeoutSecs": { "title": "Request timeout in seconds", "minimum": 1, "maximum": 300, "type": "integer", "description": "Download timeout applied to each request.", "default": 30 }, "downloadDelaySecs": { "title": "Download delay in seconds", "minimum": 0, "maximum": 60, "type": "number", "description": "Optional delay between requests to the same site.", "default": 1 }, "retryTimes": { "title": "Retry times", "minimum": 0, "maximum": 20, "type": "integer", "description": "Maximum retries for failed requests.", "default": 2 }, "useAutoThrottle": { "title": "Use AutoThrottle", "type": "boolean", "description": "Enable Scrapy AutoThrottle to adapt request rate based on observed site latency.", "default": true }, "autoThrottleTargetConcurrency": { "title": "AutoThrottle target concurrency", "minimum": 0.1, "maximum": 16, "type": "number", "description": "Average concurrent requests Scrapy should aim for per remote site.", "default": 1 }, "autoThrottleStartDelaySecs": { "title": "AutoThrottle start delay in seconds", "minimum": 0, "maximum": 60, "type": "number", "description": "Initial delay before AutoThrottle adapts.", "default": 1 }, "autoThrottleMaxDelaySecs": { "title": "AutoThrottle max delay in seconds", "minimum": 1, "maximum": 300, "type": "number", "description": "Maximum delay AutoThrottle may apply when latency rises.", "default": 15 }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Enable Scrapy robots.txt handling.", "default": true }, "useHttpCache": { "title": "Use HTTP cache", "type": "boolean", "description": "Cache responses in the default Apify key-value store to speed up repeated runs.", "default": false }, "httpCacheExpirationSecs": { "title": "HTTP cache expiration in seconds", "minimum": 0, "type": "integer", "description": "How long cached responses stay valid when HTTP cache is enabled.", "default": 7200 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Apify proxy configuration passed into the Scrapy integration." }, "spiderArgs": { "title": "Spider arguments", "type": "array", "description": "Optional spider arguments entered as key/value rows in the Console form.", "items": { "type": "object", "additionalProperties": false, "required": [ "key", "value" ], "properties": { "key": { "title": "Argument name", "type": "string", "description": "Spider argument name." }, "value": { "title": "Argument value", "type": "string", "description": "Spider argument value passed as a string." } } }, "default": [] }, "spiderArgsJson": { "title": "Spider arguments JSON", "type": "object", "description": "Optional structured spider arguments for API callers. Merged over Spider arguments on duplicate keys.", "default": {} } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```