Run Scrapy spiders on Apify
Reliable cloud infrastructure for your Scrapy project. Run, monitor, schedule, and scale your spiders in the cloud.
Easy start
From Scrapy spider to Apify Actor
Deploy your Scrapy code in the cloud with just a few commands.
1# Install Apify CLI
2brew install apify-cli
3
4# Create Apify Actor from your Scrapy spider
5cd your-scrapy-project
6apify init
7
8# Deploy it to Apify
9apify login
10apify pushFull support for Scrapy
We fully support Scrapy spiders as they are. Take advantage of the Apify platform's features without having to modify your Scrapy spider a bit.
Total package
Comprehensive tooling for your scrapers
Scheduling, monitoring, API, proxies, storage — you name it, we've got it. Explore our powerful features designed for reliable scraping at scale.
Run your spiders in the cloud
Use our CLI to quickly deploy your code to the cloud.
API and scheduler
Run your spider on schedule or call it via API.
Integration at your command
Connect your Scrapy projects with your favorite web apps.
Performance and quality monitoring
Inspect all spider runs, their logs, and runtime costs. Listen to events and get custom automated alerts.
Various formats and storage
Scalable and reliable cloud data storage designed for web scraping and automation workloads.
Proxies
Never get blocked with our carefully curated pool of datacenter and residential proxies and human-like browser fingerprints.
Monetize your spiders
Join hundreds of developers who publish their web scrapers as paid Actors on Apify Store, find users and get paid.
Other ways to create an Actor
Learn more
Looking for more Python tools? Our code templates, docs and library are a good place to start.
Platform Docs
Find more details and make your first steps on the Apify platform.
Scrapy & Python code templates
Build scrapers quickly with Python templates for Requests, Beautiful Soup, Scrapy, Playwright, and Selenium.
Python SDK
Use our library built for Actor lifecycle management, local storage emulation, and event handling.
Actorization
Actorized spiders
See the Scrapy spiders that were already transformed into Actors and published in Apify Store.
Sephora.com Scraper
scraping_empire/sephora-com-scraper
Scrape everything on Sephora.com from category products, product urls, and even search results.
scraping_empire
8
Scrapy Books Example
vdusek/scrapy-books-example
Example of Python Scrapy project. It scrapes book data from https://books.toscrape.com/.
vdusek
4
Yes, to access all the platform features you would need to get an account using your email. You can explore our platform's capabilities for free with our Free plan or choose a more extensive plan for your spiders.
You can make anything from a few bucks to a hundreds of dollars per month. The amount of money you can make from hosting spiders on the Apify platform varies based on factors like the complexity of your Actors and the demand for the data they provide.
Many developers earn passive income by sharing their Actors on the Apify Store, but the exact earnings depend on usage and user subscriptions.
The datasets in the output are available in multiple formats, including JSON, XML, CSV, HTML, and RSS. You can download your data, export it using an API or plug it into other apps.
Yes. Your Scrapy spider can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. You can integrate with Make, Zapier, Slack, LangChain, Airbyte, GitHub, Google Sheets, Google Drive, and more. Or you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever your new Actor successfully finishes a run.
Please feel free to contact us at python@apify.com