This actor allows you to run web spiders written in Python and the Scrapy framework on the Apify platform. Executing a spider is as simple as copy-pasting your Scrapy code into the actor's input. For multi-file Scrapy spiders, see the bottom of this readme.
Please note that the actor is experimental and it might change in the future.
The actor has the following input options:
- Scrapy code - Paste your Python source code with Scrapy into this field.
- Proxy - Optionally, select a proxy to be used by the actor, in order to avoid IP address-based blocking by the target website. The actor automatically executes all the Scrapy's HTTP(S) requests through the proxy.
To store your Scrapy items in Apify's Dataset
or Key-value store cloud storages,
you can use the
apify Python package.
All the methods are available for actors running both locally as well as on the Apify platform.
First, import the package by adding the following command to the top of your source file:
To push your scraped data to the Dataset associated with the actor run, use the
Note that Datasets are useful for storing large tabular results, such as a list of products from an e-commerce site.
To interact with the default Key-value store associated with the actor run,
apify.setValue('foo.txt', 'bar') apify.getValue('foo.txt') apify.deleteValue('foo.txt')
Key-value stores are useful for storing files, e.g. screenshots, PDFs of crawler state.
If your Scrapy spider contains multiple source code or configuration files, or you want to configure Scrapy settings, pipelines or middlewares, you can download the source code of this actor, import your files into it and push it to the Apify cloud for execution.
Here are instructions:
- Clone the GitHub repository with the source code of this actor:
git clone https://github.com/apifytech/actor-scrapy-executor
- Go to the repository directory and install NPM packages:
cd actor-scrapy-executor npm install
- Copy your spider(s) into the
- Make any necessary changes to files in the the
- Run the actor locally on your computer and test that it works:
- If everything works fine, upload the actor to the Apify platform, so that you can run it in the cloud:
And that's it!
If you have any problem or anything does not work, please file an issue on GitHub.
See how Scrapy Executor is used in industries around the world