Deprecated

This Actor is deprecated

This Actor is unavailable because the developer has decided to deprecate it. Would you like to try a similar Actor instead?

See alternative Actors

ScrapyFy

jupri/scrapyfy

Scrapy Runner

Spiders Code

spiders_codestringRequired

Define your spiders (multiple spiders can be specified).

AJAXCRAWL_ENABLED

AJAXCRAWL_ENABLEDbooleanOptional

Default value of this property is false

ASYNCIO_EVENT_LOOP

ASYNCIO_EVENT_LOOPstringOptional

AUTOTHROTTLE_DEBUG

AUTOTHROTTLE_DEBUGbooleanOptional

Default value of this property is false

AUTOTHROTTLE_ENABLED

AUTOTHROTTLE_ENABLEDbooleanOptional

Default value of this property is false

AUTOTHROTTLE_MAX_DELAY

AUTOTHROTTLE_MAX_DELAYintegerOptional

Default value of this property is 60

AUTOTHROTTLE_START_DELAY

AUTOTHROTTLE_START_DELAYintegerOptional

Default value of this property is 5

AUTOTHROTTLE_TARGET_CONCURRENCY

AUTOTHROTTLE_TARGET_CONCURRENCYintegerOptional

Default value of this property is 1

BOT_NAME

BOT_NAMEstringOptional

Default value of this property is "scrapybot"

CLOSESPIDER_ERRORCOUNT

CLOSESPIDER_ERRORCOUNTintegerOptional

Default value of this property is 0

CLOSESPIDER_ITEMCOUNT

CLOSESPIDER_ITEMCOUNTintegerOptional

Default value of this property is 0

CLOSESPIDER_PAGECOUNT

CLOSESPIDER_PAGECOUNTintegerOptional

Default value of this property is 0

CLOSESPIDER_TIMEOUT

CLOSESPIDER_TIMEOUTintegerOptional

Default value of this property is 0

COMMANDS_MODULE

COMMANDS_MODULEstringOptional

Default value of this property is ""

COMPRESSION_ENABLED

COMPRESSION_ENABLEDbooleanOptional

Default value of this property is true

CONCURRENT_ITEMS

CONCURRENT_ITEMSintegerOptional

Default value of this property is 100

CONCURRENT_REQUESTS

CONCURRENT_REQUESTSintegerOptional

Default value of this property is 16

CONCURRENT_REQUESTS_PER_DOMAIN

CONCURRENT_REQUESTS_PER_DOMAINintegerOptional

Default value of this property is 8

CONCURRENT_REQUESTS_PER_IP

CONCURRENT_REQUESTS_PER_IPintegerOptional

Default value of this property is 0

COOKIES_DEBUG

COOKIES_DEBUGbooleanOptional

Default value of this property is false

COOKIES_ENABLED

COOKIES_ENABLEDbooleanOptional

Default value of this property is true

DEFAULT_ITEM_CLASS

DEFAULT_ITEM_CLASSstringOptional

Default value of this property is "scrapy.item.Item"

DEFAULT_REQUEST_HEADERS

DEFAULT_REQUEST_HEADERSobjectOptional

DEPTH_LIMIT

DEPTH_LIMITintegerOptional

Default value of this property is 0

DEPTH_PRIORITY

DEPTH_PRIORITYintegerOptional

Default value of this property is 0

DEPTH_STATS_VERBOSE

DEPTH_STATS_VERBOSEbooleanOptional

Default value of this property is false

DNSCACHE_ENABLED

DNSCACHE_ENABLEDbooleanOptional

Default value of this property is true

DNSCACHE_SIZE

DNSCACHE_SIZEintegerOptional

Default value of this property is 10000

DNS_RESOLVER

DNS_RESOLVERstringOptional

Default value of this property is "scrapy.resolver.CachingThreadedResolver"

DNS_TIMEOUT

DNS_TIMEOUTintegerOptional

Default value of this property is 60

DOWNLOADER

DOWNLOADERstringOptional

Default value of this property is "scrapy.core.downloader.Downloader"

DOWNLOADER_CLIENTCONTEXTFACTORY

DOWNLOADER_CLIENTCONTEXTFACTORYstringOptional

Default value of this property is "scrapy.core.downloader.contextfactory.ScrapyClientContextFactory"

DOWNLOADER_CLIENT_TLS_CIPHERS

DOWNLOADER_CLIENT_TLS_CIPHERSstringOptional

Default value of this property is "DEFAULT"

DOWNLOADER_CLIENT_TLS_METHOD

DOWNLOADER_CLIENT_TLS_METHODstringOptional

Default value of this property is "TLS"

DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING

DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGINGbooleanOptional

Default value of this property is false

DOWNLOADER_HTTPCLIENTFACTORY

DOWNLOADER_HTTPCLIENTFACTORYstringOptional

Default value of this property is "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"

DOWNLOADER_MIDDLEWARES

DOWNLOADER_MIDDLEWARESobjectOptional

DOWNLOADER_MIDDLEWARES_BASE

DOWNLOADER_MIDDLEWARES_BASEobjectOptional

DOWNLOADER_STATS

DOWNLOADER_STATSbooleanOptional

Default value of this property is true

DOWNLOAD_DELAY

DOWNLOAD_DELAYintegerOptional

Default value of this property is 0

DOWNLOAD_FAIL_ON_DATALOSS

DOWNLOAD_FAIL_ON_DATALOSSbooleanOptional

Default value of this property is true

DOWNLOAD_HANDLERS

DOWNLOAD_HANDLERSobjectOptional

DOWNLOAD_HANDLERS_BASE

DOWNLOAD_HANDLERS_BASEobjectOptional

DOWNLOAD_MAXSIZE

DOWNLOAD_MAXSIZEintegerOptional

Default value of this property is 1073741824

DOWNLOAD_TIMEOUT

DOWNLOAD_TIMEOUTintegerOptional

Default value of this property is 180

DOWNLOAD_WARNSIZE

DOWNLOAD_WARNSIZEintegerOptional

Default value of this property is 33554432

DUPEFILTER_CLASS

DUPEFILTER_CLASSstringOptional

Default value of this property is "scrapy.dupefilters.RFPDupeFilter"

EDITOR

EDITORstringOptional

Default value of this property is "vi"

EXTENSIONS

EXTENSIONSobjectOptional

EXTENSIONS_BASE

EXTENSIONS_BASEobjectOptional

FEEDS

FEEDSobjectOptional

FEED_EXPORTERS

FEED_EXPORTERSobjectOptional

FEED_EXPORTERS_BASE

FEED_EXPORTERS_BASEobjectOptional

FEED_EXPORT_BATCH_ITEM_COUNT

FEED_EXPORT_BATCH_ITEM_COUNTintegerOptional

Default value of this property is 0

FEED_EXPORT_ENCODING

FEED_EXPORT_ENCODINGstringOptional

FEED_EXPORT_FIELDS

FEED_EXPORT_FIELDSstringOptional

FEED_EXPORT_INDENT

FEED_EXPORT_INDENTintegerOptional

Default value of this property is 0

FEED_STORAGES

FEED_STORAGESobjectOptional

FEED_STORAGES_BASE

FEED_STORAGES_BASEobjectOptional

FEED_STORAGE_FTP_ACTIVE

FEED_STORAGE_FTP_ACTIVEbooleanOptional

Default value of this property is false

FEED_STORAGE_GCS_ACL

FEED_STORAGE_GCS_ACLstringOptional

Default value of this property is ""

FEED_STORAGE_S3_ACL

FEED_STORAGE_S3_ACLstringOptional

Default value of this property is ""

FEED_STORE_EMPTY

FEED_STORE_EMPTYbooleanOptional

Default value of this property is false

FEED_TEMPDIR

FEED_TEMPDIRstringOptional

FEED_URI_PARAMS

FEED_URI_PARAMSstringOptional

FILES_STORE_GCS_ACL

FILES_STORE_GCS_ACLstringOptional

Default value of this property is ""

FILES_STORE_S3_ACL

FILES_STORE_S3_ACLstringOptional

Default value of this property is "private"

FTP_PASSIVE_MODE

FTP_PASSIVE_MODEbooleanOptional

Default value of this property is true

FTP_PASSWORD

FTP_PASSWORDstringOptional

Default value of this property is "guest"

FTP_USER

FTP_USERstringOptional

Default value of this property is "anonymous"

GCS_PROJECT_ID

GCS_PROJECT_IDstringOptional

HTTPCACHE_ALWAYS_STORE

HTTPCACHE_ALWAYS_STOREbooleanOptional

Default value of this property is false

HTTPCACHE_DBM_MODULE

HTTPCACHE_DBM_MODULEstringOptional

Default value of this property is "dbm"

HTTPCACHE_DIR

HTTPCACHE_DIRstringOptional

Default value of this property is "httpcache"

HTTPCACHE_ENABLED

HTTPCACHE_ENABLEDbooleanOptional

Default value of this property is false

HTTPCACHE_EXPIRATION_SECS

HTTPCACHE_EXPIRATION_SECSintegerOptional

Default value of this property is 0

HTTPCACHE_GZIP

HTTPCACHE_GZIPbooleanOptional

Default value of this property is false

HTTPCACHE_IGNORE_HTTP_CODES

HTTPCACHE_IGNORE_HTTP_CODESarrayOptional

HTTPCACHE_IGNORE_MISSING

HTTPCACHE_IGNORE_MISSINGbooleanOptional

Default value of this property is false

HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS

HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLSarrayOptional

HTTPCACHE_IGNORE_SCHEMES

HTTPCACHE_IGNORE_SCHEMESarrayOptional

HTTPCACHE_POLICY

HTTPCACHE_POLICYstringOptional

Default value of this property is "scrapy.extensions.httpcache.DummyPolicy"

HTTPCACHE_STORAGE

HTTPCACHE_STORAGEstringOptional

Default value of this property is "scrapy.extensions.httpcache.FilesystemCacheStorage"

HTTPPROXY_AUTH_ENCODING

HTTPPROXY_AUTH_ENCODINGstringOptional

Default value of this property is "latin-1"

HTTPPROXY_ENABLED

HTTPPROXY_ENABLEDbooleanOptional

Default value of this property is true

IMAGES_STORE_GCS_ACL

IMAGES_STORE_GCS_ACLstringOptional

Default value of this property is ""

IMAGES_STORE_S3_ACL

IMAGES_STORE_S3_ACLstringOptional

Default value of this property is "private"

ITEM_PIPELINES

ITEM_PIPELINESobjectOptional

ITEM_PIPELINES_BASE

ITEM_PIPELINES_BASEobjectOptional

ITEM_PROCESSOR

ITEM_PROCESSORstringOptional

Default value of this property is "scrapy.pipelines.ItemPipelineManager"

LOGSTATS_INTERVAL

LOGSTATS_INTERVALintegerOptional

Default value of this property is 60

LOG_DATEFORMAT

LOG_DATEFORMATstringOptional

Default value of this property is "%Y-%m-%d %H:%M:%S"

LOG_ENABLED

LOG_ENABLEDbooleanOptional

Default value of this property is true

LOG_ENCODING

LOG_ENCODINGstringOptional

Default value of this property is "utf-8"

LOG_FILE

LOG_FILEstringOptional

LOG_FILE_APPEND

LOG_FILE_APPENDbooleanOptional

Default value of this property is true

LOG_FORMAT

LOG_FORMATstringOptional

Default value of this property is "%(asctime)s [%(name)s] %(levelname)s: %(message)s"

LOG_FORMATTER

LOG_FORMATTERstringOptional

Default value of this property is "scrapy.logformatter.LogFormatter"

LOG_LEVEL

LOG_LEVELstringOptional

Default value of this property is "DEBUG"

LOG_SHORT_NAMES

LOG_SHORT_NAMESbooleanOptional

Default value of this property is false

LOG_STDOUT

LOG_STDOUTbooleanOptional

Default value of this property is false

MAIL_FROM

MAIL_FROMstringOptional

Default value of this property is "scrapy@localhost"

MAIL_HOST

MAIL_HOSTstringOptional

Default value of this property is "localhost"

MAIL_PASS

MAIL_PASSstringOptional

MAIL_PORT

MAIL_PORTintegerOptional

Default value of this property is 25

MAIL_USER

MAIL_USERstringOptional

MEMDEBUG_ENABLED

MEMDEBUG_ENABLEDbooleanOptional

Default value of this property is false

MEMDEBUG_NOTIFY

MEMDEBUG_NOTIFYarrayOptional

MEMUSAGE_CHECK_INTERVAL_SECONDS

MEMUSAGE_CHECK_INTERVAL_SECONDSintegerOptional

Default value of this property is 60

MEMUSAGE_ENABLED

MEMUSAGE_ENABLEDbooleanOptional

Default value of this property is true

MEMUSAGE_LIMIT_MB

MEMUSAGE_LIMIT_MBintegerOptional

Default value of this property is 0

MEMUSAGE_NOTIFY_MAIL

MEMUSAGE_NOTIFY_MAILarrayOptional

MEMUSAGE_WARNING_MB

MEMUSAGE_WARNING_MBintegerOptional

Default value of this property is 0

METAREFRESH_ENABLED

METAREFRESH_ENABLEDbooleanOptional

Default value of this property is true

METAREFRESH_IGNORE_TAGS

METAREFRESH_IGNORE_TAGSarrayOptional

METAREFRESH_MAXDELAY

METAREFRESH_MAXDELAYintegerOptional

Default value of this property is 100

NEWSPIDER_MODULE

NEWSPIDER_MODULEstringOptional

Default value of this property is ""

RANDOMIZE_DOWNLOAD_DELAY

RANDOMIZE_DOWNLOAD_DELAYbooleanOptional

Default value of this property is true

REACTOR_THREADPOOL_MAXSIZE

REACTOR_THREADPOOL_MAXSIZEintegerOptional

Default value of this property is 10

REDIRECT_ENABLED

REDIRECT_ENABLEDbooleanOptional

Default value of this property is true

REDIRECT_MAX_TIMES

REDIRECT_MAX_TIMESintegerOptional

Default value of this property is 20

REDIRECT_PRIORITY_ADJUST

REDIRECT_PRIORITY_ADJUSTintegerOptional

Default value of this property is 2

REFERER_ENABLED

REFERER_ENABLEDbooleanOptional

Default value of this property is true

REFERRER_POLICY

REFERRER_POLICYstringOptional

Default value of this property is "scrapy.spidermiddlewares.referer.DefaultReferrerPolicy"

REQUEST_FINGERPRINTER_CLASS

REQUEST_FINGERPRINTER_CLASSstringOptional

Default value of this property is "scrapy.utils.request.RequestFingerprinter"

REQUEST_FINGERPRINTER_IMPLEMENTATION

REQUEST_FINGERPRINTER_IMPLEMENTATIONstringOptional

Default value of this property is "2.6"

RETRY_ENABLED

RETRY_ENABLEDbooleanOptional

Default value of this property is true

RETRY_HTTP_CODES

RETRY_HTTP_CODESarrayOptional

RETRY_PRIORITY_ADJUST

RETRY_PRIORITY_ADJUSTintegerOptional

Default value of this property is -1

RETRY_TIMES

RETRY_TIMESintegerOptional

Default value of this property is 2

ROBOTSTXT_OBEY

ROBOTSTXT_OBEYbooleanOptional

Default value of this property is false

ROBOTSTXT_PARSER

ROBOTSTXT_PARSERstringOptional

Default value of this property is "scrapy.robotstxt.ProtegoRobotParser"

ROBOTSTXT_USER_AGENT

ROBOTSTXT_USER_AGENTstringOptional

SCHEDULER

SCHEDULERstringOptional

Default value of this property is "scrapy.core.scheduler.Scheduler"

SCHEDULER_DEBUG

SCHEDULER_DEBUGbooleanOptional

Default value of this property is false

SCHEDULER_DISK_QUEUE

SCHEDULER_DISK_QUEUEstringOptional

Default value of this property is "scrapy.squeues.PickleLifoDiskQueue"

SCHEDULER_MEMORY_QUEUE

SCHEDULER_MEMORY_QUEUEstringOptional

Default value of this property is "scrapy.squeues.LifoMemoryQueue"

SCHEDULER_PRIORITY_QUEUE

SCHEDULER_PRIORITY_QUEUEstringOptional

Default value of this property is "scrapy.pqueues.ScrapyPriorityQueue"

SCRAPER_SLOT_MAX_ACTIVE_SIZE

SCRAPER_SLOT_MAX_ACTIVE_SIZEintegerOptional

Default value of this property is 5000000

SPIDER_CONTRACTS

SPIDER_CONTRACTSobjectOptional

SPIDER_CONTRACTS_BASE

SPIDER_CONTRACTS_BASEobjectOptional

SPIDER_LOADER_CLASS

SPIDER_LOADER_CLASSstringOptional

Default value of this property is "scrapy.spiderloader.SpiderLoader"

SPIDER_LOADER_WARN_ONLY

SPIDER_LOADER_WARN_ONLYbooleanOptional

Default value of this property is false

SPIDER_MIDDLEWARES

SPIDER_MIDDLEWARESobjectOptional

SPIDER_MIDDLEWARES_BASE

SPIDER_MIDDLEWARES_BASEobjectOptional

SPIDER_MODULES

SPIDER_MODULESarrayOptional

STATSMAILER_RCPTS

STATSMAILER_RCPTSarrayOptional

STATS_CLASS

STATS_CLASSstringOptional

Default value of this property is "scrapy.statscollectors.MemoryStatsCollector"

STATS_DUMP

STATS_DUMPbooleanOptional

Default value of this property is true

TELNETCONSOLE_ENABLED

TELNETCONSOLE_ENABLEDintegerOptional

Default value of this property is 1

TELNETCONSOLE_HOST

TELNETCONSOLE_HOSTstringOptional

Default value of this property is "127.0.0.1"

TELNETCONSOLE_PASSWORD

TELNETCONSOLE_PASSWORDstringOptional

TELNETCONSOLE_PORT

TELNETCONSOLE_PORTarrayOptional

TELNETCONSOLE_USERNAME

TELNETCONSOLE_USERNAMEstringOptional

Default value of this property is "scrapy"

TEMPLATES_DIR

TEMPLATES_DIRstringOptional

Default value of this property is "/usr/local/lib/python3.10/dist-packages/scrapy/templates"

TWISTED_REACTOR

TWISTED_REACTORstringOptional

URLLENGTH_LIMIT

URLLENGTH_LIMITintegerOptional

Default value of this property is 2083

USER_AGENT

USER_AGENTstringOptional

Default value of this property is "Scrapy/2.9.0 (+https://scrapy.org)"

Developer

cat

Categories

Automation

Developer tools

Scrapy Books Example

vdusek/scrapy-books-example

Example of Python Scrapy project. It scrapes book data from https://books.toscrape.com/.

Vlada Dusek

Crunchbase Scraper: Reliable & Easy

scrapefull/crunchbase-scraper

Our Crunchbase data extractor reliably extracts comprehensive organization data from Crunchbase pages. Get clean, organized Crunchbase data to identify insights and seamlessly integrate into your workflow.

Scrapefull

Apollo Scraper: Get Leads & Emails

scrapefull/apollo-scraper

Get more leads from Apollo.io without the hassle of export limits. Our scraper grabs 10k emails monthly, plus phone numbers and job titles even on the free plan. It's great for filling your CRM or starting new outreach campaigns. Save time and grow your business faster.

Scrapefull

LinkedIn Jobs Scraper: Easy & Fast

scrapefull/linkedin-jobs-scraper

Efficiently extract detailed job listings from LinkedIn Jobs, including job titles, company information, locations, and application details.

Scrapefull

Ultimate Reddit Profile Scraper

potatopeeler/reddit-scraper

Seamlessly download full Reddit user accounts, capturing posts, images, activity, and historical data, including URLs and media comments. Export detailed insights to CSV, JSON, XML, EXCEL formats, or effortlessly import them into your email for comprehensive analysis and easy access.

Jamie Potato

Ulta Review Crawler

webscrapewizard/my-actor

WebScraperWizard

Linkedin Url Scrape

mikepowers/linkedin-url-scrape

Scrape Unlimited Linkedin Profile URL's

Mike Powers

568

Youtube email finder

advisable_shampoo/youtube-email-finder

This scraper checks the link of channels through the database and provides an email if it exists in the database Attention; there should not be a "/" at the end of the channel link. for individual order; telegram @free_python_co email nizamovlutfulloh@gmail.com ❌ https://www.youtube.com/@Apify/

Nizamov Lutfulloh

400

Airbnb Properties details by location search in USD

surendra/airbnb-actor

Effortlessly find Airbnb property details for any location with our web scraper. Input your desired location and access comprehensive information on titles, prices, amenities, reviews & more, saving time and hassle. Discover your perfect rental unit with ease! *Note : Try to narrow down the location

Surendra Tamang

Google Maps Reviews Scraper

webscrapewizard/google-maps-reviews-scraper

Extract all reviews of Google Maps places using place URLs or Review urls. Get review text, published date, response from owner, review URL, and reviewer's details.

WebScraperWizard

Where next?

Build new tools

Are you a developer? Build your own Actors and run them on Apify.

Learn more

Get a custom solution

Get a custom web scraping or RPA solution.

Book a demo