Under maintenance

No credit card required

View all Actors

This Actor is under maintenance.

This Actor may be unreliable while under maintenance. Would you like to try a similar Actor instead?

See alternative Actors

ScrapyFy

jupri/scrapyfy

Try for free

No credit card required

Scrapy Runner

Spiders Code

spiders_codestringRequired

Define your spiders (multiple spiders can be specified).

AJAXCRAWL_ENABLED

AJAXCRAWL_ENABLEDbooleanOptional

Default value of this property is false

ASYNCIO_EVENT_LOOP

ASYNCIO_EVENT_LOOPstringOptional

AUTOTHROTTLE_DEBUG

AUTOTHROTTLE_DEBUGbooleanOptional

Default value of this property is false

AUTOTHROTTLE_ENABLED

AUTOTHROTTLE_ENABLEDbooleanOptional

Default value of this property is false

AUTOTHROTTLE_MAX_DELAY

AUTOTHROTTLE_MAX_DELAYintegerOptional

Default value of this property is 60

AUTOTHROTTLE_START_DELAY

AUTOTHROTTLE_START_DELAYintegerOptional

Default value of this property is 5

AUTOTHROTTLE_TARGET_CONCURRENCY

AUTOTHROTTLE_TARGET_CONCURRENCYintegerOptional

Default value of this property is 1

BOT_NAME

BOT_NAMEstringOptional

Default value of this property is "scrapybot"

CLOSESPIDER_ERRORCOUNT

CLOSESPIDER_ERRORCOUNTintegerOptional

Default value of this property is 0

CLOSESPIDER_ITEMCOUNT

CLOSESPIDER_ITEMCOUNTintegerOptional

Default value of this property is 0

CLOSESPIDER_PAGECOUNT

CLOSESPIDER_PAGECOUNTintegerOptional

Default value of this property is 0

CLOSESPIDER_TIMEOUT

CLOSESPIDER_TIMEOUTintegerOptional

Default value of this property is 0

COMMANDS_MODULE

COMMANDS_MODULEstringOptional

Default value of this property is ""

COMPRESSION_ENABLED

COMPRESSION_ENABLEDbooleanOptional

Default value of this property is true

CONCURRENT_ITEMS

CONCURRENT_ITEMSintegerOptional

Default value of this property is 100

CONCURRENT_REQUESTS

CONCURRENT_REQUESTSintegerOptional

Default value of this property is 16

CONCURRENT_REQUESTS_PER_DOMAIN

CONCURRENT_REQUESTS_PER_DOMAINintegerOptional

Default value of this property is 8

CONCURRENT_REQUESTS_PER_IP

CONCURRENT_REQUESTS_PER_IPintegerOptional

Default value of this property is 0

COOKIES_DEBUG

COOKIES_DEBUGbooleanOptional

Default value of this property is false

COOKIES_ENABLED

COOKIES_ENABLEDbooleanOptional

Default value of this property is true

DEFAULT_ITEM_CLASS

DEFAULT_ITEM_CLASSstringOptional

Default value of this property is "scrapy.item.Item"

DEFAULT_REQUEST_HEADERS

DEFAULT_REQUEST_HEADERSobjectOptional

DEPTH_LIMIT

DEPTH_LIMITintegerOptional

Default value of this property is 0

DEPTH_PRIORITY

DEPTH_PRIORITYintegerOptional

Default value of this property is 0

DEPTH_STATS_VERBOSE

DEPTH_STATS_VERBOSEbooleanOptional

Default value of this property is false

DNSCACHE_ENABLED

DNSCACHE_ENABLEDbooleanOptional

Default value of this property is true

DNSCACHE_SIZE

DNSCACHE_SIZEintegerOptional

Default value of this property is 10000

DNS_RESOLVER

DNS_RESOLVERstringOptional

Default value of this property is "scrapy.resolver.CachingThreadedResolver"

DNS_TIMEOUT

DNS_TIMEOUTintegerOptional

Default value of this property is 60

DOWNLOADER

DOWNLOADERstringOptional

Default value of this property is "scrapy.core.downloader.Downloader"

DOWNLOADER_CLIENTCONTEXTFACTORY

DOWNLOADER_CLIENTCONTEXTFACTORYstringOptional

Default value of this property is "scrapy.core.downloader.contextfactory.ScrapyClientContextFactory"

DOWNLOADER_CLIENT_TLS_CIPHERS

DOWNLOADER_CLIENT_TLS_CIPHERSstringOptional

Default value of this property is "DEFAULT"

DOWNLOADER_CLIENT_TLS_METHOD

DOWNLOADER_CLIENT_TLS_METHODstringOptional

Default value of this property is "TLS"

DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING

DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGINGbooleanOptional

Default value of this property is false

DOWNLOADER_HTTPCLIENTFACTORY

DOWNLOADER_HTTPCLIENTFACTORYstringOptional

Default value of this property is "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"

DOWNLOADER_MIDDLEWARES

DOWNLOADER_MIDDLEWARESobjectOptional

DOWNLOADER_MIDDLEWARES_BASE

DOWNLOADER_MIDDLEWARES_BASEobjectOptional

DOWNLOADER_STATS

DOWNLOADER_STATSbooleanOptional

Default value of this property is true

DOWNLOAD_DELAY

DOWNLOAD_DELAYintegerOptional

Default value of this property is 0

DOWNLOAD_FAIL_ON_DATALOSS

DOWNLOAD_FAIL_ON_DATALOSSbooleanOptional

Default value of this property is true

DOWNLOAD_HANDLERS

DOWNLOAD_HANDLERSobjectOptional

DOWNLOAD_HANDLERS_BASE

DOWNLOAD_HANDLERS_BASEobjectOptional

DOWNLOAD_MAXSIZE

DOWNLOAD_MAXSIZEintegerOptional

Default value of this property is 1073741824

DOWNLOAD_TIMEOUT

DOWNLOAD_TIMEOUTintegerOptional

Default value of this property is 180

DOWNLOAD_WARNSIZE

DOWNLOAD_WARNSIZEintegerOptional

Default value of this property is 33554432

DUPEFILTER_CLASS

DUPEFILTER_CLASSstringOptional

Default value of this property is "scrapy.dupefilters.RFPDupeFilter"

EDITOR

EDITORstringOptional

Default value of this property is "vi"

EXTENSIONS

EXTENSIONSobjectOptional

EXTENSIONS_BASE

EXTENSIONS_BASEobjectOptional

FEEDS

FEEDSobjectOptional

FEED_EXPORTERS

FEED_EXPORTERSobjectOptional

FEED_EXPORTERS_BASE

FEED_EXPORTERS_BASEobjectOptional

FEED_EXPORT_BATCH_ITEM_COUNT

FEED_EXPORT_BATCH_ITEM_COUNTintegerOptional

Default value of this property is 0

FEED_EXPORT_ENCODING

FEED_EXPORT_ENCODINGstringOptional

FEED_EXPORT_FIELDS

FEED_EXPORT_FIELDSstringOptional

FEED_EXPORT_INDENT

FEED_EXPORT_INDENTintegerOptional

Default value of this property is 0

FEED_STORAGES

FEED_STORAGESobjectOptional

FEED_STORAGES_BASE

FEED_STORAGES_BASEobjectOptional

FEED_STORAGE_FTP_ACTIVE

FEED_STORAGE_FTP_ACTIVEbooleanOptional

Default value of this property is false

FEED_STORAGE_GCS_ACL

FEED_STORAGE_GCS_ACLstringOptional

Default value of this property is ""

FEED_STORAGE_S3_ACL

FEED_STORAGE_S3_ACLstringOptional

Default value of this property is ""

FEED_STORE_EMPTY

FEED_STORE_EMPTYbooleanOptional

Default value of this property is false

FEED_TEMPDIR

FEED_TEMPDIRstringOptional

FEED_URI_PARAMS

FEED_URI_PARAMSstringOptional

FILES_STORE_GCS_ACL

FILES_STORE_GCS_ACLstringOptional

Default value of this property is ""

FILES_STORE_S3_ACL

FILES_STORE_S3_ACLstringOptional

Default value of this property is "private"

FTP_PASSIVE_MODE

FTP_PASSIVE_MODEbooleanOptional

Default value of this property is true

FTP_PASSWORD

FTP_PASSWORDstringOptional

Default value of this property is "guest"

FTP_USER

FTP_USERstringOptional

Default value of this property is "anonymous"

GCS_PROJECT_ID

GCS_PROJECT_IDstringOptional

HTTPCACHE_ALWAYS_STORE

HTTPCACHE_ALWAYS_STOREbooleanOptional

Default value of this property is false

HTTPCACHE_DBM_MODULE

HTTPCACHE_DBM_MODULEstringOptional

Default value of this property is "dbm"

HTTPCACHE_DIR

HTTPCACHE_DIRstringOptional

Default value of this property is "httpcache"

HTTPCACHE_ENABLED

HTTPCACHE_ENABLEDbooleanOptional

Default value of this property is false

HTTPCACHE_EXPIRATION_SECS

HTTPCACHE_EXPIRATION_SECSintegerOptional

Default value of this property is 0

HTTPCACHE_GZIP

HTTPCACHE_GZIPbooleanOptional

Default value of this property is false

HTTPCACHE_IGNORE_HTTP_CODES

HTTPCACHE_IGNORE_HTTP_CODESarrayOptional

HTTPCACHE_IGNORE_MISSING

HTTPCACHE_IGNORE_MISSINGbooleanOptional

Default value of this property is false

HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS

HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLSarrayOptional

HTTPCACHE_IGNORE_SCHEMES

HTTPCACHE_IGNORE_SCHEMESarrayOptional

HTTPCACHE_POLICY

HTTPCACHE_POLICYstringOptional

Default value of this property is "scrapy.extensions.httpcache.DummyPolicy"

HTTPCACHE_STORAGE

HTTPCACHE_STORAGEstringOptional

Default value of this property is "scrapy.extensions.httpcache.FilesystemCacheStorage"

HTTPPROXY_AUTH_ENCODING

HTTPPROXY_AUTH_ENCODINGstringOptional

Default value of this property is "latin-1"

HTTPPROXY_ENABLED

HTTPPROXY_ENABLEDbooleanOptional

Default value of this property is true

IMAGES_STORE_GCS_ACL

IMAGES_STORE_GCS_ACLstringOptional

Default value of this property is ""

IMAGES_STORE_S3_ACL

IMAGES_STORE_S3_ACLstringOptional

Default value of this property is "private"

ITEM_PIPELINES

ITEM_PIPELINESobjectOptional

ITEM_PIPELINES_BASE

ITEM_PIPELINES_BASEobjectOptional

ITEM_PROCESSOR

ITEM_PROCESSORstringOptional

Default value of this property is "scrapy.pipelines.ItemPipelineManager"

LOGSTATS_INTERVAL

LOGSTATS_INTERVALintegerOptional

Default value of this property is 60

LOG_DATEFORMAT

LOG_DATEFORMATstringOptional

Default value of this property is "%Y-%m-%d %H:%M:%S"

LOG_ENABLED

LOG_ENABLEDbooleanOptional

Default value of this property is true

LOG_ENCODING

LOG_ENCODINGstringOptional

Default value of this property is "utf-8"

LOG_FILE

LOG_FILEstringOptional

LOG_FILE_APPEND

LOG_FILE_APPENDbooleanOptional

Default value of this property is true

LOG_FORMAT

LOG_FORMATstringOptional

Default value of this property is "%(asctime)s [%(name)s] %(levelname)s: %(message)s"

LOG_FORMATTER

LOG_FORMATTERstringOptional

Default value of this property is "scrapy.logformatter.LogFormatter"

LOG_LEVEL

LOG_LEVELstringOptional

Default value of this property is "DEBUG"

LOG_SHORT_NAMES

LOG_SHORT_NAMESbooleanOptional

Default value of this property is false

LOG_STDOUT

LOG_STDOUTbooleanOptional

Default value of this property is false

MAIL_FROM

MAIL_FROMstringOptional

Default value of this property is "scrapy@localhost"

MAIL_HOST

MAIL_HOSTstringOptional

Default value of this property is "localhost"

MAIL_PASS

MAIL_PASSstringOptional

MAIL_PORT

MAIL_PORTintegerOptional

Default value of this property is 25

MAIL_USER

MAIL_USERstringOptional

MEMDEBUG_ENABLED

MEMDEBUG_ENABLEDbooleanOptional

Default value of this property is false

MEMDEBUG_NOTIFY

MEMDEBUG_NOTIFYarrayOptional

MEMUSAGE_CHECK_INTERVAL_SECONDS

MEMUSAGE_CHECK_INTERVAL_SECONDSintegerOptional

Default value of this property is 60

MEMUSAGE_ENABLED

MEMUSAGE_ENABLEDbooleanOptional

Default value of this property is true

MEMUSAGE_LIMIT_MB

MEMUSAGE_LIMIT_MBintegerOptional

Default value of this property is 0

MEMUSAGE_NOTIFY_MAIL

MEMUSAGE_NOTIFY_MAILarrayOptional

MEMUSAGE_WARNING_MB

MEMUSAGE_WARNING_MBintegerOptional

Default value of this property is 0

METAREFRESH_ENABLED

METAREFRESH_ENABLEDbooleanOptional

Default value of this property is true

METAREFRESH_IGNORE_TAGS

METAREFRESH_IGNORE_TAGSarrayOptional

METAREFRESH_MAXDELAY

METAREFRESH_MAXDELAYintegerOptional

Default value of this property is 100

NEWSPIDER_MODULE

NEWSPIDER_MODULEstringOptional

Default value of this property is ""

RANDOMIZE_DOWNLOAD_DELAY

RANDOMIZE_DOWNLOAD_DELAYbooleanOptional

Default value of this property is true

REACTOR_THREADPOOL_MAXSIZE

REACTOR_THREADPOOL_MAXSIZEintegerOptional

Default value of this property is 10

REDIRECT_ENABLED

REDIRECT_ENABLEDbooleanOptional

Default value of this property is true

REDIRECT_MAX_TIMES

REDIRECT_MAX_TIMESintegerOptional

Default value of this property is 20

REDIRECT_PRIORITY_ADJUST

REDIRECT_PRIORITY_ADJUSTintegerOptional

Default value of this property is 2

REFERER_ENABLED

REFERER_ENABLEDbooleanOptional

Default value of this property is true

REFERRER_POLICY

REFERRER_POLICYstringOptional

Default value of this property is "scrapy.spidermiddlewares.referer.DefaultReferrerPolicy"

REQUEST_FINGERPRINTER_CLASS

REQUEST_FINGERPRINTER_CLASSstringOptional

Default value of this property is "scrapy.utils.request.RequestFingerprinter"

REQUEST_FINGERPRINTER_IMPLEMENTATION

REQUEST_FINGERPRINTER_IMPLEMENTATIONstringOptional

Default value of this property is "2.6"

RETRY_ENABLED

RETRY_ENABLEDbooleanOptional

Default value of this property is true

RETRY_HTTP_CODES

RETRY_HTTP_CODESarrayOptional

RETRY_PRIORITY_ADJUST

RETRY_PRIORITY_ADJUSTintegerOptional

Default value of this property is -1

RETRY_TIMES

RETRY_TIMESintegerOptional

Default value of this property is 2

ROBOTSTXT_OBEY

ROBOTSTXT_OBEYbooleanOptional

Default value of this property is false

ROBOTSTXT_PARSER

ROBOTSTXT_PARSERstringOptional

Default value of this property is "scrapy.robotstxt.ProtegoRobotParser"

ROBOTSTXT_USER_AGENT

ROBOTSTXT_USER_AGENTstringOptional

SCHEDULER

SCHEDULERstringOptional

Default value of this property is "scrapy.core.scheduler.Scheduler"

SCHEDULER_DEBUG

SCHEDULER_DEBUGbooleanOptional

Default value of this property is false

SCHEDULER_DISK_QUEUE

SCHEDULER_DISK_QUEUEstringOptional

Default value of this property is "scrapy.squeues.PickleLifoDiskQueue"

SCHEDULER_MEMORY_QUEUE

SCHEDULER_MEMORY_QUEUEstringOptional

Default value of this property is "scrapy.squeues.LifoMemoryQueue"

SCHEDULER_PRIORITY_QUEUE

SCHEDULER_PRIORITY_QUEUEstringOptional

Default value of this property is "scrapy.pqueues.ScrapyPriorityQueue"

SCRAPER_SLOT_MAX_ACTIVE_SIZE

SCRAPER_SLOT_MAX_ACTIVE_SIZEintegerOptional

Default value of this property is 5000000

SPIDER_CONTRACTS

SPIDER_CONTRACTSobjectOptional

SPIDER_CONTRACTS_BASE

SPIDER_CONTRACTS_BASEobjectOptional

SPIDER_LOADER_CLASS

SPIDER_LOADER_CLASSstringOptional

Default value of this property is "scrapy.spiderloader.SpiderLoader"

SPIDER_LOADER_WARN_ONLY

SPIDER_LOADER_WARN_ONLYbooleanOptional

Default value of this property is false

SPIDER_MIDDLEWARES

SPIDER_MIDDLEWARESobjectOptional

SPIDER_MIDDLEWARES_BASE

SPIDER_MIDDLEWARES_BASEobjectOptional

SPIDER_MODULES

SPIDER_MODULESarrayOptional

STATSMAILER_RCPTS

STATSMAILER_RCPTSarrayOptional

STATS_CLASS

STATS_CLASSstringOptional

Default value of this property is "scrapy.statscollectors.MemoryStatsCollector"

STATS_DUMP

STATS_DUMPbooleanOptional

Default value of this property is true

TELNETCONSOLE_ENABLED

TELNETCONSOLE_ENABLEDintegerOptional

Default value of this property is 1

TELNETCONSOLE_HOST

TELNETCONSOLE_HOSTstringOptional

Default value of this property is "127.0.0.1"

TELNETCONSOLE_PASSWORD

TELNETCONSOLE_PASSWORDstringOptional

TELNETCONSOLE_PORT

TELNETCONSOLE_PORTarrayOptional

TELNETCONSOLE_USERNAME

TELNETCONSOLE_USERNAMEstringOptional

Default value of this property is "scrapy"

TEMPLATES_DIR

TEMPLATES_DIRstringOptional

Default value of this property is "/usr/local/lib/python3.10/dist-packages/scrapy/templates"

TWISTED_REACTOR

TWISTED_REACTORstringOptional

URLLENGTH_LIMIT

URLLENGTH_LIMITintegerOptional

Default value of this property is 2083

USER_AGENT

USER_AGENTstringOptional

Default value of this property is "Scrapy/2.9.0 (+https://scrapy.org)"

Developer

cat

Actor metrics

2 monthly users
44.4% runs succeeded
days response time
Created in Jun 2023
Modified 11 months ago

Categories

Automation

Developer tools

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

Apify

63.4k

Website Content Crawler

apify/website-content-crawler

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

Apify

13.3k

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

4.3k

Content Checker

jakubbalada/content-checker

Monitor a website or web page for content changes. Automatically saves before and after screenshots and sends an email notification when content changes are detected.

Jakub Balada

2.1k

AI Web Agent

apify/ai-web-agent

Use natural language prompts to browse the web, click on elements, fill and submit forms, extract data, and take screenshots using the OpenAI API.

Apify

427

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

3.3k

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Lukáš Křivka

1.7k

Actor fail manager

lukaskrivka/actor-fail-manager

Automatically triggered on a failed run to analyze if the run should be resurrected and to create an error report for the author.

Lukáš Křivka

2.7k

WCC Pinecone Integration

tri_angle/wcc-pinecone-integration

Tri⟁angle

Pinecone GPT Chatbot

tri_angle/pinecone-gpt-chatbot

Pinecone GPT Chatbot combines OpenAI's GPT models with Pinecone's database to generate insightful responses. Its interactive chatbot interface presents precise and comprehensive answers to user queries. Benefit from semantic understanding, efficient workflows, and enriched knowledge integration!

Tri⟁angle

Scrapy vs. Beautiful Soup: which one to choose for web scraping

Alternatives to Scrapy for web scraping in 2024

Web scraping with Scrapy 101

Build new tools

Are you a developer? Build your own Actors and run them on Apify.

Learn more

Get a custom solution

Get a custom web scraping or RPA solution.

Book a demo