Pricing

Pay per usage

Go to Store

SuperScraper API

Try for free

Developed by

Apify

Generic REST API for scraping websites: send a URL and get back HTML. This Actor is a drop-in replacement for ScrapingBee, ScrapingAnt, and ScraperAPI services. And it is open-source!

4.2 (6)

Pricing

Pay per usage

Monthly users

Runs succeeded

80%

Response time

74 days

Last modified

5 months ago

Developer tools

Open source

Back to issues Create new issue

How to manage page redirects

Closed

rocksoft opened this issue

ScrapingBee use Response Headers that are useful when the page redirects.

Spb-initial-status-code: The initial status code returned by the scraped page.
Spb-resolved-url: The resolved URL of the scraped page.

Pavlína Vencovská (pavlina.vencovska)

Hi,

thanks for reaching out, we'll look into it and let you know what can be done.

Radoslav Chudovský (rado.ch)

Hi, for now, you can get both the initial status code and resolved URL in the response by setting the json_response parameter to true.

Pavlína Vencovská (pavlina.vencovska)

Hi,

I'm going to close the issue for now, but if anything else come up, please, let us know.

Add comment

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

7.2k

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

6.2k

Crawl4AI

janbuchar/crawl4ai

Wraps the Crawl4AI open-source library for retrieving text content from websites.

Jan Buchar

167

Algolia Website Indexer

apify/algolia-website-indexer

The Indexer crawls recursively a website using the Puppeteer browser (headless Chrome) and indexes the selected pages to the Algolia index.

Apify

RAG Web Browser

apify/rag-web-browser

Web browser for OpenAI Assistants, RAG pipelines, or AI agents, similar to a web browser in ChatGPT. It queries Google Search, scrapes the top N pages from the results, and returns their cleaned content as Markdown for further processing by an LLM. It can also scrape individual URLs. Supports MCP.

Apify

1.8k

Python Example

apify/python-example

Example Actor written in Python, showing how to read the Actor input and push to the Actor's default dataset.

Apify

100

Web Scraper

apify/web-scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

80.7k

Legacy PhantomJS Crawler

apify/legacy-phantomjs-crawler

Replacement for the legacy Apify Crawler product with a backward-compatible interface. The actor uses PhantomJS headless browser to recursively crawl websites and extract data from them using a piece of front-end JavaScript code.

Apify

1.6k

Algolia Webcrawler

jancurn/algolia-webcrawler

Crawls a website using one or more sitemaps and imports the data to Algolia search index. The text content is identified using simple CSS selectors.

Jan Čurn

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

45.5k