🔓 Universal Web Unblocker | AI Web Scraper Engine

What does Universal Web Unblocker & AI Extractor do?

Universal Web Unblocker & AI Extractor is an all-in-one web unblocking, listing navigation, and structured data extraction tool that unblocks any webpage URL and extracts clean AI Markdown, verified Raw HTML, Hydration JSON (__NEXT_DATA__ / Schema.org), Website Tech Stack, Custom CSS Rules, and 12 automatic structured schemas (product, productList, productNavigation, jobPosting, jobPostingList, jobPostingNavigation, realEstate, realEstateList, reviews, reviewList, contactInfo, businessLead, socialProfile, article, articleList, articleNavigation, forumThread, serp, and pageContent)—with built-in residential proxies, automatic multi-page pagination, and zero proxy setup required.

Unlike traditional web unlockers that charge a 10x–25x credit multiplier whenever JavaScript rendering or residential proxies are needed, Universal Web Unblocker & AI Extractor includes automatic anti-bot challenge solving, residential proxy rotation, and DOM hydration verification in a single flat base rate (from $1.20 / 1,000 pages on Business+ plans), with optional pay-on-success add-ons for 12-category structured JSON extraction and 7,291-technology detection.

🛡️ Zero Proxy Configuration Required: Anti-blocking, browser escalation, multi-page listing pagination, and regional proxy routing are handled automatically out of the box — just paste your target URLs and click Start.

What data can you extract with Universal Web Unblocker & AI Extractor?

🔗 Target URL & Verified Status (url, statusCode) — canonical webpage URL and verified HTTP status code after anti-bot solving and DOM integrity checks
🏷️ Detected Page Category (autoExtractedCategory) — automatically classified page archetype (product, productList, productNavigation, jobPosting, realEstate, reviews, article, pageContent)
📝 Clean AI Markdown (markdown) — token-efficient Markdown with navigation menus, cookie consent banners, ad scripts, and SVG clutter removed
🌐 Raw Unblocked HTML & Headers (html, httpResponseHeaders) — complete unblocked DOM document and HTTP response headers when enabled
⚡ Hydration, Network & Screenshot (hydrationData, networkCapture, screenshot) — extracted Next.js (__NEXT_DATA__), Nuxt (__NUXT_DATA__), Schema.org JSON-LD state, background XHR/GraphQL API calls, and base64 JPEG page screenshots
🛒 E-Commerce Products, Grids & Navigation (product, productList, productNavigation) — product name, price, currency, SKU/ASIN, brand, availability, high-resolution images, feature bullets, technical specifications, listing grids, category items links, subCategories, and nextPage pagination URL
💼 Job Postings & Search Navigation (jobPosting, jobPostingList, jobPostingNavigation) — job title, hiring organization, location, salary range, employment type, date posted, job search listings, and nextPage pagination link
🏠 Real Estate Listings (realEstate, realEstateList) — property price, address, bedrooms, bathrooms, square footage, property type, coordinates, and property search results
⭐ Customer Reviews & Histograms (reviews, reviewList) — aggregate star rating, 5-star to 1-star rating histogram breakdown, individual customer review cards, and nextPage review pagination link
📧 Contacts, Business Leads & Social Profiles (contactInfo, businessLead, socialProfile) — email addresses, phone numbers, social media profile links, company name, address, opening hours, and social profile metadata
📰 Articles, Forums, SERP & Page Metadata (article, articleList, articleNavigation, forumThread, serp, pageContent) — article headline, author, publish date, blog index lists, forum discussion threads, search engine organic results, and universal page metadata
🎯 Custom CSS Rules & Multi-Page Crawling (customData, pagesCrawled, paginatedUrls, detailPages) — custom CSS/attribute extraction results, automatic multi-page pagination summaries, and two-hop detail page schemas
🔬 Website Tech Stack (techStack) — detected CMS, e-commerce platforms, JavaScript frameworks, analytics pixels, and CDNs across 7,291 technology signatures
📊 Execution & Add-On Telemetry (structuredExtractionApplied, techStackApplied, telemetry) — transparent billing flags and execution metrics (extractFromUsed, matchedBlueprint, countryUsed, durationMs)

Why use Universal Web Unblocker & AI Extractor?

🛡️ All-inclusive anti-bot & residential unblocking — automatically navigates modern bot protection and CAPTCHA challenges using coherent TLS profiles, residential proxy rotation, and browser rendering with no 25x browser credit multiplier
🧭 Built-in listing navigation, auto-pagination & 2-hop detail crawling — automatically extracts repeating listing cards (productList, jobPostingList, realEstateList), follows nextPage links up to maxPages, and optionally crawls into each item's detail URL (followDetailLinks)
🔍 Multi-signal Soft-200 & empty SPA shell detection — never returns or bills for fake 200 OK block pages or unhydrated <div id="root"></div> Single-Page Application shells; automatically waits for MutationObserver DOM stability
🧠 12-category deterministic auto-extractor + custom CSS rules ($0.00 LLM token cost) — extracts clean structured JSON across e-commerce, jobs, real estate, reviews, contacts, and articles in < 3ms without slow or hallucinated LLM calls
⚡ Ultra-low compute footprint (128 MB RAM) — runs inside a lightweight 128 MB container so your Apify platform compute unit (CU) consumption stays near zero

How do I use Universal Web Unblocker & AI Extractor?

1. Click Try for free on the Actor page to open Apify Console.
2. Paste one or more webpage or category listing URLs into Target URLs (startUrls).
3. Leave Include Clean AI Markdown (includeMarkdown) checked, or enable Include Raw Unblocked HTML (includeHtml), Capture Background XHR / GraphQL API Calls (networkCapture), or Capture Page Screenshot (screenshot).
4. Optionally check Auto-Detect & Extract Structured Schema (autoExtract), specific category checkboxes (product, productList, productNavigation, reviews, jobPosting, realEstate, contactInfo, includeTechStack), or define Custom CSS / Attribute Extraction Rules (extractRules).
5. For listing/search URLs, optionally enable Auto-Paginate Listing Pages (autoPaginate), Two-Hop Listing -> Detail Crawling (followDetailLinks), or set Dataset Output Mode (outputMode) to "items" to emit one row per listing card.
6. Click Start and download your results from the Dataset tab in JSON, CSV, or Excel format.

Input

Click on the Input tab in Apify Console to configure your extraction parameters:

{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
    }
  ],
  "maxItems": 25,
  "includeMarkdown": true,
  "includeHtml": false,
  "product": true,
  "reviews": true,
  "country": "us",
  "device": "desktop"
}

🌐 startUrls — list of webpage or listing URLs to unblock and extract
🔢 maxItems — maximum number of dataset items to output per run (default: 25; Free plan runs are capped at 20 items per run)
📝 includeMarkdown — convert the unblocked webpage into clean, token-efficient Markdown (default: true, included in base rate)
📄 includeHtml — include the full unblocked HTML document string (html) in the output dataset (default: false, included in base rate)
⚡ includeHydration — extract embedded Next.js (__NEXT_DATA__), Nuxt, Apollo, and Schema.org JSON-LD state (default: false)
📋 httpResponseHeaders — include the target server's HTTP response headers in the output (default: false)
🕸️ networkCapture — capture background XHR, Fetch, and GraphQL JSON network requests fired during browser rendering (default: false)
📸 screenshot — capture a base64-encoded JPEG screenshot of the rendered webpage (default: false)
🧠 autoExtract — automatically classify the page and extract structured JSON across all 12 categories (optional structured-extraction add-on)
🛒 product — extract e-commerce product details, price, SKU/ASIN, brand, images, features, specifications, and rating (optional structured-extraction add-on)
🛍️ productList — extract all product cards from a category, search, or collection page, plus nextPage URL (optional structured-extraction add-on)
🧭 productNavigation — extract category name, detail item links (items), sub-category links (subCategories), breadcrumbs, and nextPage URL (optional structured-extraction add-on)
💼 jobPosting — extract single job posting details including title, company, location, and salary (optional structured-extraction add-on)
📋 jobPostingList — extract multiple job postings from a job board search or careers page (optional structured-extraction add-on)
🗺️ jobPostingNavigation — extract job detail links (items), filter links (subCategories), and nextPage pagination URL (optional structured-extraction add-on)
🏠 realEstate — extract single property listing details including price, address, beds, baths, and sqft (optional structured-extraction add-on)
🏘️ realEstateList — extract multiple property listings from a real estate search page (optional structured-extraction add-on)
⭐ reviews — extract aggregate rating, 5-star to 1-star rating histogram, and customer reviews (optional structured-extraction add-on)
🌟 reviewList — extract paginated customer review cards, rating histogram, and nextPage link (optional structured-extraction add-on)
📧 contactInfo — extract email addresses, phone numbers, and social media links found on the page (optional structured-extraction add-on)
🏢 businessLead — extract local business / company profile details (optional structured-extraction add-on)
👤 socialProfile — extract public social media profile metadata and embedded post cards (optional structured-extraction add-on)
📰 article — extract news or blog article headline, author, publish date, and body text (optional structured-extraction add-on)
📚 articleList — extract article cards and nextPage link from a news section or blog index page (optional structured-extraction add-on)
🗂️ articleNavigation — extract article links (items), section links (subCategories), and nextPage URL (optional structured-extraction add-on)
💬 forumThread — extract forum discussion topic, original post, and replies (optional structured-extraction add-on)
🔎 serp — extract organic search engine result rankings, titles, URLs, and nextPage link (optional structured-extraction add-on)
📑 pageContent — extract universal page title, meta description, canonical URL, headings, and links (optional structured-extraction add-on)
🎯 extractRules — optional JSON map of custom field names to CSS selectors / @attr extractors (optional structured-extraction add-on)
🔁 autoPaginate — automatically follow nextPage links across multiple listing/category/review pages up to maxPages (default: false)
📖 maxPages — maximum number of pagination pages to follow per listing URL when autoPaginate is enabled (default: 1)
🔗 followDetailLinks — automatically follow discovered listing links and extract full detail schemas from each detail page (default: false)
📦 outputMode — dataset output mode: "page" (1 row per URL envelope) or "items" (1 row per extracted listing card)
🔬 includeTechStack — detect website technologies across 7,291 CMS, e-commerce, framework, pixel, and CDN signatures (optional tech-stack-scan add-on)
⚙️ extractFrom — execution mode: "auto" (recommended), "httpResponseBody", or "browserHtml"
🌍 country — 2-letter ISO country code for proxy geo-targeting (e.g. "us", "gb", "de") or "auto"
📱 device — emulate a "desktop" or "mobile" browser/TLS profile (default: "desktop")
⏳ waitSelector — optional CSS selector to wait for when rendering dynamic SPA pages
🖱️ clickSelector — optional CSS selector to click before extraction (e.g. cookie banner or 'Load More' button)
📜 scrollCount — number of viewports (0–15) to scroll down before extraction to trigger lazy-loaded grids
⏱️ waitMs — optional extra wait time in milliseconds (0–10000) after hydration before extracting
🔑 sessionId — optional sticky session identifier to reuse the same proxy IP across requests
📨 customHeaders — optional JSON object of custom HTTP request headers
🍪 cookies — optional JSON object of custom request cookies

Output sample

Results are streamed in real time to your default Apify Dataset:

[
  {
    "url": "https://www.amazon.com/dp/B0HG17QF13",
    "statusCode": 200,
    "autoExtractedCategory": "product",
    "markdown": "# 2026 Laptop, 16inch Laptop Computer with Core i7-6500U Processor 16GB RAM DDR3 512GB SSD\n\n- **Brand:** DUNHOO\n- **Price:** $412.99 USD",
    "product": {
      "url": "https://www.amazon.com/dp/B0HG17QF13",
      "name": "2026 Laptop, 16inch Laptop Computer with Core i7-6500U Processor 16GB RAM DDR3 512GB SSD, 1920*1200 Display, Win 11 Pro",
      "sku": "B0HG17QF13",
      "brand": "DUNHOO",
      "price": "$412.99",
      "currency": "USD",
      "availability": "In Stock",
      "images": [
        "https://m.media-amazon.com/images/I/71wF7x1gJGL._AC_SL1500_.jpg"
      ],
      "features": [
        "Powerful Core i7-6500U Processor (up to 3.1GHz) with 16GB RAM and 512GB SSD"
      ],
      "specifications": {
        "Standing screen display size": "16 Inches",
        "Screen Resolution": "1920 x 1200 pixels",
        "RAM": "16 GB DDR3",
        "Hard Drive": "512 GB SSD"
      },
      "aggregateRating": {
        "ratingValue": "4.4 out of 5",
        "reviewCount": "5 global ratings",
        "bestRating": "5"
      }
    },
    "reviews": {
      "url": "https://www.amazon.com/dp/B0HG17QF13",
      "aggregateRating": {
        "ratingValue": "4.4 out of 5",
        "reviewCount": "5 global ratings",
        "bestRating": "5"
      },
      "ratingHistogram": {
        "5_star_percent": 70,
        "4_star_percent": 0,
        "3_star_percent": 30,
        "2_star_percent": 0,
        "1_star_percent": 0
      },
      "reviews": []
    },
    "structuredExtractionApplied": true,
    "techStackApplied": false,
    "telemetry": {
      "strategyCombinationsAvailable": 627480,
      "extractFromUsed": "httpResponseBody",
      "matchedBlueprint": "amazon-products-scraper@v5",
      "learnedDomainProfileHit": false,
      "extractionSource": "blueprint:amazon-products-scraper",
      "countryUsed": "us",
      "durationMs": 6420
    }
  }
]

How much does Universal Web Unblocker & AI Extractor cost?

Universal Web Unblocker & AI Extractor uses Pay-Per-Event (PPE) pricing with a $0.00 Actor start fee—you only pay for items successfully unblocked (statusCode == 200) and delivered to your dataset:

🆓 Free plan (result-item) — $0.005 per item ($5.00 / 1,000 items, capped at 20 items/run)
🥉 Starter (Bronze) plan (result-item) — $0.0022 per item ($2.20 / 1,000 items)
🥈 Scale (Silver) plan (result-item) — $0.0016 per item ($1.60 / 1,000 items)
🥇 Business+ (Gold) plan (result-item) — $0.0012 per item ($1.20 / 1,000 items)

🧠 Optional Add-On Event: structured-extraction (charged only when you enable autoExtract, a structured schema toggle, or extractRules and non-empty structured JSON is returned):
🆓 Free: $0.0015 ($1.50/1K) · 🥉 Bronze: $0.0010 ($1.00/1K) · 🥈 Silver: $0.00075 ($0.75/1K) · 🥇 Gold: $0.0005 ($0.50/1K)

🔬 Optional Add-On Event: tech-stack-scan (charged only when you enable includeTechStack: true and website technologies are detected):
🆓 Free: $0.0015 ($1.50/1K) · 🥉 Bronze: $0.0010 ($1.00/1K) · 🥈 Silver: $0.00075 ($0.75/1K) · 🥇 Gold: $0.0005 ($0.50/1K)

Want to extract website contacts, tech stacks, or Google Maps leads?

Combine Universal Web Unblocker & AI Extractor with our companion B2B enrichment and domain intelligence Actors:

📧 Website Contact & Email Scraper — crawl company websites and internal /contact pages for verified emails, phone numbers, and LinkedIn/social profiles
🛠️ Website Tech Stack, Contacts & AI Markdown Scanner — detect 7,000+ technologies, ad pixels, CMS platforms, and contacts across any domain list
📄 Website to Clean Text & Markdown Crawler — bulk convert documentation and marketing pages into clean RAG Markdown

Integrations and Universal Web Unblocker & AI Extractor

Connect Universal Web Unblocker & AI Extractor directly to Make, n8n, Zapier, Clay, LangChain, LlamaIndex, Google Sheets, Slack, or Amazon S3 via Apify integrations and webhooks, or call it programmatically via the official apify-client in Python and Node.js.

Frequently asked questions

🛡️ Do I need to configure or pay extra for Apify Residential Proxies?
No. Regional residential and ISP proxy rotation and anti-bot challenge solving are built in and included in the flat result-item rate.

🧭 How do I scrape multiple pages of listings from a category or search URL?
Enable productList (or jobPostingList / realEstateList / reviewList), check autoPaginate: true, and set maxPages (e.g. 5). If you want each product or job emitted as its own row in your Dataset, set outputMode to "items".

⚡ Am I charged extra if a website requires browser JavaScript rendering?
No. Automatic escalation from HTTP to full browser rendering when a site requires JavaScript or challenge solving is included in the base result-item event.

🧩 Can I extract multiple schemas (e.g., product + reviews + contactInfo) from the same URL?
Yes. You can enable multiple category checkboxes simultaneously on the same request, and the structured-extraction add-on event is charged at most once per output item.

⚖️ Is web scraping legal?
Our Actor only extracts publicly accessible web data and does not bypass private account logins. For more details on ethical and compliant web data extraction, read Apify's guide: Is Web Scraping Legal?.

Support and legal disclaimer

If you encounter any site-specific extraction issues or need a custom schema field added, please open a ticket in the Issues tab on this Actor's Apify Store page. This Actor is an independent web data extraction tool and is not affiliated with, endorsed by, or sponsored by any target website processed by the user.