# πŸ”“ Universal Web Unblocker | AI Web Scraper Engine (`misterkhan/universal-web-unblocker-extractor`) Actor Unblock any website and extract clean AI Markdown, HTML, and structured schemas from $1.20/1k results. - **URL**: https://apify.com/misterkhan/universal-web-unblocker-extractor.md - **Developed by:** [Khan](https://apify.com/misterkhan) (community) - **Categories:** AI, Developer tools, Automation - **Security:** Limited permissions - **Stats:** 2 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.20 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Universal Web Unblocker & AI Extractor do? Universal Web Unblocker & AI Extractor is an all-in-one web unblocking, listing navigation, and structured data extraction tool that unblocks any webpage URL and extracts **clean AI Markdown, verified Raw HTML, Hydration JSON (`__NEXT_DATA__` / Schema.org), Website Tech Stack, Custom CSS Rules, and 12 automatic structured schemas** (`product`, `productList`, `productNavigation`, `jobPosting`, `jobPostingList`, `jobPostingNavigation`, `realEstate`, `realEstateList`, `reviews`, `reviewList`, `contactInfo`, `businessLead`, `socialProfile`, `article`, `articleList`, `articleNavigation`, `forumThread`, `serp`, and `pageContent`)β€”with built-in residential proxies, automatic multi-page pagination, and zero proxy setup required. Unlike traditional web unlockers that charge a **10x–25x credit multiplier** whenever JavaScript rendering or residential proxies are needed, **Universal Web Unblocker & AI Extractor** includes automatic anti-bot challenge solving, residential proxy rotation, and DOM hydration verification in a single flat base rate (**from `$1.20 / 1,000 pages`** on Business+ plans), with optional pay-on-success add-ons for 12-category structured JSON extraction and 7,291-technology detection. > πŸ›‘οΈ **Zero Proxy Configuration Required:** Anti-blocking, browser escalation, multi-page listing pagination, and regional proxy routing are **handled automatically out of the box** β€” just paste your target URLs and click **Start**. ### What data can you extract with Universal Web Unblocker & AI Extractor? **πŸ”— Target URL & Verified Status (`url`, `statusCode`)** β€” canonical webpage URL and verified HTTP status code after anti-bot solving and DOM integrity checks
**🏷️ Detected Page Category (`autoExtractedCategory`)** β€” automatically classified page archetype (`product`, `productList`, `productNavigation`, `jobPosting`, `realEstate`, `reviews`, `article`, `pageContent`)
**πŸ“ Clean AI Markdown (`markdown`)** β€” token-efficient Markdown with navigation menus, cookie consent banners, ad scripts, and SVG clutter removed
**🌐 Raw Unblocked HTML & Headers (`html`, `httpResponseHeaders`)** β€” complete unblocked DOM document and HTTP response headers when enabled
**⚑ Hydration, Network & Screenshot (`hydrationData`, `networkCapture`, `screenshot`)** β€” extracted Next.js (`__NEXT_DATA__`), Nuxt (`__NUXT_DATA__`), Schema.org JSON-LD state, background XHR/GraphQL API calls, and base64 JPEG page screenshots
**πŸ›’ E-Commerce Products, Grids & Navigation (`product`, `productList`, `productNavigation`)** β€” product name, price, currency, SKU/ASIN, brand, availability, high-resolution images, feature bullets, technical specifications, listing grids, category `items` links, `subCategories`, and `nextPage` pagination URL
**πŸ’Ό Job Postings & Search Navigation (`jobPosting`, `jobPostingList`, `jobPostingNavigation`)** β€” job title, hiring organization, location, salary range, employment type, date posted, job search listings, and `nextPage` pagination link
**🏠 Real Estate Listings (`realEstate`, `realEstateList`)** β€” property price, address, bedrooms, bathrooms, square footage, property type, coordinates, and property search results
**⭐ Customer Reviews & Histograms (`reviews`, `reviewList`)** β€” aggregate star rating, 5-star to 1-star rating histogram breakdown, individual customer review cards, and `nextPage` review pagination link
**πŸ“§ Contacts, Business Leads & Social Profiles (`contactInfo`, `businessLead`, `socialProfile`)** β€” email addresses, phone numbers, social media profile links, company name, address, opening hours, and social profile metadata
**πŸ“° Articles, Forums, SERP & Page Metadata (`article`, `articleList`, `articleNavigation`, `forumThread`, `serp`, `pageContent`)** β€” article headline, author, publish date, blog index lists, forum discussion threads, search engine organic results, and universal page metadata
**🎯 Custom CSS Rules & Multi-Page Crawling (`customData`, `pagesCrawled`, `paginatedUrls`, `detailPages`)** β€” custom CSS/attribute extraction results, automatic multi-page pagination summaries, and two-hop detail page schemas
**πŸ”¬ Website Tech Stack (`techStack`)** β€” detected CMS, e-commerce platforms, JavaScript frameworks, analytics pixels, and CDNs across 7,291 technology signatures
**πŸ“Š Execution & Add-On Telemetry (`structuredExtractionApplied`, `techStackApplied`, `telemetry`)** β€” transparent billing flags and execution metrics (`extractFromUsed`, `matchedBlueprint`, `countryUsed`, `durationMs`)
### Why use Universal Web Unblocker & AI Extractor? **πŸ›‘οΈ All-inclusive anti-bot & residential unblocking** β€” automatically navigates modern bot protection and CAPTCHA challenges using coherent TLS profiles, residential proxy rotation, and browser rendering with **no 25x browser credit multiplier**
**🧭 Built-in listing navigation, auto-pagination & 2-hop detail crawling** β€” automatically extracts repeating listing cards (`productList`, `jobPostingList`, `realEstateList`), follows `nextPage` links up to `maxPages`, and optionally crawls into each item's detail URL (`followDetailLinks`)
**πŸ” Multi-signal Soft-200 & empty SPA shell detection** β€” never returns or bills for fake `200 OK` block pages or unhydrated `
` Single-Page Application shells; automatically waits for `MutationObserver` DOM stability
**🧠 12-category deterministic auto-extractor + custom CSS rules (`$0.00` LLM token cost)** β€” extracts clean structured JSON across e-commerce, jobs, real estate, reviews, contacts, and articles in `< 3ms` without slow or hallucinated LLM calls
**⚑ Ultra-low compute footprint (`128 MB` RAM)** β€” runs inside a lightweight `128 MB` container so your Apify platform compute unit (CU) consumption stays near zero
### How do I use Universal Web Unblocker & AI Extractor? 1. Click **Try for free** on the Actor page to open Apify Console. 2. Paste one or more webpage or category listing URLs into **Target URLs** (`startUrls`). 3. Leave **Include Clean AI Markdown** (`includeMarkdown`) checked, or enable **Include Raw Unblocked HTML** (`includeHtml`), **Capture Background XHR / GraphQL API Calls** (`networkCapture`), or **Capture Page Screenshot** (`screenshot`). 4. Optionally check **Auto-Detect & Extract Structured Schema** (`autoExtract`), specific category checkboxes (`product`, `productList`, `productNavigation`, `reviews`, `jobPosting`, `realEstate`, `contactInfo`, `includeTechStack`), or define **Custom CSS / Attribute Extraction Rules** (`extractRules`). 5. For listing/search URLs, optionally enable **Auto-Paginate Listing Pages** (`autoPaginate`), **Two-Hop Listing -> Detail Crawling** (`followDetailLinks`), or set **Dataset Output Mode** (`outputMode`) to `"items"` to emit one row per listing card. 6. Click **Start** and download your results from the **Dataset** tab in JSON, CSV, or Excel format. ### Input Click on the Input tab in Apify Console to configure your extraction parameters: ```json { "startUrls": [ { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" } ], "maxItems": 25, "includeMarkdown": true, "includeHtml": false, "product": true, "reviews": true, "country": "us", "device": "desktop" } ``` **🌐 `startUrls`** β€” list of webpage or listing URLs to unblock and extract
**πŸ”’ `maxItems`** β€” maximum number of dataset items to output per run (default: `25`; Free plan runs are capped at `20` items per run)
**πŸ“ `includeMarkdown`** β€” convert the unblocked webpage into clean, token-efficient Markdown (default: `true`, included in base rate)
**πŸ“„ `includeHtml`** β€” include the full unblocked HTML document string (`html`) in the output dataset (default: `false`, included in base rate)
**⚑ `includeHydration`** β€” extract embedded Next.js (`__NEXT_DATA__`), Nuxt, Apollo, and Schema.org JSON-LD state (default: `false`)
**πŸ“‹ `httpResponseHeaders`** β€” include the target server's HTTP response headers in the output (default: `false`)
**πŸ•ΈοΈ `networkCapture`** β€” capture background XHR, Fetch, and GraphQL JSON network requests fired during browser rendering (default: `false`)
**πŸ“Έ `screenshot`** β€” capture a base64-encoded JPEG screenshot of the rendered webpage (default: `false`)
**🧠 `autoExtract`** β€” automatically classify the page and extract structured JSON across all 12 categories (optional `structured-extraction` add-on)
**πŸ›’ `product`** β€” extract e-commerce product details, price, SKU/ASIN, brand, images, features, specifications, and rating (optional `structured-extraction` add-on)
**πŸ›οΈ `productList`** β€” extract all product cards from a category, search, or collection page, plus `nextPage` URL (optional `structured-extraction` add-on)
**🧭 `productNavigation`** β€” extract category name, detail item links (`items`), sub-category links (`subCategories`), breadcrumbs, and `nextPage` URL (optional `structured-extraction` add-on)
**πŸ’Ό `jobPosting`** β€” extract single job posting details including title, company, location, and salary (optional `structured-extraction` add-on)
**πŸ“‹ `jobPostingList`** β€” extract multiple job postings from a job board search or careers page (optional `structured-extraction` add-on)
**πŸ—ΊοΈ `jobPostingNavigation`** β€” extract job detail links (`items`), filter links (`subCategories`), and `nextPage` pagination URL (optional `structured-extraction` add-on)
**🏠 `realEstate`** β€” extract single property listing details including price, address, beds, baths, and sqft (optional `structured-extraction` add-on)
**🏘️ `realEstateList`** β€” extract multiple property listings from a real estate search page (optional `structured-extraction` add-on)
**⭐ `reviews`** β€” extract aggregate rating, 5-star to 1-star rating histogram, and customer reviews (optional `structured-extraction` add-on)
**🌟 `reviewList`** β€” extract paginated customer review cards, rating histogram, and `nextPage` link (optional `structured-extraction` add-on)
**πŸ“§ `contactInfo`** β€” extract email addresses, phone numbers, and social media links found on the page (optional `structured-extraction` add-on)
**🏒 `businessLead`** β€” extract local business / company profile details (optional `structured-extraction` add-on)
**πŸ‘€ `socialProfile`** β€” extract public social media profile metadata and embedded post cards (optional `structured-extraction` add-on)
**πŸ“° `article`** β€” extract news or blog article headline, author, publish date, and body text (optional `structured-extraction` add-on)
**πŸ“š `articleList`** β€” extract article cards and `nextPage` link from a news section or blog index page (optional `structured-extraction` add-on)
**πŸ—‚οΈ `articleNavigation`** β€” extract article links (`items`), section links (`subCategories`), and `nextPage` URL (optional `structured-extraction` add-on)
**πŸ’¬ `forumThread`** β€” extract forum discussion topic, original post, and replies (optional `structured-extraction` add-on)
**πŸ”Ž `serp`** β€” extract organic search engine result rankings, titles, URLs, and `nextPage` link (optional `structured-extraction` add-on)
**πŸ“‘ `pageContent`** β€” extract universal page title, meta description, canonical URL, headings, and links (optional `structured-extraction` add-on)
**🎯 `extractRules`** β€” optional JSON map of custom field names to CSS selectors / `@attr` extractors (optional `structured-extraction` add-on)
**πŸ” `autoPaginate`** β€” automatically follow `nextPage` links across multiple listing/category/review pages up to `maxPages` (default: `false`)
**πŸ“– `maxPages`** β€” maximum number of pagination pages to follow per listing URL when `autoPaginate` is enabled (default: `1`)
**πŸ”— `followDetailLinks`** β€” automatically follow discovered listing links and extract full detail schemas from each detail page (default: `false`)
**πŸ“¦ `outputMode`** β€” dataset output mode: `"page"` (1 row per URL envelope) or `"items"` (1 row per extracted listing card)
**πŸ”¬ `includeTechStack`** β€” detect website technologies across 7,291 CMS, e-commerce, framework, pixel, and CDN signatures (optional `tech-stack-scan` add-on)
**βš™οΈ `extractFrom`** β€” execution mode: `"auto"` (recommended), `"httpResponseBody"`, or `"browserHtml"`
**🌍 `country`** β€” 2-letter ISO country code for proxy geo-targeting (e.g. `"us"`, `"gb"`, `"de"`) or `"auto"`
**πŸ“± `device`** β€” emulate a `"desktop"` or `"mobile"` browser/TLS profile (default: `"desktop"`)
**⏳ `waitSelector`** β€” optional CSS selector to wait for when rendering dynamic SPA pages
**πŸ–±οΈ `clickSelector`** β€” optional CSS selector to click before extraction (e.g. cookie banner or 'Load More' button)
**πŸ“œ `scrollCount`** β€” number of viewports (`0`–`15`) to scroll down before extraction to trigger lazy-loaded grids
**⏱️ `waitMs`** β€” optional extra wait time in milliseconds (`0`–`10000`) after hydration before extracting
**πŸ”‘ `sessionId`** β€” optional sticky session identifier to reuse the same proxy IP across requests
**πŸ“¨ `customHeaders`** β€” optional JSON object of custom HTTP request headers
**πŸͺ `cookies`** β€” optional JSON object of custom request cookies
### Output sample Results are streamed in real time to your default Apify Dataset: ```json [ { "url": "https://www.amazon.com/dp/B0HG17QF13", "statusCode": 200, "autoExtractedCategory": "product", "markdown": "# 2026 Laptop, 16inch Laptop Computer with Core i7-6500U Processor 16GB RAM DDR3 512GB SSD\n\n- **Brand:** DUNHOO\n- **Price:** $412.99 USD", "product": { "url": "https://www.amazon.com/dp/B0HG17QF13", "name": "2026 Laptop, 16inch Laptop Computer with Core i7-6500U Processor 16GB RAM DDR3 512GB SSD, 1920*1200 Display, Win 11 Pro", "sku": "B0HG17QF13", "brand": "DUNHOO", "price": "$412.99", "currency": "USD", "availability": "In Stock", "images": [ "https://m.media-amazon.com/images/I/71wF7x1gJGL._AC_SL1500_.jpg" ], "features": [ "Powerful Core i7-6500U Processor (up to 3.1GHz) with 16GB RAM and 512GB SSD" ], "specifications": { "Standing screen display size": "16 Inches", "Screen Resolution": "1920 x 1200 pixels", "RAM": "16 GB DDR3", "Hard Drive": "512 GB SSD" }, "aggregateRating": { "ratingValue": "4.4 out of 5", "reviewCount": "5 global ratings", "bestRating": "5" } }, "reviews": { "url": "https://www.amazon.com/dp/B0HG17QF13", "aggregateRating": { "ratingValue": "4.4 out of 5", "reviewCount": "5 global ratings", "bestRating": "5" }, "ratingHistogram": { "5_star_percent": 70, "4_star_percent": 0, "3_star_percent": 30, "2_star_percent": 0, "1_star_percent": 0 }, "reviews": [] }, "structuredExtractionApplied": true, "techStackApplied": false, "telemetry": { "strategyCombinationsAvailable": 627480, "extractFromUsed": "httpResponseBody", "matchedBlueprint": "amazon-products-scraper@v5", "learnedDomainProfileHit": false, "extractionSource": "blueprint:amazon-products-scraper", "countryUsed": "us", "durationMs": 6420 } } ] ``` ### How much does Universal Web Unblocker & AI Extractor cost? Universal Web Unblocker & AI Extractor uses **Pay-Per-Event (PPE)** pricing with a **`$0.00` Actor start fee**β€”you only pay for items successfully unblocked (`statusCode == 200`) and delivered to your dataset: **πŸ†“ Free plan (`result-item`)** β€” `$0.005` per item (`$5.00` / 1,000 items, capped at 20 items/run)
**πŸ₯‰ Starter (Bronze) plan (`result-item`)** β€” `$0.0022` per item (`$2.20` / 1,000 items)
**πŸ₯ˆ Scale (Silver) plan (`result-item`)** β€” `$0.0016` per item (`$1.60` / 1,000 items)
**πŸ₯‡ Business+ (Gold) plan (`result-item`)** β€” `$0.0012` per item (`$1.20` / 1,000 items)
**🧠 Optional Add-On Event: `structured-extraction`** *(charged only when you enable `autoExtract`, a structured schema toggle, or `extractRules` and non-empty structured JSON is returned)*:
**πŸ†“ Free:** `$0.0015` (`$1.50`/1K) Β· **πŸ₯‰ Bronze:** `$0.0010` (`$1.00`/1K) Β· **πŸ₯ˆ Silver:** `$0.00075` (`$0.75`/1K) Β· **πŸ₯‡ Gold:** `$0.0005` (`$0.50`/1K)
**πŸ”¬ Optional Add-On Event: `tech-stack-scan`** *(charged only when you enable `includeTechStack: true` and website technologies are detected)*:
**πŸ†“ Free:** `$0.0015` (`$1.50`/1K) Β· **πŸ₯‰ Bronze:** `$0.0010` (`$1.00`/1K) Β· **πŸ₯ˆ Silver:** `$0.00075` (`$0.75`/1K) Β· **πŸ₯‡ Gold:** `$0.0005` (`$0.50`/1K)
### Want to extract website contacts, tech stacks, or Google Maps leads? Combine Universal Web Unblocker & AI Extractor with our companion B2B enrichment and domain intelligence Actors: πŸ“§ **[Website Contact & Email Scraper](https://apify.com/misterkhan/contact-enricher-scraper)** β€” crawl company websites and internal `/contact` pages for verified emails, phone numbers, and LinkedIn/social profiles
πŸ› οΈ **[Website Tech Stack, Contacts & AI Markdown Scanner](https://apify.com/misterkhan/website-tech-stack-scanner)** β€” detect 7,000+ technologies, ad pixels, CMS platforms, and contacts across any domain list
πŸ“„ **[Website to Clean Text & Markdown Crawler](https://apify.com/misterkhan/ai-markdown-crawler)** β€” bulk convert documentation and marketing pages into clean RAG Markdown
### Integrations and Universal Web Unblocker & AI Extractor Connect Universal Web Unblocker & AI Extractor directly to **Make, n8n, Zapier, Clay, LangChain, LlamaIndex, Google Sheets, Slack, or Amazon S3** via Apify integrations and webhooks, or call it programmatically via the official `apify-client` in Python and Node.js. ### Frequently asked questions **πŸ›‘οΈ Do I need to configure or pay extra for Apify Residential Proxies?**
No. Regional residential and ISP proxy rotation and anti-bot challenge solving are built in and included in the flat `result-item` rate.
**🧭 How do I scrape multiple pages of listings from a category or search URL?**
Enable `productList` (or `jobPostingList` / `realEstateList` / `reviewList`), check `autoPaginate: true`, and set `maxPages` (e.g. `5`). If you want each product or job emitted as its own row in your Dataset, set `outputMode` to `"items"`.
**⚑ Am I charged extra if a website requires browser JavaScript rendering?**
No. Automatic escalation from HTTP to full browser rendering when a site requires JavaScript or challenge solving is included in the base `result-item` event.
**🧩 Can I extract multiple schemas (e.g., `product` + `reviews` + `contactInfo`) from the same URL?**
Yes. You can enable multiple category checkboxes simultaneously on the same request, and the `structured-extraction` add-on event is charged at most once per output item.
**βš–οΈ Is web scraping legal?**
Our Actor only extracts publicly accessible web data and does not bypass private account logins. For more details on ethical and compliant web data extraction, read Apify's guide: [Is Web Scraping Legal?](https://blog.apify.com/is-web-scraping-legal/).
### Support and legal disclaimer If you encounter any site-specific extraction issues or need a custom schema field added, please open a ticket in the **Issues** tab on this Actor's Apify Store page. This Actor is an independent web data extraction tool and is not affiliated with, endorsed by, or sponsored by any target website processed by the user. # Actor input Schema ## `startUrls` (type: `array`): List of webpage URLs to unblock and extract. Works on any e-commerce store, category/search listing page, real estate portal, job board, review site, directory, news article, or Single-Page Application (SPA). ## `maxItems` (type: `integer`): Maximum number of dataset items to output in this run. Free plan runs are capped at 20 items per run; paid Apify plans support unlimited extraction. ## `includeMarkdown` (type: `boolean`): Convert the unblocked webpage into clean, token-efficient Markdown with navigation menus, cookie banners, and scripts stripped out. Included in the base result-item rate. ## `includeHtml` (type: `boolean`): Include the full unblocked HTML document string (`html`) in the output dataset. Disabled by default to keep dataset files compact and fast to download. ## `includeHydration` (type: `boolean`): Extract embedded Next.js (`__NEXT_DATA__`), Nuxt (`__NUXT_DATA__`), Apollo State, and Schema.org `application/ld+json` objects. ## `httpResponseHeaders` (type: `boolean`): Include the target server's HTTP response headers in the output. ## `networkCapture` (type: `boolean`): Capture background XHR, Fetch, and GraphQL JSON network requests fired by the page during browser rendering. ## `screenshot` (type: `boolean`): Capture a base64-encoded JPEG screenshot of the rendered webpage. ## `autoExtract` (type: `boolean`): Automatically classify the page and extract structured JSON (Product, Product List, Job Posting, Real Estate, Reviews, Article, or Page Content). Triggers the `structured-extraction` add-on event only when non-empty structured data is extracted. ## `product` (type: `boolean`): Extract e-commerce product schema: name, price, currency, SKU/ASIN, brand, availability, high-res images, feature bullets, technical specifications table, breadcrumbs, and aggregate rating. ## `productList` (type: `boolean`): Extract all product cards from a category, search, or collection page, plus `nextPage` link. ## `productNavigation` (type: `boolean`): Extract category name, detail item links (`items`), sub-category/filter links (`subCategories`), breadcrumbs, and `nextPage` URL. ## `jobPosting` (type: `boolean`): Extract job title, hiring organization, location, salary range, employment type, date posted, and job description. ## `jobPostingList` (type: `boolean`): Extract multiple job postings from a job board search or company careers page. ## `jobPostingNavigation` (type: `boolean`): Extract job detail links (`items`), filter links (`subCategories`), and `nextPage` pagination URL from a job board search page. ## `realEstate` (type: `boolean`): Extract property price, address, bedrooms, bathrooms, square footage/living area, property type, latitude/longitude, and photos. ## `realEstateList` (type: `boolean`): Extract multiple property listings from a real estate search or directory page. ## `reviews` (type: `boolean`): Extract aggregate star rating, 5-star to 1-star rating histogram breakdown, and individual customer review cards. ## `reviewList` (type: `boolean`): Extract customer review cards, rating histogram, and `nextPage` review pagination link. ## `contactInfo` (type: `boolean`): Extract email addresses, phone numbers, and social media links (LinkedIn, X/Twitter, Instagram, YouTube, TikTok, Facebook, GitHub) found on the page. ## `businessLead` (type: `boolean`): Extract company name, address, phone number, website, opening hours, and aggregate rating. ## `socialProfile` (type: `boolean`): Extract public social media profile metadata and embedded post cards. ## `article` (type: `boolean`): Extract headline, author, publish/modified dates, publisher, main image, and clean article body text. ## `articleList` (type: `boolean`): Extract article cards and `nextPage` link from a news section or blog index page. ## `articleNavigation` (type: `boolean`): Extract article links (`items`), section links (`subCategories`), and `nextPage` URL. ## `forumThread` (type: `boolean`): Extract discussion topic, original post, author, and replies/comments. ## `serp` (type: `boolean`): Extract organic search result rankings, titles, URLs, and `nextPage` link. ## `pageContent` (type: `boolean`): Extract page title, meta description, canonical URL, language, headings (H1-H3), main text, and outbound links. ## `extractRules` (type: `object`): Optional JSON object mapping custom field names to CSS selectors (e.g. `{"price": ".price_color", "bookUrls": ["article.product_pod h3 a @href"]}`). Append `@attr` to extract an attribute or wrap selector in `[...]` to extract all matching elements. ## `autoPaginate` (type: `boolean`): Automatically follow `nextPage` links across multiple listing/category/review pages up to `maxPages`. ## `maxPages` (type: `integer`): Maximum number of pagination pages to follow per listing URL when `autoPaginate` is enabled (1 to 50). ## `followDetailLinks` (type: `boolean`): When scraping a listing/category page, automatically follow the discovered item links and extract full detail schemas (`product`, `jobPosting`, `realEstate`, `article`, `contactInfo`) from each detail page. ## `outputMode` (type: `string`): Choose `page` (default: 1 dataset row per URL containing all requested schemas) or `items` (flatten `productList`, `jobPostingList`, `realEstateList`, `reviewList`, or `detailPages` so each extracted listing card is emitted as its own dataset row). ## `includeTechStack` (type: `boolean`): Identify CMS, e-commerce platforms, JavaScript frameworks, analytics pixels, CDNs, and WAFs on the page. Triggers the `tech-stack-scan` add-on event only when technologies are detected. ## `extractFrom` (type: `string`): Leave on `auto` (recommended) so the engine starts with fast Coherent TLS HTTP and automatically escalates to a Stealth Browser with fresh residential proxy sessions only when a WAF challenge or unhydrated SPA shell is detected. ## `country` (type: `string`): 2-letter ISO country code for proxy geo-targeting (e.g. `us`, `gb`, `de`, `fr`, `ca`, `au`, `jp`), or `auto` to infer from the target URL domain. ## `device` (type: `string`): Emulate a Desktop (`desktop`) or Mobile (`mobile`) browser/TLS profile. ## `waitSelector` (type: `string`): Optional CSS selector to wait for when rendering dynamic Single-Page Applications (e.g. `#productTitle`, `[data-testid='listing-card']`). ## `clickSelector` (type: `string`): Optional CSS selector to click after page load (e.g. cookie consent button, 'Show More', or 'Reviews' tab). ## `scrollCount` (type: `integer`): Number of viewports (0 to 15) to scroll down before extracting to trigger lazy-loaded listing grids or infinite scroll. ## `waitMs` (type: `integer`): Optional extra wait time in milliseconds (0 to 10000) after page hydration / interaction before extracting. ## `sessionId` (type: `string`): Optional session identifier to reuse the same residential/ISP proxy IP across multiple requests. ## `customHeaders` (type: `object`): Optional JSON object of custom HTTP request headers to send with the request. ## `cookies` (type: `object`): Optional JSON object of custom cookies (`{"name": "value"}`) to attach to the request. ## Actor input object example ```json { "startUrls": [ { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" } ], "maxItems": 25, "includeMarkdown": true, "includeHtml": false, "includeHydration": false, "httpResponseHeaders": false, "networkCapture": false, "screenshot": false, "autoExtract": false, "product": false, "productList": false, "productNavigation": false, "jobPosting": false, "jobPostingList": false, "jobPostingNavigation": false, "realEstate": false, "realEstateList": false, "reviews": false, "reviewList": false, "contactInfo": false, "businessLead": false, "socialProfile": false, "article": false, "articleList": false, "articleNavigation": false, "forumThread": false, "serp": false, "pageContent": false, "autoPaginate": false, "maxPages": 1, "followDetailLinks": false, "outputMode": "page", "includeTechStack": false, "extractFrom": "auto", "country": "auto", "device": "desktop", "scrollCount": 0, "waitMs": 0 } ``` # Actor output Schema ## `overview` (type: `string`): No description ## `dataset_items` (type: `string`): No description ## `usage_summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" } ], "maxItems": 25 }; // Run the Actor and wait for it to finish const run = await client.actor("misterkhan/universal-web-unblocker-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" }], "maxItems": 25, } # Run the Actor and wait for it to finish run = client.actor("misterkhan/universal-web-unblocker-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" } ], "maxItems": 25 }' | apify call misterkhan/universal-web-unblocker-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,misterkhan/universal-web-unblocker-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/PAwh3LpjM7Ypu4bXO/builds/iTcxOIKlOtFT83Nm6/openapi.json