HTML Extractor

🧩 HTML Extractor – Raw HTML Fetcher with Browser Impersonation & Custom Headers

The HTML Extractor fetches a single URL and returns its raw, unmodified HTML alongside the HTTP status code, content type and byte length. It is deliberately a low-level tool: no parsing, no CSS selectors, no cleaning. You give it a URL, and you get back exactly what the server sent, ready to feed into your own parser, diffing routine, LLM pipeline or archival store.

That simplicity is the point. Most scraping stacks eventually need a reliable primitive that answers one question — "what does this page actually return right now?" — without a headless browser in the loop. This HTML extractor is built on curl_cffi, which impersonates a real browser's TLS fingerprint and header ordering rather than announcing itself as a scripting library. That single detail is what lets it retrieve pages that a plain requests call would be refused on, and it does so at HTTP speed with no rendering step.

You can switch the impersonation between desktop and mobile, supply arbitrary extra HTTP headers, and set a per-attempt timeout. Proxy rotation is handled automatically inside the actor, so there are no proxy credentials for you to manage.

---

📊 What Data Can You Extract with This HTML Extractor?

Each run produces a single dataset item describing one fetch. The fields group into four small, well-defined categories.

Category	Fields	What it gives you
Request identity	url	The canonical URL that was fetched, so results remain traceable when items are appended across many runs
Response status	statusCode	The HTTP status code returned by the server — the first thing to check when output is not what you expected
Response metadata	contentType, length	The Content-Type header value and the size of the returned body, useful for validating that you got HTML and not a redirect stub or an error page
Payload	html	The complete raw HTML of the response, exactly as received, with no cleaning, minification or re-encoding
Failure detail	error	An error message populated when the fetch could not complete, so failures are recorded in the dataset rather than only in the log

The field that carries the most diagnostic weight is length. When a fetch is soft-blocked or redirected to an interstitial, statusCode is very often still 200 — the giveaway is a body far shorter than the page you expected. Comparing length against a known baseline is the quickest way to detect a block that did not announce itself as one, and it is far cheaper than parsing the HTML to find out.

---

🌟 Key Features of the HTML Extractor

Feature	Description
📄 Raw, unmodified HTML	The html field is the response body verbatim — no cleaning, stripping or re-encoding, so your own parser sees exactly what the server sent
🖥️ Desktop and mobile impersonation	useMobile switches the request between a desktop browser profile and a mobile Chrome/Safari profile, covering both TLS fingerprint and headers
🧾 Custom header injection	headersText accepts arbitrary Header-Name: value lines that override the default impersonated headers when names collide
⏱️ Per-attempt timeout control	timeoutSec sets the request timeout in seconds for each attempt, so slow origins fail predictably rather than stalling a run
🔐 Browser-grade TLS fingerprinting	Built on curl_cffi, which replicates a real browser's TLS handshake instead of a scripting library's, improving reach on protected origins
📊 Full response metadata	statusCode, contentType and length are returned alongside the body, so validation does not require parsing
🚫 Errors captured in the dataset	When a fetch fails, the error field records why, keeping failures visible to downstream consumers rather than buried in logs
🔄 Automatic proxy rotation	Proxy handling is managed inside the actor; there is nothing to configure or maintain
⚡ No headless browser	Pure HTTP fetching means low memory use and fast completion compared with browser-driven alternatives

---

🚀 Why Choose This HTML Extractor?

A composable primitive, not an opinionated scraper. Most scraping actors decide for you what the useful parts of a page are. This HTML extractor makes no such decision. It hands you the complete document so your own selectors, XPath expressions, regex or language model can operate on the real source, including inline JSON-LD, <script> payloads, meta tags and hydration state that HTML-to-text converters routinely destroy.

Browser impersonation without a browser. curl_cffi reproduces the TLS handshake, cipher ordering and header shape of a genuine Chrome or Safari client. That gets you through a meaningful share of fingerprint-based filtering while keeping the cost and latency profile of a plain HTTP request, which matters when a headless browser would be an order of magnitude heavier for no rendering benefit.

Header control where it counts. The headersText field lets you set Accept-Language, Referer, Cookie, custom API headers or anything else, one per line. Because these override the impersonated defaults on name collision, you can adjust exactly the header you need without discarding the rest of a coherent browser profile.

Honest failure reporting. A fetch that fails writes an error into the dataset item rather than silently producing nothing. Combined with statusCode and length, that gives monitoring pipelines a clean, machine-readable signal for whether a target is healthy, blocked or changed.

---

📥 Input

{
  "url": "https://example.com",
  "useMobile": false,
  "headersText": "",
  "timeoutSec": 30
}

🔧 HTML Extractor Input Fields

Field	Type	Required	Default	Description
url	string	✅ Yes	prefilled with https://example.com	The full URL of the page to fetch and extract raw HTML from.
useMobile	boolean	No	false	If enabled, the request impersonates a mobile Chrome/Safari client (TLS fingerprint + headers) instead of a desktop browser.
headersText	string	No	""	Optional extra HTTP headers to send with the request, one per line in Header-Name: value format. These override the default impersonated-browser headers when names collide.
timeoutSec	integer	No	30	Request timeout in seconds, per attempt.

💡 Input Examples

Basic desktop fetch

{
  "url": "https://news.ycombinator.com",
  "timeoutSec": 30
}

Mobile-rendered variant of a page

{
  "url": "https://www.wikipedia.org",
  "useMobile": true,
  "timeoutSec": 30
}

Custom headers for language and referer

{
  "url": "https://example.com/de/products",
  "headersText": "Accept-Language: de-DE,de;q=0.9\nReferer: https://www.google.de/\nX-Requested-With: XMLHttpRequest",
  "timeoutSec": 60
}

---

📤 Output

The dataset contains one item describing the fetch.

{
  "url": "https://example.com",
  "statusCode": 200,
  "contentType": "text/html",
  "length": 559,
  "html": "<!doctype html><html lang=\"en\"><head><title>Example Domain</title><link rel=\"icon\" href=\"data:…"
}

🧾 HTML Extractor Output Fields

Field	Type	Description
url	string | null	Canonical URL of the scraped item.
statusCode	integer | null	HTTP status code returned.
contentType	string | null	Content type of the response.
length	integer | null	Length of the item.
html	string | null	Raw HTML of the item.
error	string | null	Error message, if the item failed to process.

On a successful fetch, error is absent or null and html carries the payload. On a failure, error describes what went wrong and html may be null. Always branch on error and statusCode before attempting to parse html.

---

💻 How to Use the HTML Extractor (Step by Step)

Step 1: Open the HTML Extractor on Apify

Sign in to Apify and open the actor page. If you have not used the platform before, create a free account — you will also need an API token if you plan to call the HTML extractor programmatically, which is how most people end up using a tool of this shape. Click Try for free or Start to open the input form; all four fields are prefilled with working defaults, so you can run it immediately against https://example.com to see the output shape.

Step 2: Set the target URL

url is the only required input. Provide the complete absolute URL including the scheme — https://example.com/page, not example.com/page. The actor fetches exactly this URL and follows the server's own redirect behaviour, so the html you receive corresponds to whatever the server ultimately returned. If you are targeting a page behind a query string, include the full query as-is rather than trying to simplify it, since many sites vary their response on parameters you might assume are cosmetic.

Step 3: Choose desktop or mobile impersonation

Leave useMobile at false for the desktop profile, which is the right default for most sites. Switch it to true when you specifically want the mobile variant of a page. This is not just a User-Agent swap — the mobile setting changes the TLS fingerprint and the full header set to match a mobile Chrome or Safari client. Some sites serve materially different markup to mobile clients, with different data embedded in inline scripts, so it is worth fetching both variants when you are reverse-engineering a page's structure.

Step 4: Add custom headers if the target needs them

headersText takes one header per line in Header-Name: value format. Common uses are forcing a language with Accept-Language, setting a Referer so a page treats the request as inbound from search, adding a Cookie for consent or region state, or supplying an API key header when the target is a JSON endpoint rather than an HTML page. Anything you specify here overrides the impersonated default with the same name, and everything you do not specify is left intact, so you keep a coherent browser profile while adjusting only what you need.

Step 5: Tune the timeout

timeoutSec applies per attempt and defaults to 30 seconds. Thirty seconds is generous for a well-behaved origin. Raise it when you are fetching a slow, heavy page or one behind a sluggish origin server; lower it when you are running the extractor in a latency-sensitive loop and would rather fail fast and retry than block. A timeout produces a dataset item with error populated, which keeps the failure visible.

Step 6: Run and inspect the response metadata

Click Start. When the run finishes, open Storage → Dataset and look at statusCode, contentType and length before anything else. A 200 with a text/html content type and a plausible length means you have a real page. A 200 with a suspiciously small length usually means an interstitial or a soft block. A non-HTML contentType means the URL returned something other than a web page — JSON, a PDF, or an image — which may be exactly what you wanted or a sign that the URL was wrong.

Step 7: Parse the HTML downstream

The html field is the complete response body. Feed it into BeautifulSoup, lxml, Cheerio, Parsel, a regex, or a language model, depending on what you are extracting. Because nothing has been stripped, embedded JSON-LD in <script type="application/ld+json"> blocks, Open Graph meta tags and framework hydration state are all still present — and those are frequently a cleaner extraction target than the visible DOM, since they are structured data the site publishes deliberately.

---

🔌 API Access & Integrations

Call the HTML extractor directly from your own code.

curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~html-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "useMobile": false,
    "timeoutSec": 30
  }'

Python, using the official client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run_input = {
    "url": "https://example.com",
    "useMobile": False,
    "headersText": "Accept-Language: en-GB,en;q=0.9",
    "timeoutSec": 45,
}

run = client.actor("scrapers-hub/html-extractor").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("error"):
        print("failed:", item["error"])
    else:
        print(item["statusCode"], item["contentType"], item["length"])
        html = item["html"]

The HTML extractor also connects to Zapier, Make, Google Sheets and Slack through Apify's integrations, and webhooks can fire on run completion so a downstream parser picks up each fetch automatically.

---

💡 Best Use Cases for Raw HTML Extraction

🔬 Building custom parsers and prototyping selectors

When you are writing a scraper for a new site, the first thing you need is the real HTML to write selectors against. Fetching it with the HTML extractor and inspecting html shows you precisely what the server delivers to a browser-shaped client, which is frequently different from what your browser's developer tools display after JavaScript has run. contentType and length confirm you are looking at the real document.

📸 Page archiving and change detection

Store the html field with a timestamp on a schedule and you have a versioned archive of a page. Diffing successive snapshots surfaces pricing changes, policy edits, staff-page updates and quiet content revisions. length alone gives you a cheap first-pass change signal without diffing the full body, and statusCode records when a page went away entirely.

🤖 Feeding language models with real source markup

LLM extraction pipelines work best when they see structured signal, not a lossy text rendering. Passing raw html preserves JSON-LD blocks, microdata attributes, meta tags and table structure that text converters flatten. For extraction tasks that depend on relationships between elements, the markup is the information.

🩺 Uptime, redirect and SEO monitoring

statusCode, contentType and length are the core of a lightweight site monitor. Watch for unexpected status codes, content types that drift from text/html, or a body length that collapses. Combined with parsing html for canonical tags, robots meta directives and hreflang links, this makes a serviceable technical SEO check that needs no browser.

🌐 Comparing mobile and desktop page variants

Running the same URL twice, once with useMobile set to true and once false, exposes how a site adapts its markup. Differences in embedded data, lazy-loading strategy or served content are immediately visible in the two html payloads, which is valuable both for mobile SEO auditing and for finding which variant is easier to parse.

🧪 Debugging blocks and anti-bot behaviour

When another scraper starts returning empty results, the HTML extractor tells you why. Fetch the same URL and read the response: a challenge interstitial, a consent wall, a geographic redirect and a genuine 403 all look completely different in raw HTML, and each calls for a different fix. error captures transport-level failures that never produced a body at all.

🔗 Fetching non-HTML endpoints

Despite the name, the actor returns whatever the URL serves. Point it at a JSON API, an XML sitemap or an RSS feed and the payload arrives in html with contentType identifying what it actually is. That makes it a general-purpose authenticated-header fetcher for endpoints where you need browser-grade TLS to get a response at all.

---

⚙️ Tips for Better HTML Extraction Results

• Check statusCode and length before parsing. A 200 with a body far shorter than expected is the classic signature of a soft block or an interstitial. Validating on length is much cheaper than discovering the problem inside your parser.
• Try useMobile: true when a desktop fetch looks thin. Mobile variants are sometimes served with less aggressive protection, and they occasionally embed cleaner structured data than the desktop page does.
• Set Accept-Language explicitly for international targets. Without it, you get whatever the exit IP's geography implies, which makes results inconsistent between runs. One line in headersText removes that variability entirely.
• Add a plausible Referer for pages that expect inbound traffic. Some sites vary their response for requests that appear to arrive with no referrer at all, and setting one is a single line in headersText.
• Raise timeoutSec for slow origins rather than retrying blindly. Repeated fast-failing attempts against a genuinely slow server waste time and add load; a longer single attempt is usually the better trade.
• Keep the raw HTML rather than only your parsed output. Storing html means that when your selectors break or your requirements change, you can re-parse historical snapshots instead of re-fetching pages that may no longer exist.

---

🛠️ Troubleshooting

The html field is much shorter than the page I see in my browser. The page almost certainly renders its content with JavaScript after the initial document loads. This HTML extractor performs a single HTTP fetch and does not execute JavaScript, so it returns the server-rendered document only. Check the returned markup for an embedded JSON payload — many JavaScript-heavy sites ship their data in a hydration script that is present in the raw HTML and is often easier to parse than the rendered DOM would be.

I got a 403 or a challenge page instead of content. Some origins apply protection that TLS impersonation alone does not clear. Try switching useMobile, and add headers via headersText — a realistic Accept-Language, a Referer, or a consent Cookie will each resolve a meaningful share of cases. Proxy rotation is already automatic, so re-running can also succeed against IP-based rate limits.

The run finished but the item has an error and no HTML. That means the request never completed — typically a timeout, a DNS failure, a TLS negotiation problem or a connection reset. Read the error string for the specific cause. If it is a timeout, raise timeoutSec. If it is a DNS or certificate error, verify the URL is correct and reachable, including the scheme.

My custom headers do not seem to be applied. Check the formatting in headersText: exactly one header per line, in Header-Name: value form, with a colon separating name from value. A missing colon or a header split across lines will not parse. Remember that headers you set override the impersonated defaults of the same name, which can be a problem if you accidentally override User-Agent with an unrealistic value and break the browser profile.

Can the HTML extractor fetch multiple URLs in one run? No. The input schema takes a single url per run by design, which keeps the actor a clean primitive. For batches, call it once per URL from your own code or orchestration layer using the API examples above — that also gives you per-URL error handling, which a batched run would obscure.

---

❓ Frequently Asked Questions About HTML Extraction

What does the HTML Extractor actor do? It fetches a single URL and returns the raw HTML of the response along with the HTTP status code, the content type and the body length. It performs no parsing or cleaning of any kind.

Does the HTML extractor execute JavaScript? No. It makes a single HTTP request and returns the server's response. Content injected by client-side JavaScript after page load will not appear, although the data behind it is frequently present in inline hydration scripts within the raw HTML.

Do I need to configure proxies for the HTML extractor? No. Proxy rotation is handled automatically inside the actor. There is no proxy field in the input schema and nothing for you to set up.

Can I fetch more than one URL per run? Not in a single run — the schema accepts one url. Call the actor once per URL through the API if you need a batch, which also keeps error handling isolated per URL.

What does useMobile actually change? It switches the impersonated client from a desktop browser to a mobile Chrome or Safari profile, changing both the TLS fingerprint and the HTTP headers. It is not simply a User-Agent string swap.

How do I send cookies with the request? Add a Cookie: name=value; other=value line to headersText. It is sent with the request and overrides any default of the same name.

Can I use the HTML extractor to fetch JSON or XML? Yes. The actor returns whatever the URL serves, and contentType tells you what it was. JSON APIs, XML sitemaps and RSS feeds all work; the payload simply arrives in the html field.

Why is statusCode 200 but the content looks wrong? Many protection layers and consent walls return a 200 with an interstitial body rather than an error status. Compare length against what you would expect for the real page, and read the beginning of html to identify what was actually served.

What is a sensible value for timeoutSec? The default of 30 seconds suits most targets. Raise it towards 60 or more for known-slow origins or very large documents; lower it when you would rather fail fast in a tight automation loop.

Is there a size limit on the HTML returned? The actor returns the full response body, and length reports its size. Very large documents are constrained by Apify's dataset item size limits rather than by anything the HTML extractor imposes.

Can I use this HTML extractor behind a login? Only if you can express the session as headers. Supplying a valid session cookie through headersText works for many sites. The actor performs no login flow of its own and cannot complete interactive authentication.

How do I detect that a page has changed since the last fetch? Hash the html field and store the hash with a timestamp, or store the length value for a cheaper approximate signal. Comparing either across scheduled runs gives you a change feed without diffing full documents every time.

Does the HTML extractor follow redirects? It follows the server's redirect behaviour, so html reflects the final response received. Check statusCode and compare the returned content against the URL you requested if redirect behaviour matters to your use case.

Can I schedule the HTML extractor to run automatically? Yes. Apify's scheduler runs the actor on any cron expression, which is the standard way to build a page-archiving or uptime-monitoring workflow. Attach a webhook to hand each result to a downstream parser.

Is scraping HTML from public pages legal? Fetching publicly accessible pages is broadly permitted, but you remain responsible for compliance with the target site's terms of service, applicable data protection law, and any relevant local regulation. Respect robots.txt and rate limits, and do not use the actor against content that is behind authentication you are not entitled to.

---

🆘 Support & Feedback

If the HTML extractor misbehaves — an unexpected error, a target it cannot reach, a header that is not being applied — open a ticket on the Issues tab of the actor page and include the run ID and the URL you were fetching.

Need something more than a raw fetch, such as batch URL input, built-in parsing rules, JavaScript rendering, or a custom output shape wired into your own pipeline? Email scraperhubapi@gmail.com and describe what you are building.

If the HTML Extractor is useful to you, please leave a review on its Apify page. Ratings and written feedback shape which improvements are prioritised.

---

⚖️ Disclaimer

The HTML Extractor fetches publicly accessible URLs and returns the response the server provides. It does not bypass authentication, defeat paywalls, or access private accounts, and any HTML extraction it performs is limited to content a normal visitor could retrieve.

You are responsible for how you use this actor and the data it returns. That includes compliance with the target website's terms of service and robots.txt, with applicable data protection law such as the GDPR and the UK GDPR wherever the fetched HTML contains personal data, and with any copyright or database rights attaching to the retrieved content. Because raw HTML extraction returns everything on a page indiscriminately, take particular care when the target may include names, contact details or other personal information.

This actor is an independent tool and is not affiliated with or endorsed by any website you choose to fetch. All trademarks referenced belong to their respective owners.

If you believe data collected through this actor relates to you and you would like it removed, contact scraperhubapi@gmail.com with the details and the request will be handled promptly.