Website to Markdown Extractor: Clean Text for LLMs and RAG

Turn any website page into clean Markdown and plain text, ready for LLM prompts, RAG pipelines and knowledge bases. Give the actor a list of page addresses; for each one it reads the page once, politely, finds the main content (the article, the documentation page, the product text), cuts the menus, footers, cookie banners, ads, sidebars and comments, and returns one row with the Markdown, the plain text, the title, the description, the language, the publication date, the headings, the links and the images, as JSON, CSV or Excel.

There is no browser and no crawling: the actor reads exactly the pages you list, from their HTML, in 256 MB of memory. The content finder is Rowcraft's own code in the spirit of Readability, written for this actor and tested on articles, blogs, documentation, encyclopedia pages, forums and old table-based pages in several languages.

What you get

One row per page:

• Identity: the address as you gave it, the address that answered after redirects, the HTTP status, the page title, the meta description, the declared language and the canonical address
• Markdown: the main content with headings, paragraphs, bold and italic, nested lists, links with absolute addresses, images with alt text, simple tables, code blocks with their language and block quotes
• Plain text: the same content without any Markdown marks, paragraphs separated by blank lines, ready to be split into chunks or counted
• Publication date: read from the page metadata (meta tags, JSON-LD, a time element), as an ISO 8601 date, or null when the page does not say
• Size: wordCount and charCount of the whole extracted text, even when the Markdown was cut at your character limit
• Structure: headings (one per line, as H2: text), links and images (absolute addresses, one per line) with their counts
• Change tracking: a contentHash of the text, which changes when the text of the page changes, and the time of the read

The run also writes a COVERAGE record with one line per page you asked for (ok, no_content, blocked, robots_disallowed, not_found, error or not_checked, with the reason) and a RUN_SUMMARY with counts, the memory peak and why the run stopped.

Why this actor

• Main content, not the whole page. Navigation, headers and footers outside the article, sidebars, share bars, newsletter boxes, cookie banners, ads, comments, forms and hidden elements are cut before the Markdown is written, so your embeddings and prompts are not filled with menus.
• Clean Markdown you can chunk. Headings keep their level, lists keep their nesting, code keeps its language, tables stay tables, and every link and image has an absolute address, so a chunk still makes sense on its own.
• Markdown and plain text in the same row. Use the Markdown for LLM prompts and the text for search indexes, word counts and classifiers.
• Pay only for pages that have content. A page is charged when its main content was found and has at least 20 words. Pages behind an anti-bot page, disallowed by robots.txt, not found, not HTML, empty or too short, unchanged pages with Only new rows, invalid entries and duplicates in your list are never charged.
• No author and no contact data. The author and byline of a page are never read or delivered, mailto: and tel: links are dropped, and e-mail addresses and phone numbers in the text are replaced by placeholders.
• Polite by design. robots.txt is read and obeyed for every host first, then one request for the page, with a pause between requests to the same host (one second by default).
• Cheap to run. No browser means 256 MB of memory and a few seconds per page; the actor reads hundreds of pages an hour from one site and many more across different sites.

How to use it

1. Open the Actor and paste your page addresses into Page URLs, one per line. A column copied from a spreadsheet works, and so do addresses separated by commas. The example addresses in the form run as they are.
2. For a first try keep the list short, or set Maximum pages delivered to a small number.
3. Click Start. The status line shows progress; the run stops cleanly at your maximum or at the charge limit you set in the run options.
4. Open the Dataset tab. The Overview view shows title, size, language, date and headings, Content shows the Markdown of every page and Links and images lists what each page links to. Download JSON, CSV, Excel, XML or HTML, or send the rows to Google Sheets, Slack, Make, n8n or Zapier with the Integrations tab.
5. Open the COVERAGE record in the key-value store to see which pages were not delivered and why.
6. To keep a knowledge base up to date, run the list on a schedule with Only new rows since the last run turned on: a page is delivered, and charged, again only when its text changed.

Input

Field	What it does
urls	Page addresses, one per line (https://docs.example.com/guide/start); an address without https:// is read as https://. The #fragment is ignored. Up to 100,000 per run. Only these pages are read, no links are followed.
fullPage	Convert the whole body of the page instead of only the main content. Hidden elements and scripts are still left out. Default off.
includeLinks	Write links as Markdown links with absolute addresses and fill the links column. Default on; off leaves only the link text.
includeImages	Write images as Markdown images with alt text and fill the images column. Default on. Tracking pixels and inline data images are skipped.
keepTables	Convert simple tables to Markdown tables. Default on; off leaves tables out. Layout tables are always read as ordinary text.
maxChars	Cut the Markdown and the text of a page at this length, at a line or word boundary, and set truncated to true. Default 100,000, from 500 to 2,000,000.
removeSelectors	Extra elements to cut first, one per line: #id, .class, tag, tag.class, tag#id or [attribute]. Simple selectors only.
maxItems	Stop after this many delivered rows. 0 = the whole list.
maxConcurrency	Pages read at the same time, default 3, up to 10 (and one per 64 MB of the run's memory). Requests to the same host stay spaced.
onlyNew, stateName	Deliver a page only when its text changed since the last run with the same state name
notifyUrl, notifyWhenEmpty	A webhook that receives one message when the run ends; see Automation below
requestIntervalMs	Pause between requests to the same host, default 1000 ms, minimum 250 ms

Example: three pages, one row each.

{
  "urls": [
    "https://developer.mozilla.org/en-US/docs/Web/HTML/Element/article",
    "https://docs.apify.com/platform/actors/running",
    "https://en.wikipedia.org/wiki/Markdown"
  ]
}

Example: plain text only for a search index, without links, images and tables, cut at 20,000 characters, with a cookie box removed by its class.

{
  "urls": ["https://example.com/blog/post-one", "https://example.com/blog/post-two"],
  "includeLinks": false,
  "includeImages": false,
  "keepTables": false,
  "maxChars": 20000,
  "removeSelectors": [".cookie-box", "#promo-banner"]
}

Example: a weekly refresh of a documentation set that delivers only the pages whose text changed, with a Slack message.

{
  "urls": ["https://docs.example.com/guide/start", "https://docs.example.com/guide/install"],
  "onlyNew": true,
  "stateName": "docs-refresh",
  "notifyUrl": "https://hooks.slack.com/services/..."
}

Output

21 fields per row. Empty values are null, never missing, so CSV and Excel columns are stable.

Field	Type	Example
id	string	https://docs.apify.com/platform/actors/running
url	string	https://docs.apify.com/platform/actors/running
finalUrl	string	https://docs.apify.com/actors/running
statusCode	integer	200
title	string	Run Actors | Platform | Apify Documentation
description	string	Start an Actor from Apify Console or via API.
language	string	en
publishedAt	string	2026-03-04T08:00:00.000Z
canonical	string	https://docs.apify.com/actors/running
wordCount	integer	532
charCount	integer	3270
headings	string	H1: Run Actors H2: Run Actors manually in Apify Console
markdown	string	# Run Actors You can run Actors manually in Apify Console...
text	string	Run Actors You can run Actors manually in Apify Console...
links	string	https://apify.com/store
linkCount	integer	15
images	string	https://docs.apify.com/assets/images/example.png
imageCount	integer	2
truncated	boolean	False
contentHash	string	4d5b29c13a1d1524
fetchedAt	string	2026-10-17T08:19:32.575Z

Example rows

Two real rows from a run (long texts shortened here, complete in the dataset):

[
  {
    "id": "https://docs.apify.com/platform/actors/running",
    "url": "https://docs.apify.com/platform/actors/running",
    "finalUrl": "https://docs.apify.com/actors/running",
    "statusCode": 200,
    "title": "Run Actors | Platform | Apify Documentation",
    "description": "Start an Actor from Apify Console or via API. Learn about Actor lifecycles, how to specify settings and version, provide input, and resur…",
    "language": "en",
    "publishedAt": null,
    "canonical": "https://docs.apify.com/actors/running",
    "wordCount": 532,
    "charCount": 3270,
    "headings": "H1: Run Actors\nH2: Run Actors manually in Apify Console\nH3: 1. Choose an Actor\nH3: 2. Configure and run the Actor\nH3: 3. Wait for the res…",
    "markdown": "# Run Actors\n\nYou can run Actors manually in Apify Console, with the Apify API, programmatically from your own applications, or from AI a…",
    "text": "Run Actors\n\nYou can run Actors manually in Apify Console, with the Apify API, programmatically from your own applications, or from AI age…",
    "links": "https://apify.com/store\nhttps://docs.apify.com/actors/development\nhttps://console.apify.com/sign-up\nhttps://console.apify.com/\nhttps://co…",
    "linkCount": 15,
    "images": "https://docs.apify.com/assets/images/configure-and-run-actor-ad32be8b3ff4710df7ab3917cd0ca93d.svg\nhttps://docs.apify.com/assets/images/re…",
    "imageCount": 2,
    "truncated": true,
    "contentHash": "4d5b29c13a1d1524",
    "fetchedAt": "2026-10-17T08:29:42.820Z"
  },
  {
    "id": "https://martinfowler.com/articles/microservices.html",
    "url": "https://martinfowler.com/articles/microservices.html",
    "finalUrl": "https://martinfowler.com/articles/microservices.html",
    "statusCode": 200,
    "title": "Microservices",
    "description": "Defining the microservices architectural style by describing their nine common characteristics",
    "language": null,
    "publishedAt": null,
    "canonical": null,
    "wordCount": 6346,
    "charCount": 40768,
    "headings": "H1: Microservices\nH2: Characteristics of a Microservice Architecture\nH3: Componentization via Services\nH3: Organized around Business Capa…",
    "markdown": "# Microservices\n\na definition of this new architectural term\n\n*The term “Microservice Architecture” has sprung up over the last few years…",
    "text": "Microservices\n\na definition of this new architectural term\n\nThe term “Microservice Architecture” has sprung up over the last few years to…",
    "links": "http://2012.33degree.org/talk/show/67\nhttp://www.slideshare.net/fredgeorge/micro-service-architecure\nhttps://www.amazon.com/gp/product/B0…",
    "linkCount": 43,
    "images": "https://martinfowler.com/articles/microservices/images/sketch.png\nhttps://martinfowler.com/articles/microservices/images/conways-law.png…",
    "imageCount": 7,
    "truncated": true,
    "contentHash": "9e78166e20046521",
    "fetchedAt": "2026-10-17T08:29:42.831Z"
  }
]

Results can be downloaded as JSON, CSV, Excel, XML or HTML from the Dataset tab, or read through the Apify API.

Pricing

Pay per result. You pay for each page whose main content was extracted and delivered to your dataset, nothing else: no subscription, no minimum, no charge for pages that could not be read (anti-bot page, robots.txt, not found, errors, not HTML), for pages without main content or with fewer than 20 words, for invalid or repeated entries, or for pages skipped by Only new rows because the text did not change.

Your Apify plan	Price per 1,000 pages	Price per page
Free plan	$2.00	$0.002
Starter plan	$1.80	$0.0018
Scale plan	$1.60	$0.0016
Business plan and above	$1.50	$0.0015

Example: 1,000 pages cost $2.00 on the Free plan and $1.50 on the Business plan. The Apify platform usage of a run (compute and storage) is billed by Apify separately, as for every Actor; the actor runs in 256 MB and spends most of its time waiting between requests. A start event of $0.00005 is charged once per run (Apify default). Cap your spend: set Maximum total charge in the run options. The actor stops cleanly when the cap is reached and you keep everything delivered until then. The run summary (RUN_SUMMARY in the key-value store) says why a run stopped.

Automation: only new rows, notifications, schedules

Only new rows since the last run. Turn on onlyNew and the actor remembers, for every page it delivered to you, a fingerprint of its text (in a key-value store named rowcraft-website-markdown-extractor-state in your Apify account). The next run with the same stateName delivers a page again only when its text changed; pages with the same text are skipped and not charged. A change of the menu, the footer or a banner does not count, because only the extracted main content is compared. The first run delivers every page it can read. Use a different stateName for each list so they keep separate memories. Pages that print the current date or a counter in their body look changed on every run.

Notification when the run ends. Put a webhook address in notifyUrl and the actor sends one JSON POST with the number of delivered rows, links to the dataset (JSON and CSV) and the run. It works with Slack and Discord incoming webhooks (the message is in text and content), with Make, n8n and Zapier catch hooks, and with your own endpoint. By default it is sent only when at least one row was delivered; notifyWhenEmpty sends it every time.

Schedules. Create a schedule in Apify Console (daily, weekly, monthly or cron) with onlyNew on, and you get a feed of the pages of your list that changed, for the price of those pages only.

Integrations. Send the results to Make, n8n, Zapier, Google Sheets, Slack, Google Drive, Airtable or any webhook from the Integrations tab of the Actor, with no code. The dataset can also be loaded into LangChain and LlamaIndex pipelines and vector databases through the Apify integrations: the markdown column is the document text and finalUrl and title are natural metadata.

API, SDKs and AI agents. Every run is available through the Apify API and the official JavaScript and Python clients; the Actor can also be called by AI agents through the Apify MCP server. The example input in the API tab is a working call.

Use cases

• RAG and knowledge bases: convert documentation, help-center articles and policies to Markdown, split them by headings, embed them, and refresh the set on a schedule with Only new rows.
• LLM context: fetch the clean text of a handful of pages as the grounding for a prompt or an agent step, without feeding the model menus and cookie banners.
• Content monitoring: watch competitor pages, price lists, terms or release notes and get a row only when the text changes.
• Archiving and migration: keep a Markdown copy of articles or move content from an old site to a new one, with images and links already absolute.
• SEO and content audits: word counts, headings, descriptions, canonical addresses and publication dates for a list of pages, in one table.
• Translation and summarisation pipelines: start from clean text with the headings kept, instead of raw HTML.
• Dataset building: collect article text in many languages for classification, topic models and evaluation sets.

Ready-made examples, each with the input filled in (open one and press Start, or save it as your own task):

• Documentation pages as clean Markdown for RAG
• Long essays and articles as plain text with word counts
• Encyclopedia pages in three languages as plain text

What this actor does not do

• Crawl a website: only the addresses you list are read, never the links on them, the sitemap or the rest of the site. To get a list of addresses first, use the Sitemap URL Extractor and paste the result here.
• Run JavaScript: pages that build their text in the browser after loading (many single-page apps) are not rendered and come back as no_content. Only the HTML the server sends is read.
• Read PDF files, images, feeds or other files: an address that answers with something other than an HTML page is reported as error with the content type and not charged.
• Log in, accept cookie banners, use proxies or solve CAPTCHAs: pages that answer with an anti-bot page are reported as blocked and not charged.
• Return authors, bylines or contact data: no author, creator, e-mail address, phone number or social profile is delivered.
• Guarantee that every page has one clear article: for listings, home pages and forums the result is the most text-like part of the page, which may not be what you want. Whole page and Remove these elements first let you steer it.

Limits and fair use

• One request per second to each host by default (minimum 250 ms). A page takes two requests the first time its host is seen (robots.txt and the page) and one after that, plus any redirects. Different hosts are read in parallel.
• Page size: the first 5 MB of a page are read; larger pages are cut there and counted in the run summary. Scripts, styles and inline images are removed before the page is parsed. A page with more elements than the memory of the run allows (about 150,000 on the default 256 MB, more with more memory) is used as far as it was read and noted in COVERAGE.
• Row size: the Markdown and the text of a page are cut at maxChars (100,000 by default); wordCount and charCount always describe the whole text.
• Pages with fewer than 20 words of main content are not delivered or charged: they are listed as no_content. A page without a clear article, such as a home page made of links, is listed the same way.
• Anti-bot pages. Pages behind a bot challenge (Cloudflare, Akamai, DataDome and similar) are reported as blocked and not charged. After 10 blocked pages in a row the run stops and says why in the run summary, because the rest of the list would most likely be blocked too.
• robots.txt is obeyed for every address the visit touches, including redirect targets, for the exact path. If robots.txt disallows the page it is robots_disallowed; if robots.txt answers with a server error the site is not read, as the robots.txt standard (RFC 9309) requires.
• Up to 100,000 pages per run. The run stops 90 seconds before its timeout with the summary, the coverage and the Only new rows memory saved; for long lists raise the timeout or split the list.

Data source and licence

The data is what each page sends to every visitor: its HTML. The actor reads the pages you list, once each, as a browser does on the first visit, after checking robots.txt, and turns the text of the main content into Markdown and plain text. The content of a page belongs to its publisher and may be protected by copyright and by the terms of the website. You are responsible for checking that you may copy, store, index or feed to a model the pages you convert; the actor does not change those rights.

Rowcraft is independent: this actor is not provided, endorsed or authorised by the websites it reads. Website and product names belong to their owners and appear only in examples.

Privacy: no personal data

Rowcraft actors are built not to collect personal data:

• The author, byline, creator and contact blocks of a page are cut and never read from the metadata or the structured data.
• mailto:, tel:, sms: and javascript: links are never written anywhere; only their link text stays.
• E-mail addresses, phone numbers and personal profile links written in the text are replaced by [email removed], [phone removed] and [profile link removed], and contact-person names next to an honorific or a contact label such as "Contact:" by [name removed]. Other names in running text cannot be told apart from ordinary words and stay, so a page about people can still name them.
• The actor sends no cookies, fills no forms and never logs in.

You are responsible for using the data in line with the law that applies to you.

Other Rowcraft actors

All Rowcraft actors read official or public sources, deliver flat rows without personal data and support only-new runs and notifications:

• Ashby Jobs Scraper – Job Postings with Salary Ranges — Job postings from Ashby job boards (jobs.ashbyhq.com) by company: title, department, team, locations with country, employment and workplace type, published salary range, full text.
• Broken Link Checker – Find 404s and Dead Links API — Pay per crawled page, not per link: crawl a site within robots.txt and check every link on each page for 404s, 5xx errors, DNS and TLS failures and redirects.
• Bulk Image Downloader API: Save Page Images with Sizes — Download the images of any web pages, or direct image links, into a key-value store with width, height, format, size and a download link per file.
• Career Site Jobs Scraper: Greenhouse, Lever, Ashby, Workday — Job listings from the career pages of the companies you choose on Greenhouse, Lever, Ashby, Workday, Workable, Personio, Recruitee, Teamtailor and Gem, in one run and one format: title, locations, department, salary when published, full text.
• Chrome Web Store Scraper: Users, Ratings & Versions — Reads public Chrome Web Store listing pages with no login, no proxy and no personal data: users, rating, category, version, update date, permissions and company developers.
• Domain Email and DNS Checker API: MX, SPF, DMARC, Providers — Find which email provider, sending services and DNS host each domain uses, from public DNS records only.
• SIRENE France Company Scraper: SIREN, NAF, Revenue, Status — French companies from the official Recherche d'entreprises API (INSEE SIRENE + INPI): SIREN and SIRET, legal form, NAF activity, head-office address, employee band, revenue and net income, labels, creation date and VAT number.
• Gem Jobs Scraper – Job Board Listings from jobs.gem.com — Job listings from Gem job boards (jobs.gem.com) by company: title, department, offices with country, employment type, remote, hybrid or on-site, requisition ID, publication date and full ad text, from the public Gem Job Board API.
• Google Play Apps Scraper: App Details, Installs, Ratings — Public Google Play app details as flat rows: installs, rating, number of ratings, price, version, last update, minimum Android, ads and in-app purchases.
• Greenhouse Jobs Scraper – Job Listings by Company Board — Job listings from Greenhouse job boards by company: title, departments, offices, publication date, full ad text, published pay ranges.
• Himalayas Jobs Scraper: Remote Jobs API with Salaries — Remote jobs from Himalayas through its public jobs API: title, company, salary range, seniority, categories, the countries and time zones open to candidates, full description and job link.
• KRS Poland Companies Scraper: Register Data, PKD, Status — Polish companies from the official KRS register API of the Ministry of Justice: name, KRS, NIP and REGON numbers, legal form, status, address, PKD activity codes, share capital, branches and registration dates.
• LEI Records Scraper: Legal Entity Identifier Lookup API — Legal Entity Identifier (LEI) records from the Global LEI Index, read through the public GLEIF API: legal name, legal form, register number, addresses, status, renewal date and parent companies.
• Lever Jobs Scraper – Job Postings from jobs.lever.co — Job postings from Lever career pages (jobs.lever.co) by company: title, team, department, location, commitment, workplace type, salary range when published, description, requirements and benefits.
• Norway Company Register Scraper: Brønnøysund Business Data — Norwegian companies from the Central Coordinating Register for Legal Entities (Brønnøysund, NLOD open data): organisation number, legal form, industry, employees, share capital, addresses, website, VAT and bankruptcy status.
• PDF Text Extractor: Text, Pages and Metadata from PDF Links — Get the text of every PDF you link, page by page, with page count, language and word count.
• Personio Jobs Scraper: Job Listings, Salaries, Departments — Job ads from any company's Personio careers feed: title, department, office, employment type, schedule, seniority, salary when published, keywords and the full description in the language you choose.
• Platsbanken Scraper: Sweden Jobs, Employers, Org Numbers — Every job ad on Platsbanken (Arbetsförmedlingen) from the official JobTech API: headline, employer and organisation number, occupation codes, municipality, coordinates, salary type, deadline and full text.
• Recruitee Jobs Scraper – Job Listings by Company Career Site — Job listings from Recruitee career sites by company: title, department, locations with country, employment type, remote flag, published salary and full text, one request per company.
• Robots.txt and AI Crawler Access Checker API — Check which AI and search crawlers a website lets in: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, read from robots.txt by the RFC 9309 rules.
• RSS Feed Finder and Reader – Blog and News Article API — Find the RSS, Atom and JSON feeds of any website and read their newest articles, from a domain name alone.
• SEO Audit Tool: Titles, Meta Tags, Headings and Links API — Audit the on-page SEO of any list of pages or websites: titles, meta descriptions, headings, canonicals, indexing, links, images and structured data, with a score and the issues found per page.
• Shopify Products Scraper: Store Catalog Export to JSON, CSV — Products from any Shopify store's public catalog: title, vendor, type, tags, description, variants with price, compare-at price, SKU and stock, images, options, store currency.
• Sitemap URL Extractor: Every Page with Last-Modified Dates — Get every page a site lists in its sitemaps, with last-modified dates, from robots.txt through sitemap indexes and .gz files.
• SSL Certificate and Security Headers Checker API — Check the TLS certificate and the HTTP security headers of every domain in one run, with an A+ to F grade and the exact issues to fix.
• Store Products Scraper – Shopify, WooCommerce, VTEX — Export the product catalogue of any Shopify, WooCommerce or VTEX store from its public catalogue endpoint: title, brand, prices, discounts, stock, categories, images and links, one row per product, the same columns on every platform.
• Structured Data Extractor: JSON-LD, Schema.org, Open Graph — Pull the schema.org JSON-LD, Microdata, RDFa and Open Graph of any page list into flat rows: types, product price, availability, ratings, organization, breadcrumbs, with validation issues.
• Teamtailor Jobs Scraper – Career Site Jobs by Company — Job listings from Teamtailor career sites by company: title, department, role, locations with city and country, remote or hybrid status, publication date and full ad text, from the official job feed.
• TED Tenders Scraper: EU Public Procurement Notices API — EU public procurement notices from TED (Tenders Electronic Daily), the official EU tenders portal, through its public Search API: title, buyer, country, CPV codes, values, deadlines, lots and document links.
• UK Companies House Scraper: Company Search & Profiles API — UK companies from the official Companies House API with your free key: search by name, status, type, SIC code, date or location, or look up company numbers.
• UK Tenders Scraper: Find a Tender + Contracts Finder API — UK public procurement notices from Find a Tender and Contracts Finder, the official UK government portals, through their open OCDS APIs: title, buyer, region, CPV codes, values, deadlines, notice type and links.
• Bulk URL Status and Redirect Chain Checker API — Check the status code, redirect chain, canonical and indexing signals of every URL in a list, with the final destination and a loop flag.
• VTEX Products Scraper – Public Catalog Export — Export the catalogue of any VTEX store (Auchan, Chedraui and thousands of retailers in Europe and Latin America) from its public search endpoint: title, brand, EAN, prices, list prices, stock, categories, specifications and images, one row per product.
• Website Screenshot API: Full-Page PNG, JPEG and PDF Captures — Capture full-page or viewport screenshots of any public web page as PNG, JPEG or WebP, with an optional A4 PDF.
• Website Tech Stack Detector – Technology Lookup API — Find the technologies behind a list of websites: CMS, e-commerce platform, analytics, tag managers, ad pixels, consent tools, chat, CDN and payments, from one polite home-page request per domain.
• WooCommerce Products Scraper – Store API Export — Export every product of a WooCommerce store through its public Store API: title, SKU, prices and sale prices, stock, categories, tags, attributes, ratings, images and links, one row per product.
• Workable Jobs Scraper: Job Listings from Career Pages — Job listings from any company's Workable career page: title, department, location, remote or on-site, employment type, description, requirements, benefits and salary when published.
• Workday Jobs Scraper – Jobs from myworkdayjobs.com Sites — Job listings from Workday career sites (myworkdayjobs.com) by company: title, locations, country, posting date, time type, requisition ID and the full description.

FAQ

How does it find the main content? The page is parsed, noise is cut (navigation, footers, sidebars, forms, banners, ads, comments, hidden elements, share bars), every paragraph is scored by its length, punctuation and the hints in its class names, and the container with the best score is chosen. A parent container is added when it only adds text and not links (a title and a date above the body, the other sections of a long article). Links-only blocks inside the chosen container are cut.

Which pages work best? Articles, blog posts, documentation, help-center pages, encyclopedia pages, release notes, policies and product descriptions with real paragraphs. Home pages, category pages and search results have little running text, so the result is the most text-like part of them or no_content.

Why is the result empty? Nothing was delivered because every page was skipped or failed. The COVERAGE record gives the reason per page: an anti-bot page (blocked), robots.txt (robots_disallowed), no page at that address (not_found), something that is not an HTML page or a connection problem (error), a page with no readable main content or under 20 words (no_content), or an unchanged text with Only new rows on. Nothing is charged for those, apart from the start event.

Are pages rendered with JavaScript supported? No. Pages rendered with JavaScript are not supported: the actor reads the HTML the server sends, as a search engine bot without a browser would. If the text only appears after scripts run, the page comes back as no_content. Many sites that look dynamic still send their article text in the HTML, so it is worth a short test.

Is there any author or personal data in the output? There is no author or personal data field. The author, byline and contact blocks are cut, mailto: and tel: links are dropped, and e-mail addresses, phone numbers and personal profile links in the text are replaced by placeholders. See the Privacy section for what the filter does and does not cover.

Is it legal? The actor reads public pages that any visitor can open, obeys robots.txt, uses a clear user agent and keeps its request rate low. Whether you may copy and use the text of a page depends on the page's copyright and terms and on the law that applies to you; check that before you build a dataset or a model from other people's content.

Do I pay for pages that fail or have no content? No. Blocked, robots-disallowed, not found, not HTML, empty and too short pages, invalid entries, duplicates and unchanged pages (with Only new rows) are never charged. The only extra charge is the start event of $0.00005 per run.

Can it read PDF files or crawl a whole site? No to both. PDF files and other non-HTML addresses are reported as error and not charged. Only the addresses you list are read; use a sitemap list as the input when you need a whole section of a site.

How are tables, code and lists converted? Tables become Markdown tables when they are simple (a header row, at least two columns); tables used for page layout are read as ordinary paragraphs. Code blocks become fenced blocks with the language taken from the class of the block when the page names it. Lists keep their nesting and their numbering.

A banner or a box is still in my result. What can I do? Add it to Remove these elements first by its id or class (#newsletter, .promo), or switch Whole page on or off to see which gives the better result for that site. Send a short example to the Issues tab and the rules are improved.

How does Only new rows work here? The actor remembers a fingerprint of the extracted text of each page delivered to you. The next run with the same state name delivers a page only when that text changed, so a weekly schedule costs only the pages that changed.

What is the difference between url and finalUrl? url is the address you gave; finalUrl is the address that answered after redirects, and canonical is the address the page itself declares. The row id is the address you gave without its #fragment.

Support and reviews

Problems, missing fields, questions: open an issue on the Issues tab of this Actor. We answer within one working day and fix confirmed bugs in the next release; the changelog lists every change.

If this Actor saved you time, a short review on its page helps other users find it and helps us keep improving it.