No credit card required

Playwright Scraper

apify/playwright-scraper

No credit card required

Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

When trying to scrape a sitemap.xml - getting back a "document.body is null" error

Open

oren_clearya opened this issue

When running the Playwright Actor with a startUrl which is a sitemap XML - getting back the following error:

12024-10-19T14:54:41.339Z DEBUG PlaywrightCrawler:AutoscaledPool: scaling up {"oldConcurrency":2,"newConcurrency":3,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}}
22024-10-19T14:54:49.152Z WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.evaluate: document.body is null
32024-10-19T14:54:49.153Z @debugger eval code line 226 > eval:1:7
42024-10-19T14:54:49.154Z evaluate@debugger eval code:228:17
52024-10-19T14:54:49.155Z @debugger eval code:1:44
62024-10-19T14:54:49.155Z
72024-10-19T14:54:49.156Z     at CrawlerSetup._requestHandler (/home/myuser/dist/internals/crawler_setup.js:379:35) {"id":"vwv0onJJ2YlCPdo","url":"https://apify.com/sitemap.xml","retryCount":1}

It seems to fail before reaching the page function itself. However, here is the pageFunction that was used:

1async function pageFunction(context) {
2  const { page, request, log } = context;
3
4  async function pageEvaluate(context) {
5    return {
6      url: document.URL,
7      html: document.body?.innerHTML ?? document.querySelector('urlset')?.innerHTML,
8    };
9  }
10
11  let data = await page.evaluate(pageEva... [trimmed]

Add comment

Developer

Apify

Actor Metrics

68 monthly users
19 stars
>99% runs succeeded
54 days response time
Created in Aug 2022
Modified 6 months ago

Categories

Developer tools

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

5.7k

Facebook Marketplace

shmlkv/facebook-marketplace

This is a simple scraper for Facebook Marketplace. It uses Playwright to scrape the data

Andre Sh

450

Yellow-Pages-Scraper-withEmail

krish_patel/yellow-pages-scraper-withEmail

Allows you to gather leads from yellow pages along with their corresponding email if any.

Krish Patel

610

Redfin Fast Scraper

mantisus/redfin-fast-scraper

Redfin: Scrape fast, stay light! Skip bloated browser tools. My Redfin scraper extracts property data in a flash, no heavy lifting is needed. Scrape/monitor listings with ease, all without Puppeteer or Playwright. ⚡️

Maksym Bohomolov

Thefork Fast Scraper

mantisus/thefork-fast-scraper

Scrape TheFork.com quickly and easily! Skip bloated browser tools. This scraper extracts restaurant data in a flash, no heavy lifting is needed. Scrape and monitor data with ease, all without Puppeteer or Playwright. ⚡️

Maksym Bohomolov

Redfin Fast Scraper Per Results

mantisus/redfin-fast-scraper-per-results

Maksym Bohomolov

Zoopla.co.uk Fast Scraper

mantisus/zoopla-actor

Zoopla.co.uk: Scrape fast, stay light! Skip bloated browser tools. My Zoopla scraper extracts property data in a flash, no heavy lifting is needed. Scrape/monitor listings with ease, all without Puppeteer or Playwright. ⚡️

Maksym Bohomolov