Web Scraping API
Pricing
from $2.00 / 1,000 pages
Web Scraping API
Web Scraping API that quickly and reliably scrapes any website—no selectors required. Premium proxies, CAPTCHA solving, JavaScript rendering, and automated structured data extraction are all included. It’s just $2 per 1,000 web pages scraped, with no minimum spend.
5.0 (5)
Pricing
from $2.00 / 1,000 pages
44
Total users
1K
Monthly users
611
Runs succeeded
96%
Issues response
8.2 hours
Last modified
13 days ago
AI Scraper Feedback: Rendering Issues and Next Link Robustness
Closed
Thanks for this Actor but I've been working extensively with the Web Scraping API actor and wanted to share some feedback based on our experience with scraping various e-commerce sites (public knowledge).
I've encountering a couple of patterns: Page Rendering Challenges: On certain product listing pages, the actor sometimes returns a "temporary error occurred while rendering the requested URL." I've implemented retries in My script, which helps, but some pages consistently fail to render for the actor across these retries. This leads to gaps in data collection from those specific pages. Next Page Link Consistency: In some earlier tests using the /next endpoint, I found it could occasionally return next page URLs with an altered path structure compared to the current page, which would break sequential subcategory scraping. I've had more consistent success with path integrity by switching to the /scrape endpoint and using a specific field prompt to ask the AI for the "next page URL href". Then I constructs the absolute URL. This seems to be a more reliable approach for complex pagination if the /next endpoint struggles. Minor URL Formatting Quirk (Less Frequent): I've also occasionally seen relative URLs in the AI's output with unusual prefixing (e.g., extra escaped slashes). I've added my-side cleaning for this, but thought it was worth noting. The primary concern for me is the rendering issue (Point 1), as it's the main factor currently leading to incomplete scrapes when it occurs. Any improvements to the actor's stability and rendering capabilities for diverse e-commerce page structures would be fantastic.
Hello! For each of the endpoints you are using, can you please share your input JSON payloads? I will test these out independently to figure out what's going wrong. Meanwhile, if you would like, it may be easier to discuss in detail via Discord. Feel free to connect with me there also: https://discord.gg/pZEJMCTzA3
Just a little update: a bug fix went out today, which could possibly be responsible for the errors you experienced in the rendering stage. This fix went out at about 2:30 am, Sunday, 25 May 2025, Eastern Time (ET). Again I welcome you to please update me and also join the Discord (https://discord.gg/pZEJMCTzA3) to share JSON payloads / troubleshoot things one-on-one with me. Thanks!
Hello, just following up on this one. I will soon close the issue if I don't hear back, but I would definitely love to see how you got on, if the bug fixes helped? Also please join our Discord if you like :)
MoeTheGreat
Im so sorry for the late reply U actually hs some irl stuff to do these 2 days
I replied in discord
Perfect, I got your message. I'll assist further there, and close this issue on Apify. Thanks!
