No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoIntegrating with Datafetcher || Run Actor without "startUrls" ?
Closed
Has anybody integrated this with Datafetcher? It's a service to run the Actor against multiple URL's. I get an API error (400) message stating that "startUrls" is empty.
Is it possible to send a call without that parameter?
Hello and thank you for your interest in this Actor!
While I don't know anything about Datafetcher, it seems like you're passing the start URLs as a query parameter (in the URL, something like example.com?startUrls=...). That's not what the Apify API is expecting - if you want to run an Actor with input, you have to pass it as a request body of a POST request (see the details in the documentation).
Although there is a GET version of that endpoint that allows you to run an Actor without input, it doesn't work with this specific Actor (it needs to know, what to scrape - the startUrls input option in required).
If you find all of this inconvenient for some reason (e.g. you only want to scrape one / few pages), you can turn your Apify inputs into Saved tasks. A saved task is basically an Actor with "prefilled" input - you can call this using the "Run task synchronously" endpoint (and since you have already prefilled the input, you don't need to pass it from the Datafetcher and you can simply use the GET version of the endpoint).
You can create a Saved task from an Actor configuration by clicking the Create task button under the Actor title above.
Does this answer your question? If so, feel free to close this issue - or let us know, in case you need any more guidance. Cheers!
Closing due to inactivity.
Apify- 2.8k monthly users
- 434 stars
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago
