tsboi index
Pricing
Pay per usage
Go to Apify Store
tsboi index
Indexing for LLMs. This application crawls specified websites, processes their content for LLMs.
tsboi index
Pricing
Pay per usage
Indexing for LLMs. This application crawls specified websites, processes their content for LLMs.
One or more URLs of pages where the crawler will start. Note that the Actor will additionally only crawl sub-pages of these URLs. For example, for start URL https://www.example.com/blog, it will crawl pages like https://example.com/blog/article-1, but will skip https://example.com/docs/something-else.
The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.
Enter your OpenAI account and an API key. This is needed for vectorizing the data and also to be able to prompt the OpenAI model.
