GPT Scraper

A static list of URLs to scrape.

For details, see Start URLs in README.

Instruct GPT how to generate text. For example: "Summarize this page in three sentences."

You can instruct OpenAI to answer with "skip this page", which will skip the page. For example: "Summarize this page in three sentences. If the page is about Apify Proxy, answer with 'skip this page'.".

Glob patterns matching URLs of pages that will be included in crawling. Combine them with the link selector to tell the scraper where to find links. You need to use both globs and link selector to crawl further pages.

Glob patterns matching URLs of pages that will be excluded from crawling. Note that this affects only links found on pages, but not Start URLs, which are always crawled.

This specifies how many links away from the Start URLs the scraper will descend. This value is a safeguard against infinite crawling depths for misconfigured scrapers.

If set to 0, there is no limit.

Maximum number of pages that the scraper will open. 0 means unlimited.

This is a CSS selector that says which links on the page (<a> elements with href attribute) should be followed and added to the request queue. To filter the links added to the queue, use the Pseudo-URLs setting.

If Link selector is empty, the page links are ignored.

For details, see Link selector in README.

Cookies that will be pre-set to all pages the scraper opens. This is useful for pages that require login. The value is expected to be a JSON array of objects with name, value, 'domain' and 'path' properties. For example: [{"name": "cookieName", "value": "cookieValue"}, "domain": ".domain.com", "path": "/"}].

You can use the EditThisCookie browser extension to copy browser cookies in this format, and paste it here.

This specifies the proxy servers that will be used by the scraper in order to hide its origin.

For details, see Proxy configuration in README.

Controls randomness: Lowering results in less random completions. As the temperature approaches zero, the model will become deterministic and repetitive. For consistent results, we recommend setting the temperature to 0.

Controls diversity via nucleus sampling: 0.5 means half of all likelihood-weighted options are considered.

How much to penalize new tokens based on their existing frequency in the text so far. Decreases the model's likelihood to repeat the same line verbatim.

How much to penalize new tokens based on whether they appear in the text so far. Increases the model's likelihood to talk about new topics.

A CSS selector of the HTML element on the page that will be used in the instruction. Instead of a whole page, you can use only part of the page. For example: "div#content".

A CSS selector matching HTML elements that will be removed from the DOM, before sending it to GPT processing. This is useful to skip irrelevant page content and save on GPT input tokens.

By default, the Actor removes usually unwanted elements like scripts, styles and inline images. You can disable the removal by setting this value to some non-existent CSS selector like dummy_keep_everything.

In what format to send the content extracted from the page to the GPT. Markdown will take less space allowing for larger requests, while HTML may help include some information like attributes that may otherwise be omitted.

The maximum time to wait for dynamic page content to load. The crawler will continue either if this time elapses, or if it detects the network became idle as there are no more requests for additional resources.

Removes web link URLs while keeping the text content they display.

• This helps reduce the total page content by eliminating unnecessary URLs before sending to GPT
• Useful if you are hitting maximum input tokens limits

If true, the answer will be transformed into a structured format based on the schema in the jsonAnswer attribute.

Defines how the output will be stored in structured format using the JSON Schema. Keep in mind that it uses function, so by setting the description of the fields and the correct title, you can get better results.

Description of the schema function. Use this to provide more context for the schema.

By default, the instructions field's value is used as the schema description, you can change it here.

For each page store its HTML, screenshot and parsed content (markdown/HTML as it was sent to ChatGPT) adding links to these into the output