Dynamic Web Scraper

josejet/dynamic-web-scraper

Dynamic Web Scraper is an Apify Actor that gathers information online by simulating user browsing behavior on the web. It reduces the time and amount of scraped web pages by using a model (ChatGPT) to make decisions regarding browser navigation and results evaluation.

Developer

Pepa <b>J</b>

Actor Metrics

35 monthly users
No reviews yet
1 bookmark
>99% runs succeeded
Created in Feb 2025
Modified 13 days ago

Categories

You can access the Dynamic Web Scraper programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

1{
2  "openapi": "3.0.1",
3  "info": {
4    "version": "0.0",
5    "x-build-id": "3RUqw5zwczJUEKCIq"
6  },
7  "servers": [
8    {
9      "url": "https://api.apify.com/v2"
10    }
11  ],
12  "paths": {
13    "/acts/josejet~dynamic-web-scraper/run-sync-get-dataset-items": {
14      "post": {
15        "operationId": "run-sync-get-dataset-items-josejet-dynamic-web-scraper",
16        "x-openai-isConsequential": false,
17        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
18        "tags": [
19          "Run Actor"
20        ],
21        "requestBody": {
22          "required": true,
23          "content": {
24            "application/json": {
25              "schema": {
26                "$ref": "#/components/schemas/inputSchema"
27              }
28            }
29          }
30        },
31        "parameters": [
32          {
33            "name": "token",
34            "in": "query",
35            "required": true,
36            "schema": {
37              "type": "string"
38            },
39            "description": "Enter your Apify token here"
40          }
41        ],
42        "responses": {
43          "200": {
44            "description": "OK"
45          }
46        }
47      }
48    },
49    "/acts/josejet~dynamic-web-scraper/runs": {
50      "post": {
51        "operationId": "runs-sync-josejet-dynamic-web-scraper",
52        "x-openai-isConsequential": false,
53        "summary": "Executes an Actor and returns information about the initiated run in response.",
54        "tags": [
55          "Run Actor"
56        ],
57        "requestBody": {
58          "required": true,
59          "content": {
60            "application/json": {
61              "schema": {
62                "$ref": "#/components/schemas/inputSchema"
63              }
64            }
65          }
66        },
67        "parameters": [
68          {
69            "name": "token",
70            "in": "query",
71            "required": true,
72            "schema": {
73              "type": "string"
74            },
75            "description": "Enter your Apify token here"
76          }
77        ],
78        "responses": {
79          "200": {
80            "description": "OK",
81            "content": {
82              "application/json": {
83                "schema": {
84                  "$ref": "#/components/schemas/runsResponseSchema"
85                }
86              }
87            }
88          }
89        }
90      }
91    },
92    "/acts/josejet~dynamic-web-scraper/run-sync": {
93      "post": {
94        "operationId": "run-sync-josejet-dynamic-web-scraper",
95        "x-openai-isConsequential": false,
96        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
97        "tags": [
98          "Run Actor"
99        ],
100        "requestBody": {
101          "required": true,
102          "content": {
103            "application/json": {
104              "schema": {
105                "$ref": "#/components/schemas/inputSchema"
106              }
107            }
108          }
109        },
110        "parameters": [
111          {
112            "name": "token",
113            "in": "query",
114            "required": true,
115            "schema": {
116              "type": "string"
117            },
118            "description": "Enter your Apify token here"
119          }
120        ],
121        "responses": {
122          "200": {
123            "description": "OK"
124          }
125        }
126      }
127    }
128  },
129  "components": {
130    "schemas": {
131      "inputSchema": {
132        "type": "object",
133        "required": [
134          "startSources",
135          "prompt"
136        ],
137        "properties": {
138          "startSources": {
139            "title": "Start Sources",
140            "minItems": 1,
141            "type": "array",
142            "description": "Sources to start with, could be .",
143            "items": {
144              "type": "string"
145            }
146          },
147          "prompt": {
148            "title": "Prompt",
149            "type": "string",
150            "description": "Prompt to evaluate on content."
151          },
152          "maxDepth": {
153            "title": "Max Depth",
154            "type": "integer",
155            "description": "Maximum depth to scrape.",
156            "default": 3
157          }
158        }
159      },
160      "runsResponseSchema": {
161        "type": "object",
162        "properties": {
163          "data": {
164            "type": "object",
165            "properties": {
166              "id": {
167                "type": "string"
168              },
169              "actId": {
170                "type": "string"
171              },
172              "userId": {
173                "type": "string"
174              },
175              "startedAt": {
176                "type": "string",
177                "format": "date-time",
178                "example": "2025-01-08T00:00:00.000Z"
179              },
180              "finishedAt": {
181                "type": "string",
182                "format": "date-time",
183                "example": "2025-01-08T00:00:00.000Z"
184              },
185              "status": {
186                "type": "string",
187                "example": "READY"
188              },
189              "meta": {
190                "type": "object",
191                "properties": {
192                  "origin": {
193                    "type": "string",
194                    "example": "API"
195                  },
196                  "userAgent": {
197                    "type": "string"
198                  }
199                }
200              },
201              "stats": {
202                "type": "object",
203                "properties": {
204                  "inputBodyLen": {
205                    "type": "integer",
206                    "example": 2000
207                  },
208                  "rebootCount": {
209                    "type": "integer",
210                    "example": 0
211                  },
212                  "restartCount": {
213                    "type": "integer",
214                    "example": 0
215                  },
216                  "resurrectCount": {
217                    "type": "integer",
218                    "example": 0
219                  },
220                  "computeUnits": {
221                    "type": "integer",
222                    "example": 0
223                  }
224                }
225              },
226              "options": {
227                "type": "object",
228                "properties": {
229                  "build": {
230                    "type": "string",
231                    "example": "latest"
232                  },
233                  "timeoutSecs": {
234                    "type": "integer",
235                    "example": 300
236                  },
237                  "memoryMbytes": {
238                    "type": "integer",
239                    "example": 1024
240                  },
241                  "diskMbytes": {
242                    "type": "integer",
243                    "example": 2048
244                  }
245                }
246              },
247              "buildId": {
248                "type": "string"
249              },
250              "defaultKeyValueStoreId": {
251                "type": "string"
252              },
253              "defaultDatasetId": {
254                "type": "string"
255              },
256              "defaultRequestQueueId": {
257                "type": "string"
258              },
259              "buildNumber": {
260                "type": "string",
261                "example": "1.0.0"
262              },
263              "containerUrl": {
264                "type": "string"
265              },
266              "usage": {
267                "type": "object",
268                "properties": {
269                  "ACTOR_COMPUTE_UNITS": {
270                    "type": "integer",
271                    "example": 0
272                  },
273                  "DATASET_READS": {
274                    "type": "integer",
275                    "example": 0
276                  },
277                  "DATASET_WRITES": {
278                    "type": "integer",
279                    "example": 0
280                  },
281                  "KEY_VALUE_STORE_READS": {
282                    "type": "integer",
283                    "example": 0
284                  },
285                  "KEY_VALUE_STORE_WRITES": {
286                    "type": "integer",
287                    "example": 1
288                  },
289                  "KEY_VALUE_STORE_LISTS": {
290                    "type": "integer",
291                    "example": 0
292                  },
293                  "REQUEST_QUEUE_READS": {
294                    "type": "integer",
295                    "example": 0
296                  },
297                  "REQUEST_QUEUE_WRITES": {
298                    "type": "integer",
299                    "example": 0
300                  },
301                  "DATA_TRANSFER_INTERNAL_GBYTES": {
302                    "type": "integer",
303                    "example": 0
304                  },
305                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
306                    "type": "integer",
307                    "example": 0
308                  },
309                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
310                    "type": "integer",
311                    "example": 0
312                  },
313                  "PROXY_SERPS": {
314                    "type": "integer",
315                    "example": 0
316                  }
317                }
318              },
319              "usageTotalUsd": {
320                "type": "number",
321                "example": 0.00005
322              },
323              "usageUsd": {
324                "type": "object",
325                "properties": {
326                  "ACTOR_COMPUTE_UNITS": {
327                    "type": "integer",
328                    "example": 0
329                  },
330                  "DATASET_READS": {
331                    "type": "integer",
332                    "example": 0
333                  },
334                  "DATASET_WRITES": {
335                    "type": "integer",
336                    "example": 0
337                  },
338                  "KEY_VALUE_STORE_READS": {
339                    "type": "integer",
340                    "example": 0
341                  },
342                  "KEY_VALUE_STORE_WRITES": {
343                    "type": "number",
344                    "example": 0.00005
345                  },
346                  "KEY_VALUE_STORE_LISTS": {
347                    "type": "integer",
348                    "example": 0
349                  },
350                  "REQUEST_QUEUE_READS": {
351                    "type": "integer",
352                    "example": 0
353                  },
354                  "REQUEST_QUEUE_WRITES": {
355                    "type": "integer",
356                    "example": 0
357                  },
358                  "DATA_TRANSFER_INTERNAL_GBYTES": {
359                    "type": "integer",
360                    "example": 0
361                  },
362                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
363                    "type": "integer",
364                    "example": 0
365                  },
366                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
367                    "type": "integer",
368                    "example": 0
369                  },
370                  "PROXY_SERPS": {
371                    "type": "integer",
372                    "example": 0
373                  }
374                }
375              }
376            }
377          }
378        }
379      }
380    }
381  }
382}

Dynamic Web Scraper OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Dynamic Web Scraper from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Dynamic Web Scraper API in Python

Dynamic Web Scraper API in JavaScript

Dynamic Web Scraper API through CLI

Dynamic Web Scraper API

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

78.5k

4.5/5

Dynamic Markdown Scraper

louisdeconinck/dynamic-markdown-scraper

Effortlessly feed LLM AIs with clean Markdown using our advanced web scraper. Seamlessly scrape dynamic, JavaScript-rendered websites while preserving original formatting. Ideal for AI training, documentation, and content migration.

Louis Deconinck

5.0/5

RegExp Scraper

ib4ngz/regexp-scraper

This actor scrapes data from a list of provided URLs using regular expressions for precise and customizable pattern matching. It can handle both static and dynamic web pages and supports depth-based crawling to explore links and extract data from multiple levels of the web.

Iqbal R

Ai Web Scraper - Extract Data With Ease

eloquent_mountain/ai-web-scraper-extract-data-with-ease

Ai Web Scraper enables scraping for everyone, including non-techies! It uses Google's Gemini LLM to scrape websites with natural language commands. It dynamically extracts data, no selector input needed, handles dynamic content and cookie consent, avoids bot detection, outputs JSON or other formats.

Paco

201

Ai Web Scraper - Natural language and Vision scraper

eloquent_mountain/ai-universal-web-scraper-natural-language

Powerful AI Web Scraper using Google's Gemini Vision. Specify data extraction in natural language. Supports infinite scroll, above-the-fold analysis, automatic cookie consent, pay-per-event pricing, and screenshot storage for debugging.

Paco

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

6.8k

4.7/5

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

208

Legacy PhantomJS Crawler

apify/legacy-phantomjs-crawler

Replacement for the legacy Apify Crawler product with a backward-compatible interface. The actor uses PhantomJS headless browser to recursively crawl websites and extract data from them using a piece of front-end JavaScript code.

Apify

1.6k

5.0/5

Kroger Fast Product Scraper

tri_angle/kroger-fast-product-scraper

Quickly scrape product details from search listing pages on Kroger

Tri⟁angle

Actors MCP Server

apify/actors-mcp-server

Connect Apify Actors to AI applications (AI Agents) using Anthropic's Model Context Protocol (MCP). This MCP server provides integration of any Apify Actor through the MCP framework.

Apify

220

4.4/5