1# Apify Actors Development Guide
2
3Important: Before you begin, fill in the model property in the meta section of `.actor/actor.json`. Replace it with the model you're currently using. This helps Apify monitor and improve AGENTS.md for specific LLM models.
4
5## What are Apify Actors?
6
7- Actors are serverless programs that run in the cloud. They're inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems.
8- Actors are programs packaged as Docker images that run in isolated containers
9
10## Core Concepts
11
12- Accept well-defined JSON input
13- Perform isolated tasks (web scraping, automation, data processing)
14- Produce structured JSON output to datasets and/or store data in key-value stores
15- Can run from seconds to hours or even indefinitely
16- Persist state and can be restarted
17
18## Do
19
20- accept well-defined JSON input and produce structured JSON output
21- use Apify SDK (`apify`) for code running ON Apify platform
22- validate input early with proper error handling and fail gracefully
23- use CheerioCrawler for static HTML content (10x faster than browsers)
24- use PlaywrightCrawler only for JavaScript-heavy sites and dynamic content
25- use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls
26- implement retry strategies with exponential backoff for failed requests
27- use proper concurrency settings (HTTP: 10-50, Browser: 1-5)
28- set sensible defaults in `.actor/input_schema.json` for all optional fields
29- set up output schema in `.actor/output_schema.json`
30- clean and validate data before pushing to dataset
31- use semantic CSS selectors and fallback strategies for missing elements
32- respect robots.txt, ToS, and implement rate limiting with delays
33- check which tools (cheerio/playwright/crawlee) are installed before applying guidance
34
35## Don't
36
37- do not rely on `Dataset.getInfo()` for final counts on Cloud platform
38- do not use browser crawlers when HTTP/Cheerio works (massive performance gains with HTTP)
39- do not hard code values that should be in input schema or environment variables
40- do not skip input validation or error handling
41- do not overload servers - use appropriate concurrency and delays
42- do not scrape prohibited content or ignore Terms of Service
43- do not store personal/sensitive data unless explicitly permitted
44- do not use deprecated options like `requestHandlerTimeoutMillis` on CheerioCrawler (v3.x)
45- do not use `additionalHttpHeaders` - use `preNavigationHooks` instead
46
47## Commands
48
49```bash
50# Local development
51apify run # Run Actor locally
52
53# Authentication & deployment
54apify login # Authenticate account
55apify push # Deploy to Apify platform
56
57# Help
58apify help # List all commands
59```
60
61## Safety and Permissions
62
63Allowed without prompt:
64
65- read files with `Actor.getValue()`
66- push data with `Actor.pushData()`
67- set values with `Actor.setValue()`
68- enqueue requests to RequestQueue
69- run locally with `apify run`
70
71Ask first:
72
73- npm/pip package installations
74- apify push (deployment to cloud)
75- proxy configuration changes (requires paid plan)
76- Dockerfile changes affecting builds
77- deleting datasets or key-value stores
78
79## Project Structure
80
81.actor/
82├── actor.json # Actor config: name, version, env vars, runtime settings
83├── input_schema.json # Input validation & Console form definition
84└── output_schema.json # Specifies where an Actor stores its output
85src/
86└── main.js # Actor entry point and orchestrator
87storage/ # Local storage (mirrors Cloud during development)
88├── datasets/ # Output items (JSON objects)
89├── key_value_stores/ # Files, config, INPUT
90└── request_queues/ # Pending crawl requests
91Dockerfile # Container image definition
92AGENTS.md # AI agent instructions (this file)
93
94## Actor Input Schema
95
96The input schema defines the input parameters for an Actor. It's a JSON object comprising various field types supported by the Apify platform.
97
98### Structure
99
100```json
101{
102 "title": "<INPUT-SCHEMA-TITLE>",
103 "type": "object",
104 "schemaVersion": 1,
105 "properties": {
106 /* define input fields here */
107 },
108 "required": []
109}
110```
111
112### Example
113
114```json
115{
116 "title": "E-commerce Product Scraper Input",
117 "type": "object",
118 "schemaVersion": 1,
119 "properties": {
120 "startUrls": {
121 "title": "Start URLs",
122 "type": "array",
123 "description": "URLs to start scraping from (category pages or product pages)",
124 "editor": "requestListSources",
125 "default": [{ "url": "https://example.com/category" }],
126 "prefill": [{ "url": "https://example.com/category" }]
127 },
128 "followVariants": {
129 "title": "Follow Product Variants",
130 "type": "boolean",
131 "description": "Whether to scrape product variants (different colors, sizes)",
132 "default": true
133 },
134 "maxRequestsPerCrawl": {
135 "title": "Max Requests per Crawl",
136 "type": "integer",
137 "description": "Maximum number of pages to scrape (0 = unlimited)",
138 "default": 1000,
139 "minimum": 0
140 },
141 "proxyConfiguration": {
142 "title": "Proxy Configuration",
143 "type": "object",
144 "description": "Proxy settings for anti-bot protection",
145 "editor": "proxy",
146 "default": { "useApifyProxy": false }
147 },
148 "locale": {
149 "title": "Locale",
150 "type": "string",
151 "description": "Language/country code for localized content",
152 "default": "cs",
153 "enum": ["cs", "en", "de", "sk"],
154 "enumTitles": ["Czech", "English", "German", "Slovak"]
155 }
156 },
157 "required": ["startUrls"]
158}
159```
160
161## Actor Output Schema
162
163The Actor output schema builds upon the schemas for the dataset and key-value store. It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results.
164
165### Structure
166
167```json
168{
169 "actorOutputSchemaVersion": 1,
170 "title": "<OUTPUT-SCHEMA-TITLE>",
171 "properties": {
172 /* define your outputs here */
173 }
174}
175```
176
177### Example
178
179```json
180{
181 "actorOutputSchemaVersion": 1,
182 "title": "Output schema of the files scraper",
183 "properties": {
184 "files": {
185 "type": "string",
186 "title": "Files",
187 "template": "{{links.apiDefaultKeyValueStoreUrl}}/keys"
188 },
189 "dataset": {
190 "type": "string",
191 "title": "Dataset",
192 "template": "{{links.apiDefaultDatasetUrl}}/items"
193 }
194 }
195}
196```
197
198### Output Schema Template Variables
199
200- `links` (object) - Contains quick links to most commonly used URLs
201- `links.publicRunUrl` (string) - Public run url in format `https://console.apify.com/view/runs/:runId`
202- `links.consoleRunUrl` (string) - Console run url in format `https://console.apify.com/actors/runs/:runId`
203- `links.apiRunUrl` (string) - API run url in format `https://api.apify.com/v2/actor-runs/:runId`
204- `links.apiDefaultDatasetUrl` (string) - API url of default dataset in format `https://api.apify.com/v2/datasets/:defaultDatasetId`
205- `links.apiDefaultKeyValueStoreUrl` (string) - API url of default key-value store in format `https://api.apify.com/v2/key-value-stores/:defaultKeyValueStoreId`
206- `links.containerRunUrl` (string) - URL of a webserver running inside the run in format `https://<containerId>.runs.apify.net/`
207- `run` (object) - Contains information about the run same as it is returned from the `GET Run` API endpoint
208- `run.defaultDatasetId` (string) - ID of the default dataset
209- `run.defaultKeyValueStoreId` (string) - ID of the default key-value store
210
211## Dataset Schema Specification
212
213The dataset schema defines how your Actor's output data is structured, transformed, and displayed in the Output tab in the Apify Console.
214
215### Example
216
217Consider an example Actor that calls `Actor.pushData()` to store data into dataset:
218
219```javascript
220import { Actor } from 'apify';
221// Initialize the JavaScript SDK
222await Actor.init();
223
224/**
225 * Actor code
226 */
227await Actor.pushData({
228 numericField: 10,
229 pictureUrl: 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png',
230 linkUrl: 'https://google.com',
231 textField: 'Google',
232 booleanField: true,
233 dateField: new Date(),
234 arrayField: ['#hello', '#world'],
235 objectField: {},
236});
237
238// Exit successfully
239await Actor.exit();
240```
241
242To set up the Actor's output tab UI, reference a dataset schema file in `.actor/actor.json`:
243
244```json
245{
246 "actorSpecification": 1,
247 "name": "book-library-scraper",
248 "title": "Book Library Scraper",
249 "version": "1.0.0",
250 "storages": {
251 "dataset": "./dataset_schema.json"
252 }
253}
254```
255
256Then create the dataset schema in `.actor/dataset_schema.json`:
257
258```json
259{
260 "actorSpecification": 1,
261 "fields": {},
262 "views": {
263 "overview": {
264 "title": "Overview",
265 "transformation": {
266 "fields": [
267 "pictureUrl",
268 "linkUrl",
269 "textField",
270 "booleanField",
271 "arrayField",
272 "objectField",
273 "dateField",
274 "numericField"
275 ]
276 },
277 "display": {
278 "component": "table",
279 "properties": {
280 "pictureUrl": {
281 "label": "Image",
282 "format": "image"
283 },
284 "linkUrl": {
285 "label": "Link",
286 "format": "link"
287 },
288 "textField": {
289 "label": "Text",
290 "format": "text"
291 },
292 "booleanField": {
293 "label": "Boolean",
294 "format": "boolean"
295 },
296 "arrayField": {
297 "label": "Array",
298 "format": "array"
299 },
300 "objectField": {
301 "label": "Object",
302 "format": "object"
303 },
304 "dateField": {
305 "label": "Date",
306 "format": "date"
307 },
308 "numericField": {
309 "label": "Number",
310 "format": "number"
311 }
312 }
313 }
314 }
315 }
316}
317```
318
319### Structure
320
321```json
322{
323 "actorSpecification": 1,
324 "fields": {},
325 "views": {
326 "<VIEW_NAME>": {
327 "title": "string (required)",
328 "description": "string (optional)",
329 "transformation": {
330 "fields": ["string (required)"],
331 "unwind": ["string (optional)"],
332 "flatten": ["string (optional)"],
333 "omit": ["string (optional)"],
334 "limit": "integer (optional)",
335 "desc": "boolean (optional)"
336 },
337 "display": {
338 "component": "table (required)",
339 "properties": {
340 "<FIELD_NAME>": {
341 "label": "string (optional)",
342 "format": "text|number|date|link|boolean|image|array|object (optional)"
343 }
344 }
345 }
346 }
347 }
348}
349```
350
351**Dataset Schema Properties:**
352
353- `actorSpecification` (integer, required) - Specifies the version of dataset schema structure document (currently only version 1)
354- `fields` (JSONSchema object, required) - Schema of one dataset object (use JsonSchema Draft 2020-12 or compatible)
355- `views` (DatasetView object, required) - Object with API and UI views description
356
357**DatasetView Properties:**
358
359- `title` (string, required) - Visible in UI Output tab and API
360- `description` (string, optional) - Only available in API response
361- `transformation` (ViewTransformation object, required) - Data transformation applied when loading from Dataset API
362- `display` (ViewDisplay object, required) - Output tab UI visualization definition
363
364**ViewTransformation Properties:**
365
366- `fields` (string[], required) - Fields to present in output (order matches column order)
367- `unwind` (string[], optional) - Deconstructs nested children into parent object
368- `flatten` (string[], optional) - Transforms nested object into flat structure
369- `omit` (string[], optional) - Removes specified fields from output
370- `limit` (integer, optional) - Maximum number of results (default: all)
371- `desc` (boolean, optional) - Sort order (true = newest first)
372
373**ViewDisplay Properties:**
374
375- `component` (string, required) - Only `table` is available
376- `properties` (Object, optional) - Keys matching `transformation.fields` with ViewDisplayProperty values
377
378**ViewDisplayProperty Properties:**
379
380- `label` (string, optional) - Table column header
381- `format` (string, optional) - One of: `text`, `number`, `date`, `link`, `boolean`, `image`, `array`, `object`
382
383## Key-Value Store Schema Specification
384
385The key-value store schema organizes keys into logical groups called collections for easier data management.
386
387### Example
388
389Consider an example Actor that calls `Actor.setValue()` to save records into the key-value store:
390
391```javascript
392import { Actor } from 'apify';
393// Initialize the JavaScript SDK
394await Actor.init();
395
396/**
397 * Actor code
398 */
399await Actor.setValue('document-1', 'my text data', { contentType: 'text/plain' });
400
401await Actor.setValue(`image-${imageID}`, imageBuffer, { contentType: 'image/jpeg' });
402
403// Exit successfully
404await Actor.exit();
405```
406
407To configure the key-value store schema, reference a schema file in `.actor/actor.json`:
408
409```json
410{
411 "actorSpecification": 1,
412 "name": "data-collector",
413 "title": "Data Collector",
414 "version": "1.0.0",
415 "storages": {
416 "keyValueStore": "./key_value_store_schema.json"
417 }
418}
419```
420
421Then create the key-value store schema in `.actor/key_value_store_schema.json`:
422
423```json
424{
425 "actorKeyValueStoreSchemaVersion": 1,
426 "title": "Key-Value Store Schema",
427 "collections": {
428 "documents": {
429 "title": "Documents",
430 "description": "Text documents stored by the Actor",
431 "keyPrefix": "document-"
432 },
433 "images": {
434 "title": "Images",
435 "description": "Images stored by the Actor",
436 "keyPrefix": "image-",
437 "contentTypes": ["image/jpeg"]
438 }
439 }
440}
441```
442
443### Structure
444
445```json
446{
447 "actorKeyValueStoreSchemaVersion": 1,
448 "title": "string (required)",
449 "description": "string (optional)",
450 "collections": {
451 "<COLLECTION_NAME>": {
452 "title": "string (required)",
453 "description": "string (optional)",
454 "key": "string (conditional - use key OR keyPrefix)",
455 "keyPrefix": "string (conditional - use key OR keyPrefix)",
456 "contentTypes": ["string (optional)"],
457 "jsonSchema": "object (optional)"
458 }
459 }
460}
461```
462
463**Key-Value Store Schema Properties:**
464
465- `actorKeyValueStoreSchemaVersion` (integer, required) - Version of key-value store schema structure document (currently only version 1)
466- `title` (string, required) - Title of the schema
467- `description` (string, optional) - Description of the schema
468- `collections` (Object, required) - Object where each key is a collection ID and value is a Collection object
469
470**Collection Properties:**
471
472- `title` (string, required) - Collection title shown in UI tabs
473- `description` (string, optional) - Description appearing in UI tooltips
474- `key` (string, conditional) - Single specific key for this collection
475- `keyPrefix` (string, conditional) - Prefix for keys included in this collection
476- `contentTypes` (string[], optional) - Allowed content types for validation
477- `jsonSchema` (object, optional) - JSON Schema Draft 07 format for `application/json` content type validation
478
479Either `key` or `keyPrefix` must be specified for each collection, but not both.
480
481## Apify MCP Tools
482
483If MCP server is configured, use these tools for documentation:
484
485- `search-apify-docs` - Search documentation
486- `fetch-apify-docs` - Get full doc pages
487
488Otherwise, reference: `@https://mcp.apify.com/`
489
490## Resources
491
492- [docs.apify.com/llms.txt](https://docs.apify.com/llms.txt) - Quick reference
493- [docs.apify.com/llms-full.txt](https://docs.apify.com/llms-full.txt) - Complete docs
494- [crawlee.dev](https://crawlee.dev) - Crawlee documentation
495- [whitepaper.actor](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete Actor specification