LLM Dataset Processor

Pricing

Pay per usage

LLM Dataset Processor

Allows you to process output of other actors or stored dataset with single LLM prompt. It's useful if you need to enrich data, summarize content, extract specific information, or manipulate data in a structured way using AI.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Dušan Vystrčil

Maintained by Community

Actor stats

Bookmarked

112

Total users

Monthly active users

21 days ago

Last modified

.actor/Dockerfile

# Specify the base Docker image. You can read more about
# the available images at https://crawlee.dev/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node:20 AS builder

# Check preinstalled packages
RUN npm ls crawlee apify puppeteer playwright

# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY package*.json ./

# Install all dependencies. Don't audit to speed up the installation.
RUN npm install --include=dev --audit=false

# Next, copy the source files using the user set
# in the base image.
COPY . ./

# Install all dependencies and build the project.
# Don't audit to speed up the installation.
RUN npm run build

# Create final image
FROM apify/actor-node:20

# Check preinstalled packages
RUN npm ls crawlee apify puppeteer playwright

# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY package*.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
    && npm install --omit=dev --omit=optional \
    && echo "Installed NPM packages:" \
    && (npm list --omit=dev --all || true) \
    && echo "Node.js version:" \
    && node --version \
    && echo "NPM version:" \
    && npm --version \
    && rm -r ~/.npm

# Copy built JS files from builder image
COPY --from=builder /usr/src/app/dist ./dist

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY . ./


# Run the image.
CMD npm run start:prod --silent

.actor/actor.json

{
	"actorSpecification": 1,
	"name": "llm-dataset-processor",
	"title": "LLM Dataset Processor",
	"description": "Process or enrich datasets with LLM-generated content.",
	"version": "0.0",
	"meta": {
		"templateId": "ts-crawlee-cheerio"
	},
	"input": "./input_schema.json",
	"dockerfile": "./Dockerfile"
}

.actor/input_schema.json

{
    "title": "LLM Dataset Processor",
    "description": "Choose specific dataset to process, select LLM, provide API token and craft your prompt template. We recommend testing your prompt first by enabling `Test Prompt Mode`. ",
    "type": "object",
    "schemaVersion": 1,
    "required": [
        "llmProviderApiKey",
        "prompt",
        "model",
        "temperature",
        "maxTokens"
    ],
    "properties": {
        "inputDatasetId": {
            "type": "string",
            "title": "Input Dataset ID",
            "description": "The ID of the dataset to process.",
            "resourceType": "dataset"
        },
        "model": {
            "type": "string",
            "title": "Large Language Model",
            "description": "The LLM to use for processing. Each model has different capabilities and pricing. GPT-4o-mini and Claude 3.5 Haiku are recommended for cost-effective processing, while models like Claude 3 Opus or GPT-4o offer higher quality but at a higher cost.",
            "editor": "select",
            "enumTitles": ["GPT-4o mini (Recommended)", "GPT-4o", "Claude 3.5 Haiku (Recommended)", "Claude 3.5 Sonnet", "Claude 3 Opus", "Gemini 1.5 Flash", "Gemini 1.5 Flash-8B (Recommended)" ,"Gemini 1.5 Pro"],
            "enumSuggestedValues": ["gpt-4o-mini", "gpt-4o", "claude-3-5-haiku-latest", "claude-3-5-sonnet-latest", "claude-3-opus-latest", "gemini-1.5-flash", "gemini-1.5-flash-8b", "gemini-1.5-pro"]
        },
        "llmProviderApiKey": {
            "type": "string",
            "title": "LLM Provider API Key",
            "editor": "textfield",
            "description": "Your API key for the LLM Provider (e.g., OpenAI).",
            "isSecret": true
        },
        "temperature": {
            "type": "string",
            "title": "Temperature",
            "editor": "textfield",
            "description": "Sampling temperature for the LLM API (controls randomness). We recommend using a value closer to 0 for exact results. In case of more 'creative' results, we recommend to use a value closer to 1.",
            "default": "0.1"
        },
        "multipleColumns": {
            "type": "boolean",
            "title": "Multiple columns in output",
            "description": "When enabled, instructs the LLM to return responses as JSON objects, creating multiple columns in the output dataset. The columns need to be named and described in the prompt. If disabled, responses are stored in a single `llmresponse` column.",
            "default": false
        },
        "prompt": {
            "type": "string",
            "title": "Prompt Template",
            "description": "The prompt template to use for processing. You can use ${fieldName} placeholders to reference fields from the input dataset.",
            "editor": "textarea",
            "minLength": 1,
            "prefill": "Summarize this text: ${text}"
        },
        "skipItemIfEmpty": {
            "type": "boolean",
            "title": "Skip item if one or more ${fields} are empty",
            "description": "When enabled, items will be skipped if any ${field} referenced in the prompt is empty, null, undefined, or contains only whitespace. This helps prevent processing incomplete data.",
            "default": true
        },
        "maxTokens": {
            "type": "integer",
            "title": "Max Tokens",
            "editor": "number",
            "description": "Maximum number of tokens in the LLM API response for each item.",
            "default": 300
        },
        "testPrompt": {
            "type": "boolean",
            "title": "Test Prompt Mode",
            "description": "Test mode that processes only a limited number of items (defined by `testItemsCount`). Use this to validate your prompt and configuration before running on the full dataset. We highly recommend enabling this option first to validate your prompt because of ambiguity of the LLM responses.",
            "default": true
        },
        "testItemsCount": {
            "type": "integer",
            "title": "Test Items Count",
            "description": "Number of items to process when `Test Prompt Mode` is enabled.",
            "default": 3,
            "minimum": 1
        },
        "preprocessingFunction": {
            "title": "Preprocessing function",
            "type": "string",
            "description": "Function to transform item before they are put into the promp.",
            "editor": "javascript",
            "prefill": "(item) => {\n    return item;\n}",
            "sectionCaption": "Transforming functions",
            "sectionDescription": "Optionally transform the data. The function is called for each item."
        }
    }
}

src/providers/anthropic.ts

1import Anthropic from '@anthropic-ai/sdk';
2import { LLMProvider } from '../types.js';
3
4export class AnthropicProvider implements LLMProvider {
5    private client: Anthropic;
6
7    constructor(apiKey: string) {
8        this.client = new Anthropic({ apiKey });
9    }
10
11    async call(promptText: string, model: string, temperature: number, maxTokens: number): Promise<string> {
12        const message = await this.client.messages.create({
13            model,
14            max_tokens: maxTokens,
15            temperature,
16            messages: [{ role: 'user', content: promptText }],
17        });
18
19        if (!message.content || message.content.length === 0) {
20            return '';
21        }
22
23        const textContent = message.content.find((c) => c.type === 'text');
24        return textContent?.text || '';
25    }
26}

src/providers/google.ts

1import { GoogleGenerativeAI } from '@google/generative-ai';
2import { LLMProvider } from '../types.js';
3
4export class GoogleProvider implements LLMProvider {
5    private client: GoogleGenerativeAI;
6
7    constructor(apiKey: string) {
8        this.client = new GoogleGenerativeAI(apiKey);
9    }
10
11    async call(promptText: string, model: string, temperature: number, maxTokens: number): Promise<string> {
12        const genModel = this.client.getGenerativeModel({ model });
13
14        const result = await genModel.generateContent({
15            contents: [{ role: 'user', parts: [{ text: promptText }] }],
16            generationConfig: {
17                temperature,
18                maxOutputTokens: maxTokens,
19            },
20        });
21
22        const { response } = result;
23        return response.text();
24    }
25}

src/providers/index.ts

1export * from './openai.js';
2export * from './anthropic.js';
3export * from './google.js';
4
5export const getProvider = (model: string): 'openai' | 'anthropic' | 'google' => {
6    if (model.includes('claude-')) return 'anthropic';
7    if (model.startsWith('gemini-')) return 'google';
8    if (model.startsWith('gpt-')) return 'openai';
9    throw new Error(`Unknown model provider for model: ${model}`);
10};

src/providers/openai.ts

1import OpenAI from 'openai';
2import { LLMProvider } from '../types.js';
3
4export class OpenAIProvider implements LLMProvider {
5    private client: OpenAI;
6
7    constructor(apiKey: string) {
8        this.client = new OpenAI({ apiKey });
9    }
10
11    async call(promptText: string, model: string, temperature: number, maxTokens: number): Promise<string> {
12        const completion = await this.client.chat.completions.create({
13            messages: [{ role: 'user', content: promptText }],
14            model,
15            temperature,
16            max_tokens: maxTokens,
17        });
18        return completion.choices[0]?.message?.content || '';
19    }
20}

src/main.ts

1// main.ts
2
3import { Actor, log } from 'apify';
4import { OpenAIProvider, AnthropicProvider, GoogleProvider, getProvider } from './providers/index.js';
5import { Input, OutputItem, ValidatedInput } from './types.js';
6import { parseCustomPreprocessingFunction } from './utils.js';
7
8// Rate limits for OpenAI API lowest tier
9const RATE_LIMIT_PER_MINUTE = 500;
10const REQUEST_INTERVAL_MS = Math.ceil(60000 / RATE_LIMIT_PER_MINUTE); // Interval between requests in ms
11
12await Actor.init();
13
14// Helper function to get nested field value using dot notation
15function getNestedValue(obj: any, path: string): any {
16    return path.split('.').reduce((current, key) => current && current[key], obj);
17}
18
19// Helper function to check if a value is empty
20function isEmpty(value: any): boolean {
21    if (value === undefined || value === null) return true;
22    if (typeof value === 'string') return value.trim() === '';
23    if (Array.isArray(value)) return value.length === 0;
24    if (typeof value === 'object') return Object.keys(value).length === 0;
25    return false;
26}
27
28// Helper function to check if any placeholder field is empty
29function hasEmptyFields(promptStr: string, item: any): boolean {
30    const fieldMatches = promptStr.match(/\$\{([^}]+)\}/g) || [];
31    return fieldMatches.some((match) => {
32        const field = match.slice(2, -1).trim(); // Remove ${ and }
33        const value = getNestedValue(item, field);
34        return isEmpty(value);
35    });
36}
37
38// Helper function to replace field placeholders in prompt with actual values
39function replacePlaceholders(promptStr: string, item: any): string {
40    return promptStr.replace(/\$\{([^}]+)\}/g, (_match, fieldName: string) => {
41        const value = getNestedValue(item, fieldName.trim());
42        return value !== undefined ? String(value) : '';
43    });
44}
45
46async function validateInput(): Promise<ValidatedInput> {
47    const input = await Actor.getInput() as Input;
48    if (!input) {
49        throw new Error('No input provided. Please provide the necessary input parameters.');
50    }
51
52    const {
53        llmProviderApiKey,
54        prompt,
55        model,
56        temperature,
57        maxTokens,
58        skipItemIfEmpty,
59        multipleColumns = false,
60        testPrompt = false,
61        testItemsCount = 3,
62    } = input;
63
64    const inputDatasetId = input?.inputDatasetId || input?.payload?.resource?.defaultDatasetId;
65
66    if (!inputDatasetId) {
67        throw new Error('No inputDatasetId provided. Please provide the necessary input parameters.');
68    }
69
70    const preprocessingFunction = parseCustomPreprocessingFunction(input.preprocessingFunction);
71
72    return {
73        inputDatasetId,
74        llmProviderApiKey,
75        prompt,
76        model,
77        temperature,
78        maxTokens,
79        skipItemIfEmpty: skipItemIfEmpty ?? false,
80        multipleColumns,
81        testPrompt,
82        testItemsCount,
83        preprocessingFunction,
84    };
85}
86
87async function fetchDatasetItems(inputDatasetId: string, testPrompt: boolean, testItemsCount: number): Promise<OutputItem[]> {
88    try {
89        const dataset = await Actor.apifyClient.dataset(inputDatasetId).get();
90        if (!dataset) {
91            throw new Error(`Dataset with ID ${inputDatasetId} does not exist`);
92        }
93
94        const inputDataset = await Actor.openDataset<OutputItem>(inputDatasetId, { forceCloud: true });
95        const { items: fetchedItems } = await inputDataset.getData();
96
97        if (testPrompt) {
98            const itemCount = Math.min(testItemsCount, fetchedItems.length);
99            const items = fetchedItems.slice(0, itemCount);
100            log.info(`Test mode enabled - processing ${itemCount} items out of ${fetchedItems.length}`);
101            return items;
102        }
103
104        log.info(`Fetched ${fetchedItems.length} items from the input dataset.`);
105        return fetchedItems;
106    } catch (error) {
107        if (error instanceof Error) {
108            log.error(`Error accessing dataset: ${error.message}`);
109        } else {
110            log.error('Error accessing dataset: Unknown error occurred');
111        }
112        throw error;
113    }
114}
115
116async function processItems(
117    items: OutputItem[],
118    providers: Record<string, OpenAIProvider | AnthropicProvider | GoogleProvider>,
119    config: ValidatedInput,
120): Promise<void> {
121    const temperatureNum = parseFloat(config.temperature);
122    const { preprocessingFunction } = config;
123
124    for (let i = 0; i < items.length; i++) {
125        const item = preprocessingFunction(items[i]);
126
127        try {
128            if (config.skipItemIfEmpty && hasEmptyFields(config.prompt, item)) {
129                log.info(`Skipping item ${i + 1} due to empty fields`);
130                continue;
131            }
132
133            const finalPrompt = replacePlaceholders(buildFinalPrompt(config.prompt, config.multipleColumns), item);
134            log.info(`Processing item ${i + 1}/${items.length}`, { prompt: finalPrompt });
135
136            const provider = getProvider(config.model);
137            const llmresponse = await providers[provider].call(
138                finalPrompt,
139                config.model,
140                temperatureNum,
141                config.maxTokens,
142            );
143
144            log.info(`Item ${i + 1} response:`, { response: llmresponse });
145
146            await handleItemResponse(item, llmresponse, config.multipleColumns, {
147                provider,
148                model: config.model,
149                temperature: temperatureNum,
150                maxTokens: config.maxTokens,
151                providers,
152                finalPrompt,
153            });
154
155            await new Promise((resolve) => setTimeout(resolve, REQUEST_INTERVAL_MS));
156        } catch (error) {
157            if (error instanceof Error) {
158                log.error(`Error processing item ${i + 1}: ${error.message}`);
159            } else {
160                log.error(`Error processing item ${i + 1}: Unknown error occurred`);
161            }
162            throw error;
163        }
164    }
165}
166
167async function handleItemResponse(
168    item: any,
169    llmresponse: string,
170    multipleColumns: boolean,
171    config: {
172        provider: string;
173        model: string;
174        temperature: number;
175        maxTokens: number;
176        providers: Record<string, OpenAIProvider | AnthropicProvider | GoogleProvider>;
177        finalPrompt: string;
178    },
179): Promise<void> {
180    if (multipleColumns) {
181        let parsedData: any;
182        let attemptsLeft = 2;
183        let currentResponse = llmresponse;
184        let success = false;
185
186        while (attemptsLeft >= 0) {
187            try {
188                parsedData = JSON.parse(currentResponse);
189                success = true;
190                break;
191            } catch (err) {
192                if (attemptsLeft > 0) {
193                    log.warning(`Failed to parse JSON. Retrying...`);
194                    const retryPrompt = `${config.finalPrompt}\n\nThe last response was not valid JSON. Please return valid JSON this time.`;
195                    currentResponse = await config.providers[config.provider].call(
196                        retryPrompt,
197                        config.model,
198                        config.temperature,
199                        config.maxTokens,
200                    );
201                    attemptsLeft--;
202                } else {
203                    log.error(`Failed to parse JSON after multiple attempts. Using raw response as single column.`);
204                    break;
205                }
206            }
207        }
208
209        if (success && typeof parsedData === 'object' && parsedData !== null) {
210            const outputItem: Record<string, unknown> = { ...item };
211            for (const key of Object.keys(parsedData)) {
212                outputItem[key] = parsedData[key];
213            }
214            await Actor.pushData(outputItem);
215        } else {
216            const fallbackItem = { ...item, llmresponse: currentResponse };
217            await Actor.pushData(fallbackItem);
218        }
219    } else {
220        item.llmresponse = llmresponse;
221        await Actor.pushData(item);
222    }
223}
224
225function buildFinalPrompt(promptText: string, multipleColumns: boolean): string {
226    if (!multipleColumns) {
227        return promptText;
228    }
229
230    return `${promptText}
231
232Important: Return only a strict JSON object with the requested fields as keys. No extra text or explanations, no markdown, just JSON.`;
233}
234
235async function validateJsonFormat(testItem: any, config: {
236    providers: Record<string, OpenAIProvider | AnthropicProvider | GoogleProvider>;
237    model: string;
238    temperature: string;
239    maxTokens: number;
240    prompt: string;
241}): Promise<boolean> {
242    const provider = getProvider(config.model);
243    let finalPrompt = replacePlaceholders(buildFinalPrompt(config.prompt, true), testItem);
244
245    for (let attempt = 1; attempt <= 3; attempt++) {
246        try {
247            const testResponse = await config.providers[provider].call(
248                finalPrompt,
249                config.model,
250                parseFloat(config.temperature),
251                config.maxTokens,
252            );
253
254            // First check if we got an empty response
255            if (!testResponse) {
256                log.error('Empty response received from the API');
257                await Actor.fail('Empty response received from the API');
258                return false;
259            }
260
261            // Try parsing as JSON:
262            try {
263                JSON.parse(testResponse);
264                return true; // JSON parsed successfully
265            } catch (jsonError) {
266                if (attempt < 3) {
267                    log.warning(`JSON validation attempt ${attempt} failed. Retrying...`);
268                    log.debug('Response that failed JSON parsing:', { response: testResponse });
269                    finalPrompt = `${finalPrompt}\n\nThe last response was not valid JSON. Please return valid JSON this time.`;
270                    // Continue to next attempt
271                } else {
272                    // No attempts left
273                    log.error('JSON validation attempts exhausted. The prompt may not produce valid JSON.');
274                    log.debug('Final response that failed JSON parsing:', { response: testResponse });
275                    return false;
276                }
277            }
278        } catch (apiError: any) {
279            // Log the full error for debugging
280            log.error('API call failed:', {
281                error: apiError.message,
282                type: apiError.type,
283                code: apiError.code,
284                param: apiError.param,
285            });
286
287            // Rethrow API errors immediately instead of retrying
288            throw apiError;
289        }
290    }
291    return false; // Ensure we always return a boolean
292}
293
294async function run(): Promise<void> {
295    try {
296        const validatedInput = await validateInput();
297
298        // Log configuration details
299        const configDetails = {
300            datasetId: validatedInput.inputDatasetId,
301            model: validatedInput.model,
302            promptTemplate: validatedInput.prompt,
303            multipleColumns: validatedInput.multipleColumns,
304        };
305        log.info('Configuration details:', configDetails);
306
307        const items = await fetchDatasetItems(
308            validatedInput.inputDatasetId,
309            validatedInput.testPrompt,
310            validatedInput.testItemsCount,
311        );
312
313        const providers = {
314            openai: new OpenAIProvider(validatedInput.llmProviderApiKey),
315            anthropic: new AnthropicProvider(validatedInput.llmProviderApiKey),
316            google: new GoogleProvider(validatedInput.llmProviderApiKey),
317        };
318
319        if (items.length > 0 && validatedInput.multipleColumns) {
320            const firstItem = validatedInput.preprocessingFunction(items[0]);
321            const validationResult = await validateJsonFormat(firstItem, {
322                providers,
323                model: validatedInput.model,
324                temperature: validatedInput.temperature,
325                maxTokens: validatedInput.maxTokens,
326                prompt: validatedInput.prompt,
327            });
328
329            if (!validationResult) {
330                throw new Error('Failed to produce valid JSON after multiple attempts. Please adjust your prompt or disable multiple columns.');
331            }
332        }
333
334        await processItems(items, providers, validatedInput);
335
336        log.info('Actor finished successfully');
337        await Actor.exit();
338    } catch (error) {
339        if (error instanceof Error) {
340            log.error('Actor failed:', { error: error.message });
341            await Actor.fail(error.message);
342        } else {
343            log.error('Actor failed with unknown error');
344            await Actor.fail('Unknown error occurred');
345        }
346    }
347}
348
349await run();

src/types.ts

1export interface Input {
2    inputDatasetId: string;
3    defaultDatasetId: string;
4    llmProviderApiKey: string;
5    prompt: string;
6    model: string;
7    temperature: string;
8    maxTokens: number;
9    skipItemIfEmpty?: boolean;
10    multipleColumns?: boolean;
11    testPrompt?: boolean;
12    testItemsCount?: number;
13    payload: Payload | null;
14    preprocessingFunction?: string
15}
16
17export interface ValidatedInput {
18    inputDatasetId: string;
19    llmProviderApiKey: string;
20    prompt: string;
21    model: string;
22    temperature: string;
23    maxTokens: number;
24    skipItemIfEmpty: boolean;
25    multipleColumns: boolean;
26    testPrompt: boolean;
27    testItemsCount: number;
28    preprocessingFunction: CustomPreprocessingFunction;
29}
30
31export interface Payload {
32    resource: Resource;
33}
34
35export interface Resource {
36    defaultDatasetId: string;
37}
38
39export interface OutputItem extends Record<string, any> {
40    LLMResponse: string | null;
41}
42
43export interface LLMProvider {
44    call(promptText: string, model: string, temperature: number, maxTokens: number): Promise<string>;
45}
46
47export type CustomPreprocessingFunction = (item: unknown) => unknown

src/utils.ts

1import { log } from 'apify';
2import vm from 'vm';
3import { CustomPreprocessingFunction } from './types.js';
4
5export const parseCustomPreprocessingFunction = (funcString: string | undefined): CustomPreprocessingFunction => {
6    if (!funcString?.trim()) {
7        return (item) => item;
8    }
9    let func: (input: unknown) => unknown;
10    try {
11        func = vm.runInNewContext(funcString);
12    } catch (err) {
13        log.exception(err as Error, 'Cannot compile custom data function!');
14        throw err;
15    }
16
17    if (typeof func !== 'function') {
18        throw new Error('Custom preprocessing function is not a function!');
19    }
20
21    return (item: any) => {
22        try {
23            return func(item);
24        } catch (error) {
25            log.error(`Preprocessing function failed, returning the original item`, { error });
26            return item;
27        }
28    };
29};

.dockerignore

# configurations
.idea
.vscode

# crawlee and apify storage folders
apify_storage
crawlee_storage
storage

# installed files
node_modules

# git folder
.git

.editorconfig

root = true

[*]
indent_style = space
indent_size = 4
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
end_of_line = lf

.eslintrc

{
    "root": true,
    "env": {
        "browser": true,
        "es2020": true,
        "node": true
    },
    "extends": [
        "@apify/eslint-config-ts"
    ],
    "parserOptions": {
        "project": "./tsconfig.json",
        "ecmaVersion": 2020
    },
    "rules": {
        "@typescript-eslint/no-explicit-any": "off"
    },
    "ignorePatterns": [
        "node_modules",
        "dist",
        "**/*.d.ts"
    ]
}

.gitignore

# This file tells Git which files shouldn't be added to source control

.DS_Store
.idea
.vscode
dist
node_modules
apify_storage
storage
.env

# Added by Apify CLI
.venv

package.json

{
	"name": "LLMDatasetTest",
	"version": "0.0.1",
	"type": "module",
	"description": "This is a boilerplate of an Apify actor.",
	"engines": {
		"node": ">=18.0.0"
	},
	"dependencies": {
		"@anthropic-ai/sdk": "^0.32.1",
		"@google/generative-ai": "^0.21.0",
		"apify": "^3.2.6",
		"crawlee": "^3.11.5",
		"openai": "^4.76.0"
	},
	"devDependencies": {
		"@apify/eslint-config-ts": "^0.3.0",
		"@apify/tsconfig": "^0.1.0",
		"@typescript-eslint/eslint-plugin": "^7.18.0",
		"@typescript-eslint/parser": "^7.18.0",
		"eslint": "^8.50.0",
		"tsx": "^4.6.2",
		"typescript": "^5.3.3"
	},
	"scripts": {
		"start": "npm run start:dev",
		"start:prod": "node dist/main.js",
		"start:dev": "tsx src/main.ts",
		"build": "tsc",
		"lint": "eslint ./src --ext .ts",
		"lint:fix": "eslint ./src --ext .ts --fix",
		"test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
	},
	"author": "It's not you it's me",
	"license": "ISC"
}

tsconfig.json

{
    "extends": "@apify/tsconfig",
    "compilerOptions": {
        "module": "NodeNext",
        "moduleResolution": "NodeNext",
        "target": "ES2022",
        "outDir": "dist",
        "noUnusedLocals": false,
        "skipLibCheck": true,
        "lib": ["DOM"]
    },
    "include": [
        "./src/**/*"
    ]
}

Extract-any-webpage-content-for-llm

ai-developer/extract-any-webpage-content-for-llm

Fast and easy way to extract data from any webpage and are LLM friendly. The tool lets you easily extract content from any website. Ideal for researchers, marketers, and developers.

aideveloper

582

Output to Dataset

njoylab/apify-output-to-dataset

Merges outputs from multiple actors into a single dataset. Execute actors in series or parallel, combine data from datasets, key-value stores, webhooks, and export the final output in various formats.

njoylab

5.0

Markdown Table Generator

visita/markdown-table-generator

This Actor converts data from multiple sources into a clean, presentation-ready table. You can provide raw text, a direct URL to a file (like `.xlsx` or `.csv`), or Run ID of another Apify Actor, and this tool will automatically format it into Markdown, HTML, or Confluence Wiki markup.

Visita AI & Automation

🔥 Power Data Transformer

wiseek/power-data-transformer

🔥 Unlock your scraped data—clean, merge, split, deduplicate, filter, standardize, validate, enrich and sync—using built-in transformations and powerful SQL pipelines for ETL/ELT workflows. Seamlessly integrate processed datasets with automation platforms like n8n, Make.com, and Zapier.

wiseek

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

220

5.0

Sort Dataset Items

lukaskrivka/sort-dataset-items

Add this actor as a webhook to your scraper to sort the dataset by index field

Lukáš Křivka

Dataset Toolbox

cyberfly/dataset-toolbox

Perform common actions on datasets - merge, unify, validate, transform, order fields etc.

Vasek Codey Vlcek

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Lukáš Křivka

4.7K

5.0

MongoDB Import

drobnikj/mongodb-import

Import items from dataset into MongoDB

Jakub Drobník

Data Cleaning Actor

tonic_teak/data-cleaning-actor

This actor automatically cleans, analyzes, and summarizes spreadsheet data. It handles different file types (CSV, XLSX), fixes missing values, detects outliers, generates charts, computes correlations, and returns a cleaned dataset along with downloadable files and visual insights.