AI Training Data Generator

An Apify Actor that converts web pages, documentation, blog posts, and other web content into structured AI training datasets. Supports multiple output formats suitable for fine-tuning LLMs.

Features

• Convert any web page into structured training data
• Three output formats:
  • Q&A Pairs - Headings become questions, content becomes answers. Ideal for training QA models.
  • Instruction Following - Each section becomes an instruction with expected completion. Perfect for instruction-tuning datasets.
  • Raw Text - Cleaned, structured text without instruction framing. Great for continued pre-training.
• Smart content extraction - Automatically removes navigation, headers, footers, sidebars, and other non-content elements.
• Metadata enrichment - Optionally includes source URL, section heading, and extraction method.
• Deduplication - Automatically removes duplicate examples by content hash.
• Graceful abort - Handles platform shutdown signals cleanly.

Input

Field	Type	Required	Description
sourceUrls	array[string]	✅	URLs of web pages/documentation to convert
outputFormat	enum	❌	qa_pairs, instruction_following, or raw_text (default: qa_pairs)
maxExamples	integer	❌	Maximum examples per URL (default: 100)
includeMetadata	boolean	❌	Include source metadata in output (default: true)
webhookUrl	string	❌	Optional webhook to call on completion

Example input

{
  "sourceUrls": [
    "https://docs.example.com/getting-started",
    "https://docs.example.com/advanced-usage"
  ],
  "outputFormat": "qa_pairs",
  "maxExamples": 50,
  "includeMetadata": true
}

Output

Each dataset item has the following structure:

Field	Description
exampleId	Unique content-hash identifier
exampleFormat	The output format used
instruction	Question, instruction, or empty (for raw_text)
input	Additional input context (currently empty, reserved for future use)
output	The answer, completion, or structured text
source	Source URL
generatedAt	ISO timestamp of generation

When includeMetadata is true, each item also includes a _metadata field with sourceUrl, sectionHeading, and extractionMethod.

Use Cases

• Fine-tuning LLMs - Generate instruction-following datasets from documentation
• RAG training - Create Q&A pairs from knowledge base articles
• Domain adaptation - Extract clean text from domain-specific websites
• Synthetic data generation - Convert technical documentation into training examples
• Dataset augmentation - Add web-sourced examples to existing training pipelines

Pricing

Pay-per-event: $0.02 USD per run. Covers up to 100 URLs per run. Each URL generates up to maxExamples training examples.