Robots.txt Generator

🤖 Robots.txt generator for SEO and crawler control

Robots.txt Generator creates deployable robots.txt files from presets, custom bot rules, sitemap URLs, and host directives. Use this robots txt generator when you need a clean file for SEO work, AI crawler blocking, staging privacy, WordPress, e-commerce, or repeatable multi-site setup.

The Actor follows the public robots.txt protocol format and returns both the raw text file and structured validation data. You can generate one file, batch multiple sites in a single run, save the raw files from the key-value store, and export the dataset as JSON, CSV, Excel, XML, RSS, or HTML.

⚙️ What this Actor does

• Generates ready-to-deploy robots.txt text.
• Starts from practical presets such as allow all, block all, SEO-friendly, WordPress SEO, e-commerce SEO, staging private, block AI crawlers, and AI training opt-out.
• Adds custom User-agent, Allow, Disallow, Crawl-delay, Sitemap, and Host lines.
• Supports multiple generated files in one run through the sites batch input.
• Saves each raw robots.txt file to the key-value store.
• Emits one dataset row per generated file with counts, warnings, validation details, file metadata, and a SHA-256 hash.
• Lets you append custom rules to a preset or override the preset rules entirely.

This Actor is a generator, not a crawler. It does not visit your website, deploy the file, audit live crawler behavior, or guarantee that every bot will obey the file.

📥 Input

Use the top-level fields for a single file, or use sites when you want separate files for multiple websites.

{
	"preset": "block-ai-crawlers",
	"sitemaps": ["https://example.com/sitemap.xml"],
	"host": "example.com",
	"rules": [
		{
			"userAgents": ["Googlebot"],
			"allow": ["/"],
			"disallow": ["/private/"],
			"comment": "Keep search access while hiding private paths"
		}
	],
	"presetConflictMode": "append",
	"validateRules": true,
	"includeTimestamp": true,
	"includeGeneratorComment": true
}

For batch generation, add items to sites. Each site can define its own label, preset, sitemap URLs, host directive, file name, and rule blocks.

📤 Output

Each dataset item represents one generated robots.txt file. Rows can include:

• robotsTxt - the generated file contents.
• preset - the preset used for that file.
• siteLabel and inputIndex - batch context when provided.
• fileName, contentType, byteSize, sha256, kvStoreKey, and kvStoreUrl - file export metadata.
• host and sitemaps - directives included in the file.
• ruleBlockCount, sitemapCount, and lineCount - quick file-size and structure counts.
• warnings, validation, and validationSummary - rule issues and suggestions when validation is enabled.
• ruleSummary - per-rule counts for user agents, allowed paths, blocked paths, and crawl delay.
• botListVersion and generatedAt - generation metadata.

{
	"success": true,
	"preset": "block-ai-crawlers",
	"fileName": "robots.txt",
	"kvStoreKey": "robots-txt/robots.txt",
	"host": "example.com",
	"sitemaps": ["https://example.com/sitemap.xml"],
	"ruleBlockCount": 2,
	"sitemapCount": 1,
	"lineCount": 18,
	"warnings": [],
	"validationSummary": {
		"errors": 0,
		"warnings": 0
	},
	"robotsTxt": "# Generated robots.txt..."
}

Open kvStoreUrl or fetch kvStoreKey from the run's key-value store when you want the raw .txt file.

💳 Pricing

This Actor uses pay-per-event pricing. You are charged once for each generated robots.txt file that is saved as an output row. There is no separate Actor-start charge in the local pricing artifact.

Use one generated file for a quick first run, then switch to the sites batch input when you are ready to generate files for multiple domains.

⚠️ Limits and caveats

• robots.txt is a voluntary crawler directive file. Well-behaved crawlers respect it, but it is not an access-control system.
• The Actor does not upload the file to your website. You still need to deploy it at the root path, such as https://example.com/robots.txt.
• The Actor does not crawl your website, verify live sitemap URLs, or check whether a deployed file is currently reachable.
• AI crawler presets are based on the maintained bot list included with this Actor at generation time.
• Validation catches common formatting and rule issues, but final SEO policy choices remain yours.

❓ FAQ

🧭 Can I use this as a robots.txt API?

Yes. Run the Actor through the Apify API, pass your preset and rule fields, then read the dataset row or key-value-store file from the finished run.

🧱 Can it block AI crawlers?

It can generate rule blocks for known AI crawlers and AI training bots. The output is still a robots.txt file, so enforcement depends on each crawler respecting the protocol.

🗺️ Can I include sitemap URLs?

Yes. Add one or more sitemap URLs and the Actor writes Sitemap: lines into the generated file.

🏢 Can I generate files for multiple sites?

Yes. Use the sites array to create one output file per site with separate labels, presets, sitemap URLs, host directives, custom rules, and file names.

📝 Changelog

• 0.1: Initial release.

🆘 Support

For issues, questions, or feature requests, file a ticket and I'll fix or implement it in less than 24h 🫡

🔗 Other actors

• Website URL Crawler ↗ - Build rendered website link maps for SEO audits and migrations.
• Font Detector ↗ - Audit fonts, font files, and typography metadata from public pages.
• SEMrush Free Website Stats Scraper ↗ - Collect public SEMrush website metrics for domains and URLs.
• Ahrefs Free Website Stats Scraper ↗ - Collect public Ahrefs website stats for SEO research.
• Email MX Verifier ↗ - Check email syntax, MX records, disposable domains, and delivery risk.

Made with ❤️ by Maxime Dupré