🔎 Ads.txt Scraper - Publisher Ad Seller Data

Scrape ads.txt files and extract authorized ad sellers, publisher IDs, relationship types and certification authority IDs. Export to JSON/CSV/Excel, run on a schedule, call via API, or connect to Make, Zapier or n8n.

What is the Ads.txt Scraper?

The Ads.txt Scraper turns public website standard files into clean structured data. These small files are easy to miss manually, but they are useful for audits, lead qualification, compliance checks, SEO operations, advertising research, and automated monitoring. Instead of opening every domain in a browser, you can provide a list of domains or direct file URLs and receive normalized rows in an Apify dataset.

The actor is intentionally lightweight. It uses direct HTTP, follows redirects, and parses the public text file without launching a browser. That keeps runs fast, inexpensive, and suitable for scheduled checks across many domains.

What data does it extract?

• Final source URL and input URL
• Domain and HTTP status code
• Content type
• Parsed fields relevant to this file type
• Raw line or raw text excerpts for auditability
• URLs and email addresses where present
• Structured result rows ready for export
• Source marker for downstream pipelines

Input

Field	Type	Description
targets	array	Domains or direct URLs to check. Domains are automatically converted to the standard file path.
maxResults	integer	Maximum rows to return per target. Useful for large files such as ads.txt or llms.txt.

Example input

{
  "targets": [
    "https://www.cnn.com/ads.txt"
  ],
  "maxResults": 25
}

Output

{"exchange_domain": "google.com", "publisher_id": "pub-123456789", "relationship": "DIRECT", "certification_authority_id": "f08c47fec0942fa0", "source": "ads"}

Use cases

• SEO audits: collect machine-readable public metadata from client or competitor domains.
• Lead generation: qualify companies by their public operational, advertising, or contact signals.
• Compliance monitoring: schedule checks and detect missing, changed, or expired files.
• Advertising research: extract authorized seller relationships from publisher domains.
• AI visibility work: monitor llms.txt files and documentation links as AI search practices evolve.
• Data enrichment: add public standards metadata to domain lists, CRM accounts, and prospect databases.

Practical tips

For domain lists, pass bare domains such as example.com and the actor will request the standard path automatically. For custom locations, pass the full URL. Keep maxResults small for quick health checks and larger for full exports. When monitoring a portfolio, schedule the actor daily or weekly and compare rows by domain, source_url, and file-specific identifiers.

The actor returns raw excerpts because public standards files are not always perfectly formatted. Keeping the original line or text makes it easier to debug unusual publishers, malformed entries, or custom fields that are still valuable to a human analyst.

Data quality notes

Public website-standard files are simple on paper, but real-world files vary a lot. Some domains add comments, blank lines, custom fields, redirects, casing differences, or extra whitespace. This actor keeps both normalized fields and the original source values so you can trust the output in automated workflows while still having enough context for manual audits.

For files that contain multiple records, such as advertising seller lists or documentation indexes, the actor emits one row per useful record. For files that describe one website-level policy or team file, it emits one normalized row per target. That keeps the dataset natural to join with domain lists, CRM exports, spreadsheet audits, and BI tools.

Workflow ideas

Start with a list of customer domains, competitor domains, publishers, SaaS companies, app developers, or prospect websites. Run the actor once to create a baseline, then schedule it weekly to detect changes. You can send the output to a webhook, compare it with the previous dataset, and alert your team when a file appears, disappears, expires, changes seller relationships, or adds new contact details.

For agencies, this can become a repeatable audit product: upload a domain list, export the results, and include missing or malformed standards files in the client report. For lead-generation teams, the same data can help prioritize companies that show active advertising operations, security maturity, AI documentation readiness, or public site-maintenance signals.

Reliability and performance

This actor does not use a browser, paid unblocker, or residential proxy. It is designed for public text files that are meant to be fetched by automated systems. If a target returns no file, the actor logs the failure and continues with the next target. That makes it safe for larger domain lists where some websites are expected to be missing the standard file.

Because the parser is conservative, it avoids pretending malformed files are perfect. Rows include HTTP metadata and raw source values so your workflow can decide whether to accept, clean, or review them.

FAQ

Does it require an account? No. It reads public files from the provided websites.

Can I pass domains instead of full URLs? Yes. Bare domains are converted to the correct standard path.

Can I run it on many websites? Yes. Add multiple targets and use Apify scheduling for recurring checks.

Does it crawl the whole website? No. It fetches only the standard file URL for each target.

Can I export to Excel? Yes. Apify datasets export to JSON, CSV, Excel, XML, and RSS.

Can I connect it to automation tools? Yes. Use Apify API, webhooks, Make, Zapier, n8n, or dataset exports.

What happens when a file is missing? The actor logs the target and continues with the next one.

Is it legal? The actor reads public website files. You are responsible for respecting website terms, rate limits, privacy rules, and applicable laws.

How is billing calculated? Pay-per-event billing charges per result pushed to the dataset.

Why include raw text? Raw excerpts make audits easier and preserve fields that may not be part of the common format yet.

You might also like

• Website Contact Extractor
• Meta Tags Extractor
• HTTP Headers Checker
• Robots Sitemap Analyzer
• Schema Markup Extractor

Keywords: ads.txt scraper, authorized digital sellers, publisher ad sellers, programmatic ads data, ad verification scraper, web standards scraper, domain audit tool, website metadata extractor, Apify scraper, SEO automation, lead generation data, compliance monitoring, website intelligence