Website Preflight Intelligence – HTTP, SEO, Tech & Security

Fast, low-cost website preflight intelligence for AI agents, crawlers, SEO workflows, QA systems, lead enrichment, and automation.

Give the Actor one or many public website URLs or domains. It performs lightweight HTTP checks and returns one structured report per website.

What it checks

• HTTP status and redirect chain
• Final URL and HTTPS usage
• Response time
• Content type and response size
• Page title and meta description
• Canonical URL, H1, language, and robots meta
• robots.txt availability
• Sitemap discovery and lightweight URL-count estimates
• Common security headers
• Server and X-Powered-By headers
• Website technology signals
• Contact page and About page discovery
• Public emails and phone links visible on the homepage
• Social profile links
• JSON-LD schema types

Technology signals

The Actor can identify common website and infrastructure hints such as:

• WordPress
• Shopify
• Webflow
• Wix
• Squarespace
• Next.js
• Nuxt
• HubSpot
• Cloudflare
• Vercel
• Netlify
• Google Tag Manager
• Google Analytics

Technology detection is heuristic and should be treated as a signal, not a guarantee.

Common use cases

• Website preflight checks
• HTTP status checker
• Redirect checker
• Website metadata extraction
• SEO metadata checks
• robots.txt checker
• Sitemap checker
• Security-header inspection
• Website technology detection
• Website health intelligence
• AI-agent preprocessing
• Lead-generation and CRM enrichment
• Deciding whether a site needs deeper crawling or browser rendering

Why use it

This Actor is designed as a cheap machine-to-machine first pass before a more expensive browser, crawler, enrichment service, or AI agent.

It uses direct HTTP requests with bounded response reads and avoids heavyweight browser rendering, residential proxies, external LLM calls, and paid third-party APIs.

Example input

{
  "websites": [
    "https://apify.com",
    "stripe.com"
  ],
  "requestTimeoutSecs": 12,
  "maxConcurrency": 20,
  "checkRobotsTxt": true,
  "checkSitemap": true
}

Example output

{
  "requestedUrl": "https://example.com/",
  "domain": "example.com",
  "finalUrl": "https://example.com/",
  "status": "success",
  "httpStatus": 200,
  "responseTimeMs": 184,
  "https": true,
  "title": "Example Company",
  "metaDescription": "Example website",
  "canonicalUrl": "https://example.com/",
  "contactPage": "https://example.com/contact",
  "aboutPage": "https://example.com/about",
  "emails": ["hello@example.com"],
  "technologies": ["Cloudflare"],
  "robotsTxt": {
    "exists": true,
    "status": 200
  },
  "sitemap": {
    "exists": true,
    "status": 200
  },
  "error": null
}

Security

The Actor accepts public HTTP and HTTPS targets only. It resolves hostnames and rejects loopback, private, link-local, and reserved network addresses before requests and before following redirects.

Limitations

• JavaScript-rendered content is not executed.
• Some websites block automated HTTP requests or vary responses by region.
• Technology detection is heuristic.
• Missing contact details mean they were not visible in the inspected static HTML.
• Sitemap URL counts are lightweight estimates, not a full crawl.

Output

One structured website-intelligence report is written to the default dataset for every submitted website, including blocked or failed targets. The output is easy to consume from APIs, MCP tools, n8n, Make, Zapier, crawlers, and custom AI-agent pipelines.