Website Intelligence

A reusable, structured website analysis Actor — the foundation other actors build on. Give it one or more URLs and it returns the raw structured intelligence for each site: HTTP status, title, meta description, headings, links, images, CMS, technologies, analytics, advertising, social links, contact info, structured data, robots.txt/sitemap.xml summaries and basic SEO signals. No scoring, no opinion — just what is publicly observable on the page.

What does this Actor do?

For each target URL it fetches the homepage (and, optionally, robots.txt, sitemap.xml, and a few contact/about pages) and extracts a single structured report:

• Identity & content — final URL, HTTP status, HTTPS, title, meta description, headings, links, images.
• Stack — CMS, technologies, analytics platforms, advertising pixels, e‑commerce and booking signals.
• Contact & social — emails, phones, contact-form presence, and social profile links.
• Discoverability — structured data (schema.org types), robots.txt and sitemap.xml summaries, and SEO signals (title length, H1 count, viewport, canonical, Open Graph, image alt coverage).

It produces one record per URL.

Why use it?

• A clean, consistent structured snapshot of any website that you can feed into lead scoring, competitive analysis, tech-stack research, or your own downstream Actors.
• Because it only reports observable facts, the output is auditable and safe to build on.

How to use it

1. Provide a single url, a list of urls, or Apify startUrls.
2. Toggle checkRobots, checkSitemap and crawlContactPages as needed (all default on).
3. Run the Actor and download the dataset as JSON, CSV or Excel.

Input

Field	Type	Description
url	string	A single website to analyze.
urls	array	A list of websites to analyze (one record per URL).
startUrls	array	Apify request-list sources ({ "url": "…" }).
checkRobots	boolean	Fetch & summarise robots.txt (default true).
checkSitemap	boolean	Fetch & summarise sitemap.xml (default true).
crawlContactPages	boolean	Crawl a few contact/about pages for more contact info (default true).
maxPages	integer	Max extra contact/about pages per site (default 2).
proxyConfiguration	object	Proxy settings.
cookies	string	Optional Cookie header to replay on every request.

At least one of url / urls / startUrls is required.

Input example

{
  "url": "https://example.com",
  "checkRobots": true,
  "checkSitemap": true,
  "crawlContactPages": true
}

Output

One structured record per URL. Keys include: url, status, reachable, https, title, metaDescription, headings, links, images, cms, technologies, analytics, advertising, ecommerce, booking, contact (emails, phones, hasContactForm, formCount), socialLinks, structuredData, seo, robots, sitemap, domain.

Output example

{
  "url": "https://acme.example",
  "status": 200,
  "reachable": true,
  "https": true,
  "title": "Acme Widgets Co.",
  "cms": ["WordPress"],
  "technologies": ["WordPress"],
  "analytics": { "Google Analytics": true },
  "socialLinks": { "linkedin": "https://www.linkedin.com/company/acme-widgets" },
  "contact": { "emails": ["sales@acme.example"], "phones": [], "hasContactForm": false, "formCount": 0 },
  "structuredData": ["Organization"],
  "robots": { "exists": true, "hasSitemap": true, "disallowCount": 1 },
  "sitemap": { "exists": true, "urlCount": 2 },
  "seo": { "titleLength": 16, "h1Count": 1, "hasViewport": true, "hasCanonical": false }
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Sources & configuration

• The target websites themselves — only publicly served HTML, robots.txt and sitemap.xml are fetched. No authentication is bypassed.
• Environment variables — none required. The Apify Proxy password is injected by the platform.