🌐 Website Metadata Extractor (Sitemaps, Social Links, Robots.txt & AI Industry Classifier)

A high-performance, enterprise-grade Apify Actor that extracts comprehensive website metadata, SEO tags, robots.txt rules, sitemap files, verified social media profiles, Open Graph & Twitter Cards, technology stacks, JSON-LD schemas, content headings, direct contact details, and AI-powered business industry classification from any list of domains or URLs.

Built for SEO professionals, data engineers, marketing agencies, growth hackers, and competitive intelligence teams, it delivers deep website audits in milliseconds with zero browser overhead at a fraction of the cost of headless scrapers.

---

⚡ What’s New in Version 2.0?

• 🤖 AI Business & Company Intelligence (natureOfBusiness): Integrated with Groq’s high-speed openai/gpt-oss-120b model to accurately extract complete company intelligence: 100+ standard industry categories, 1-sentence executive summary, B2B vs. B2C model, business & revenue model, ideal customer profile (target audience), core offerings & products, estimated company stage/scale, and compliance/trust signals (SOC2, GDPR, HIPAA, etc.).
• 💳 Transparent Pay-Per-Event (PPE) Monetization: Pay only for successful domains. Unreachable, invalid, or dead domains cost $0.00.
  • Standard Extraction: $1.50 per 1,000 domains ($1.00 domain analyzed + $0.50 dataset result).
  • With AI Classification Enabled: $4.50 per 1,000 domains ($1.00 domain analyzed + $0.50 dataset result + $3.00 AI intelligence).
• 📱 Verified Social Links Extraction: Multi-source detection covering 15+ major platforms (Facebook, Twitter/X, LinkedIn, Instagram, YouTube, TikTok, Pinterest, Reddit, GitHub, Discord, Telegram, WhatsApp, Snapchat, Threads, Medium) with intelligent noise and tracking parameter filtering.
• 🤖 Structured Robots.txt & Sitemap Engine: Intelligently parses user-agent rules (Allow, Disallow, Crawl-Delay) and discovers XML sitemaps without hanging on bot protections or WAFs.
• 🧱 30+ Modern Technology Fingerprints: Detects CMS platforms (WordPress, Shopify, Wix, Squarespace, Webflow), modern frameworks (Next.js, React, Vue, Nuxt, Angular, Astro), analytics (GA4, GTM, Clarity, Hotjar), pixels, and CDNs (Cloudflare, Fastly).
• 🏢 Company Logo & Legal Name: Parses Schema.org Organization and Corporation schemas to extract high-res logos and official legal entity names.
• ⚡ Ultra-Fast AI Processing: High-speed company intelligence inference with intelligent rate optimization and high throughput.
• 🌐 Universal URL Flexibility: Accepts bare domains (apify.com), subdomain prefixes (www.apify.com), or full URLs (https://...). Output provides both original and normalized URLs.

---

🚀 Key Features & Extracted Data

Category	Extracted Data Fields
🤖 AI Nature of Business	Industry Classification (100+ standard sectors), 1-Sentence Company Summary, B2B vs. B2C Model, Business Model, Target Audience (ICP), Core Offerings, Company Stage, Compliance Signals
🏢 Company & Branding	Official Legal Name, Company Logo URL, Favicon, Apple Touch Icon
Core SEO Tags	Page Title, Meta Description, Canonical URL, Charset, Language, Theme Color
Robots & Sitemaps	Full robots.txt rules parsed by User-Agent (Allow / Disallow / Crawl-Delay), Validated Sitemap XML URLs, RSS / Atom Feeds
15+ Social Profiles	Facebook, Twitter / X, LinkedIn, Instagram, YouTube, TikTok, Pinterest, Reddit, GitHub, Discord, Telegram, WhatsApp, Snapchat, Threads, Medium
Contact Information	Direct email addresses (mailto:) and phone numbers (tel:) (optional)
Open Graph (OG)	og:title, og:description, og:image, og:url, og:type, og:site_name, og:locale, and all custom OG attributes
Twitter Cards	twitter:card, twitter:site, twitter:creator, twitter:title, twitter:description, twitter:image
Structured Data	Full Schema.org JSON-LD objects (Organization, WebSite, Product, Article, FAQPage, LocalBusiness, etc.)
Headings & Hierarchy	Primary H1, All H1 tags, All H2 tags, All H3 tags
Content Metrics	Total Word Count and Total Character Count of body text
Site Verification	Google Site Verification, Bing / Microsoft, Yandex, Baidu, Pinterest, and Facebook Domain Verification
International SEO	Hreflang alternate language and regional URL mapping
Technology Stack	CMS, E-Commerce platforms, JS frameworks, Analytics tools, Pixels, and CDNs

---

💰 Transparent Pay-Per-Event Pricing

This Actor utilizes Apify’s official Pay-Per-Event (PPE) pricing model. You only pay for what you extract, with zero monthly rental fees.

Pricing Events & Tiers

Pricing Event	Event Code	Free Plan (No discount)	Starter Plan (Bronze discount)	Scale Plan (Silver discount)	Business Plan (Gold discount)
Domain Analyzed & Extracted	domain-processed	$1.00 / 1,000	$0.90 / 1,000	$0.80 / 1,000	$0.70 / 1,000
Result Pushed	apify-default-dataset-item	$0.50 / 1,000	$0.50 / 1,000	$0.50 / 1,000	$0.50 / 1,000
AI Business Classification (Add-On)	ai-nature-of-business	$3.00 / 1,000	$2.80 / 1,000	$2.70 / 1,000	$2.60 / 1,000
Actor Start (Infrequent)	apify-actor-start	$0.005	$0.005	$0.005	$0.005
Platform Usage (Compute Units)	—	FREE (Included)	FREE (Included)	FREE (Included)	FREE (Included)

Effective Combined Totals:

• Standard Metadata Extraction (AI Off): $1.50 / 1,000 (Starter: $1.40 | Scale: $1.30 | Business: $1.20)
• AI-Enriched Extraction (AI On): $4.50 / 1,000 (Starter: $4.20 | Scale: $4.00 | Business: $3.80)

🛡️ Zero-Cost Guarantee on Failed Domains: If a website is down, blocked, or has a dead DNS record, you are charged $0.00 for that domain. You only pay for successfully extracted results.

---

💵 Cost Calculation Examples

Option A: Standard Metadata Extraction (AI Disabled — $1.50 / 1,000 domains)

• 1,000 Domains: $0.005 (start) + (1 × $1.50) = $1.50 (Starter: $1.40 | Scale: $1.30 | Business: $1.20)
• 5,000 Domains: $0.005 + (5 × $1.50) = $7.50 (Starter: $7.00 | Scale: $6.50 | Business: $6.00)
• 10,000 Domains: $0.005 + (10 × $1.50) = $15.00 (Starter: $14.00 | Scale: $13.00 | Business: $12.00)
• 50,000 Domains: $0.005 + (50 × $1.50) = $75.00 (Starter: $70.00 | Scale: $65.00 | Business: $60.00)

Option B: AI-Enriched Extraction with Nature of Business ($4.50 / 1,000 domains)

• 1,000 Domains: $0.005 (start) + (1 × $4.50) = $4.50 (Starter: $4.20 | Scale: $4.00 | Business: $3.80)
• 5,000 Domains: $0.005 + (5 × $4.50) = $22.50 (Starter: $21.00 | Scale: $20.00 | Business: $19.00)
• 10,000 Domains: $0.005 + (10 × $4.50) = $45.00 (Starter: $42.00 | Scale: $40.00 | Business: $38.00)
• 50,000 Domains: $0.005 + (50 × $4.50) = $225.00 (Starter: $210.00 | Scale: $200.00 | Business: $190.00)

---

🤖 AI Company Intelligence & Scalability

The AI classification is powered by Groq’s ultra-low latency inference engine using the high-performance openai/gpt-oss-120b model:

High-Throughput & Enterprise Scale

• Ultra-Fast Inference: Delivers complete company profiling in milliseconds per domain.
• Proprietary Smart Context Distillation: Intelligent semantic pre-filtering removes website noise, boilerplate navigation, and cookie banners to focus exclusively on core business signals.
• Massive Concurrency: Capable of analyzing up to 1,000 domains per minute smoothly without rate bottlenecks.
• Built-in Resilience: Enterprise-grade automatic retry mechanisms with dynamic backoff guarantee uninterrupted execution even under large batch processing loads.

Configuring Your Groq API Key

You can provide your Groq API key in two ways:

• Option 1 (Recommended for Automation): In the Apify Console under Actor Settings → Environment Variables, add GROQ_API_KEY = your_key_here.
• Option 2: Provide it directly in the Actor input field groqApiKey (stored securely as a secret).

---

⚙️ Input Parameters

Parameter	Type	Default	Description
startUrls	Array	["https://apify.com"]	(Required) List of website URLs or domains. Accepts domain.com, www.domain.com, http://, or https://.
enableAiClassification	Boolean	false	AI Nature of Business: Enables industry categorization, 1-sentence summary, and B2B/B2C detection using Groq AI.
groqApiKey	String	""	Optional: Your Groq API key. If omitted, uses the GROQ_API_KEY Apify environment variable.
customIndustries	Array	[]	Optional: Custom list of industries for AI classification. If empty, the default 100+ global industries list is used.
disableDomainAnalysis	Boolean	false	Fast Mode: When true, skips fetching robots.txt and sitemap.xml for maximum crawl speed.
maxConcurrency	Integer	20	Maximum parallel domain requests (1 to 50). Recommended 20 for balanced speed and stability.
extractMetaTags	Boolean	true	Extract page title, meta description, Open Graph, Twitter Cards, and verification tags.
extractRobotsTxt	Boolean	true	Fetch and parse robots.txt rules by user-agent (Allow / Disallow / Crawl-Delay).
extractSitemaps	Boolean	true	Discover and validate sitemap XML files declared in robots.txt or at standard paths.
extractSocialLinks	Boolean	true	Extract verified social media profiles across 15+ networks.
extractTechnologies	Boolean	true	Detect CMS, JavaScript frameworks, e-commerce platforms, analytics, and CDNs.
extractHeadings	Boolean	true	Extract primary H1, all H1s, all H2s, and all H3 content headings.
extractJsonLd	Boolean	true	Extract and flatten Schema.org structured JSON-LD objects.
extractHreflang	Boolean	true	Extract international alternate language links (hreflang).
extractContentMetrics	Boolean	true	Count body text words and characters.
includeContactInfo	Boolean	false	Extract verified direct email addresses (mailto:) and phone numbers (tel:).
socialPlatforms	Array	(All 15)	Specific social platforms to target (e.g. ["linkedin", "twitter", "facebook"]).

---

🧩 Input Configurations

1. Full-Featured Enterprise Audit + AI Nature of Business

{
  "startUrls": [
    "apify.com",
    "https://shopify.com",
    "github.com",
    "https://stripe.com"
  ],
  "enableAiClassification": true,
  "disableDomainAnalysis": false,
  "maxConcurrency": 20,
  "extractMetaTags": true,
  "extractRobotsTxt": true,
  "extractSitemaps": true,
  "extractSocialLinks": true,
  "extractTechnologies": true,
  "extractHeadings": true,
  "extractJsonLd": true,
  "extractHreflang": true,
  "extractContentMetrics": true,
  "includeContactInfo": true,
  "socialPlatforms": [
    "facebook",
    "twitter",
    "linkedin",
    "instagram",
    "youtube",
    "tiktok",
    "pinterest",
    "reddit",
    "github",
    "discord",
    "telegram",
    "whatsapp",
    "snapchat",
    "threads",
    "medium"
  ]
}

2. High-Speed On-Page SEO Mode (Fast Mode)

Skips external robots.txt and sitemap.xml network queries for ultra-fast on-page processing:

{
  "startUrls": [
    "airbnb.com",
    "booking.com",
    "expedia.com"
  ],
  "disableDomainAnalysis": true,
  "extractRobotsTxt": false,
  "extractSitemaps": false,
  "maxConcurrency": 30
}

3. B2B Lead Enrichment (Social Profiles + Contact Details + AI)

Optimized for lead qualification with company social links, contact info, and B2B classification:

{
  "startUrls": [
    "https://hubspot.com",
    "https://salesforce.com",
    "https://notion.so"
  ],
  "enableAiClassification": true,
  "extractSocialLinks": true,
  "includeContactInfo": true,
  "socialPlatforms": [
    "linkedin",
    "twitter",
    "facebook",
    "youtube",
    "github"
  ],
  "extractMetaTags": true,
  "extractTechnologies": true,
  "disableDomainAnalysis": true
}

4. Simple Minimalist Input

{
  "startUrls": [
    "apify.com",
    "craigslist.org",
    "wikipedia.org"
  ]
}

---

📊 Output Example

Below is a complete output record showcasing the rich structured data extracted:

{
  "url": "https://www.shopify.com/",
  "inputUrl": "shopify.com",
  "normalizedUrl": "https://shopify.com",
  "domain": "shopify.com",
  "brandName": "shopify",
  "statusCode": 200,
  "responseTimeMs": 348,
  "title": "Shopify: The All-in-One Commerce Platform for Businesses",
  "description": "Try Shopify free and start a business or grow an existing one. Get more than ecommerce software with tools to manage every part of your business.",
  "canonical": "https://www.shopify.com/",
  "favicon": "https://cdn.shopify.com/shopifycloud/web/assets/v1/favicon-default-6cbad9de243dbae3.ico",
  "appleTouchIcon": "https://cdn.shopify.com/b/shopify-brochure2-assets/c97c60ca19c64a8b5378d9f9e971f7bd.png",
  "logo": "https://cdn.shopify.com/b/shopify-brochure2-assets/logo.png",
  "legalName": "Shopify Inc.",
  "natureOfBusiness": {
    "industry": "Software as a Service (SaaS)",
    "summary": "Shopify provides an all-in-one cloud-based commerce platform and point-of-sale solutions for merchants worldwide.",
    "isB2B": true,
    "businessModel": "B2B SaaS / Subscription Commerce Platform",
    "targetAudience": [
      "E-commerce entrepreneurs and DTC brands",
      "Retail merchants migrating online",
      "Enterprise retailers needing headless checkout"
    ],
    "coreOfferings": [
      "Online store builder and hosted checkout",
      "Shopify POS hardware and software",
      "Shopify Payments and Merchant Solutions",
      "Enterprise headless commerce (Shopify Plus)"
    ],
    "estimatedStage": "Established Public Enterprise",
    "complianceSignals": [
      "PCI-DSS Level 1",
      "SOC2 Type II",
      "GDPR",
      "CCPA"
    ]
  },
  "themeColor": "#008060",
  "language": "en",
  "charset": "utf-8",
  "robotsTxt": {
    "userAgents": {
      "*": {
        "allow": [
          "/",
          "/pricing"
        ],
        "disallow": [
          "/admin",
          "/cart",
          "/checkout",
          "/checkouts/",
          "/orders",
          "/account"
        ],
        "crawlDelay": 1
      },
      "Googlebot": {
        "allow": [
          "/",
          "/products/"
        ],
        "disallow": [
          "/admin",
          "/cart"
        ]
      }
    }
  },
  "sitemapFileUrls": [
    "https://www.shopify.com/sitemap.xml",
    "https://www.shopify.com/sitemaps_list.xml"
  ],
  "feeds": [
    "https://www.shopify.com/blog.atom"
  ],
  "socialLinks": {
    "facebook": "https://facebook.com/shopify",
    "twitter": "https://x.com/shopify",
    "linkedin": "https://linkedin.com/company/shopify",
    "instagram": "https://instagram.com/shopify",
    "youtube": "https://youtube.com/shopify",
    "tiktok": "https://tiktok.com/@shopify",
    "pinterest": "https://pinterest.com/shopify",
    "reddit": "https://reddit.com/r/shopify",
    "github": "https://github.com/shopify",
    "discord": "https://discord.com/invite/shopify",
    "telegram": "https://t.me/shopifyofficial",
    "whatsapp": "https://wa.me/18005550199",
    "snapchat": "https://snapchat.com/add/shopify",
    "threads": "https://threads.net/@shopify",
    "medium": "https://medium.com/@shopify"
  },
  "contactInfo": {
    "emails": [
      "support@shopify.com",
      "press@shopify.com"
    ],
    "phones": [
      "+1-888-746-7439",
      "+1-855-837-7773"
    ]
  },
  "metaTags": {
    "viewport": "width=device-width, initial-scale=1",
    "robots": "index, follow",
    "author": "Shopify Inc.",
    "publisher": "https://www.shopify.com",
    "format-detection": "telephone=no"
  },
  "openGraph": {
    "title": "Shopify: The All-in-One Commerce Platform for Businesses",
    "description": "Try Shopify free and start a business or grow an existing one. Get tools to manage every part of your business.",
    "image": "https://cdn.shopify.com/b/shopify-brochure2-assets/ad99387e77223a56caca1c7b18e31970.png",
    "url": "https://www.shopify.com/",
    "type": "website",
    "siteName": "Shopify",
    "locale": "en_US"
  },
  "twitterCard": {
    "card": "summary_large_image",
    "site": "@Shopify",
    "creator": "@Shopify",
    "title": "Shopify: The All-in-One Commerce Platform",
    "description": "Start, run, and grow your business with Shopify ecommerce software.",
    "image": "https://cdn.shopify.com/b/shopify-brochure2-assets/ad99387e77223a56caca1c7b18e31970.png"
  },
  "securityVerification": {
    "googleSiteVerification": "g_verification_token_sample_8492049102",
    "bingSiteVerification": "BING_VERIFICATION_TOKEN_ABC123",
    "facebookDomainVerification": "fb_domain_verify_xyz789",
    "pinterestVerification": "pinterest_token_456"
  },
  "detectedTechnologies": [
    "Shopify",
    "React",
    "Next.js",
    "Ruby on Rails",
    "Google Analytics 4",
    "Google Tag Manager",
    "Cloudflare"
  ],
  "h1": "Be the next big thing",
  "allH1s": [
    "Be the next big thing"
  ],
  "allH2s": [
    "The one commerce platform behind it all",
    "There’s no better place for you to build",
    "Start selling in no time",
    "Move fast, grow faster"
  ],
  "allH3s": [
    "Customizable themes",
    "Built-in payment processing",
    "World-class checkout experience"
  ],
  "wordCount": 1420,
  "characterCount": 9850,
  "hreflangLinks": {
    "en": "https://www.shopify.com/",
    "en-gb": "https://www.shopify.com/uk",
    "de": "https://www.shopify.com/de",
    "fr": "https://www.shopify.com/fr",
    "es": "https://www.shopify.com/es",
    "ja": "https://www.shopify.com/jp"
  },
  "jsonLd": [
    {
      "@context": "https://schema.org/",
      "@type": "Corporation",
      "name": "Shopify",
      "url": "https://www.shopify.com/",
      "logo": "https://cdn.shopify.com/b/shopify-brochure2-assets/logo.png",
      "sameAs": [
        "https://facebook.com/shopify",
        "https://x.com/shopify",
        "https://linkedin.com/company/shopify",
        "https://instagram.com/shopify",
        "https://youtube.com/shopify",
        "https://github.com/shopify"
      ]
    }
  ],
  "scrapedAt": "2026-10-18T12:30:15.210Z"
}

Failed / Unreachable Domain Output ($0.00 Cost Guarantee)

When a domain has a DNS resolution failure, is permanently down, or refuses connections, a clean record is output with zero charge:

{
  "url": "https://invalid-non-existent-domain-40918.com",
  "inputUrl": "invalid-non-existent-domain-40918.com",
  "normalizedUrl": "https://invalid-non-existent-domain-40918.com",
  "domain": "invalid-non-existent-domain-40918.com",
  "brandName": "invalid-non-existent-domain-40918",
  "statusCode": 500,
  "error": "Failed to load website: ENOTFOUND invalid-non-existent-domain-40918.com",
  "errorDetails": "getaddrinfo ENOTFOUND invalid-non-existent-domain-40918.com",
  "scrapedAt": "2026-10-18T12:30:18.420Z"
}

---

💡 Practical Use Cases

• 🔍 Enterprise SEO Audits: Audit meta descriptions, titles, robots.txt crawl rules, XML sitemaps, canonical tags, and H1/H2 content structures across thousands of domains in minutes.
• 💼 B2B Lead Generation & Qualification: Filter leads by confirmed B2B industry, company scale, verified LinkedIn handles, and direct contact details.
• 🧱 Technology Stack Intelligence: Detect adoption of Shopify, WordPress, Next.js, React, Google Analytics 4, and Cloudflare across entire industries.
• 🛡️ Brand Protection & Competitor Tracking: Track competitor tech migrations, logo changes, and schema updates programmatically.
• 🗺️ Sitemap & Content Discovery: Map out competitor site hierarchies, product feeds, and RSS updates programmatically.

---

❓ Frequently Asked Questions (FAQ)

How does Pay-Per-Event billing work?

Under Apify's Pay-Per-Event system, you only pay for actual domains processed and pushed:

• Standard Extraction: $1.50 per 1,000 domains ($1.00 domain analyzed + $0.50 dataset result pushed, or down to $1.20 / 1k on Apify subscription plans).
• AI-Enriched Extraction: $4.50 per 1,000 domains ($1.00 domain analyzed + $0.50 dataset result pushed + $3.00 AI intelligence, or down to $3.80 / 1k on Apify subscription plans). There are zero server rental fees or hidden usage costs.

Am I charged for domains that fail or cannot be reached?

No, absolutely not. If a website returns a network error, DNS failure, or cannot be loaded, it is charged $0.00. You only pay for successful results.

Is platform usage charged separately?

No. Apify compute unit (CU) platform usage is Free / Included. You only pay the listed event fee.

Can I provide domains without http:// or https://?

Yes. You can paste bare domains (google.com, www.nytimes.com, sub.domain.org) or full URLs. The actor automatically normalizes every input into a valid HTTP URL.

How does the social profile extractor ensure accuracy?

The actor uses intelligent multi-layer verification to eliminate tracking parameters, embedded sharing widgets, and non-profile links, ensuring you only receive verified company handles and official profiles.

Are proxies included?

Yes. The actor automatically routes requests through Apify Datacenter Proxies to ensure maximum reliability and bypass anti-bot rate limits.

How can I export the results?

Datasets can be exported directly from the Apify Console in JSON, CSV, Excel (XLSX), XML, or HTML, or ingested into your CRM or database via the Apify REST API and Webhooks.

---

🏷️ Tags & Search Keywords

website-metadata-extractor · ai-industry-classifier · seo-analyzer · robots-txt-parser · sitemap-extractor · social-links-scraper · jsonld-extractor · technology-detector · open-graph-scraper · twitter-cards · meta-tags-scraper · lead-enrichment · web-intelligence · apify · crawlee · groq