Website Tech Stack and robots.txt Checker: Shopify, WordPress, analytics, AI bot blocking

Find out what any website is built with, and what its robots.txt allows, in bulk. For each domain you get the e-commerce platform, CMS, framework, analytics, tag managers, ad pixels, marketing automation, chat, payments, consent tools, CDN and hosting. You also get the robots.txt analysis: declared sitemaps, Disallow rules, and which AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) are blocked.

Use it for B2B lead generation and list segmentation (for example, all Shopify stores using Klaviyo), sales prospecting for agencies and SaaS, competitor research, SEO audits, and AI-crawler policy research.

Features

• 90+ technology signatures across HTML, headers and cookies.
• robots.txt parsing with the standard robots-parser: AI-bot allow/block matrix, sitemaps, crawl-delay.
• Security headers (HSTS, CSP), redirects, title, meta description, language.
• No charge for unreachable sites. They are listed in SUMMARY. If all domains fail, the run fails loudly.
• Dedupe: www. and non-www are treated as the same domain.
• Hard caps: maxDomains, per-request timeouts, retries, and a concurrency limit.
• Collects no personal data: no emails or names, only public technical signals.

Use cases

• Lead generation: filter a list of domains down to Shopify or WordPress stores that use (or don't use) a given tool.
• Competitor tracking: see when a competitor switches platform or adds a new analytics, chat or A/B tool.
• AI crawler research: check which sites block GPTBot, ClaudeBot, CCBot and other AI crawlers in robots.txt.
• Agency audits: quick tech and robots.txt audit for a prospect list before outreach.

Try it free in 1 click

The input form is prefilled with a small, working example. Press Start without changing anything: the run detects the tech stack and AI-bot robots.txt rules of allbirds.com and apify.com. Our test run of exactly this input finished in about 3 seconds with 2 results. At list price that is about $0.00, which fits inside the free monthly credit of Apify's free plan. Then swap in your own values.

Prefilled input:

{
  "domains": [
    "https://www.allbirds.com",
    "apify.com"
  ]
}

Sample output

Real rows from the prefilled run on 2026-10-14 (2 of 2 results shown; long fields trimmed). Download the full dataset as JSON, CSV or Excel.

domain	httpStatus	cms	ecommercePlatform	technologies	technologyCount	aiBotsBlocked	blocksAllAiBots
apify.com	200	HubSpot CMS	–	HubSpot CMS, Next.js, Bootstrap (+16)	19	none	no
allbirds.com	200	–	Shopify	Shopify, Google Tag Manager, Microsoft Clarity (+4)	7	none	no

Input (copy-paste)

{
  "domains": ["https://www.allbirds.com", "apify.com", "https://www.nytimes.com"],
  "checkRobots": true,
  "detectTech": true,
  "maxDomains": 1000
}

API (copy-paste)

curl -X POST "https://api.apify.com/v2/acts/appetizing_horsefly~tech-stack-robots-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' -d '{"domains":["allbirds.com","apify.com"]}'

Output

{"domain":"allbirds.com","ecommercePlatform":"Shopify","technologies":["Shopify","Google Tag Manager","Klaviyo","Cloudflare"],"robotsTxtFound":true,"sitemaps":["https://www.allbirds.com/sitemap.xml"],"aiBotsBlocked":["GPTBot","CCBot"],"httpStatus":200}

Full JSON of one result (real run)

{
  "domain": "apify.com",
  "inputUrl": "https://apify.com/",
  "finalUrl": "https://apify.com/",
  "httpStatus": 200,
  "https": true,
  "redirected": false,
  "title": "Apify: Marketplace of ready-to-run tools for AI",
  "metaDescription": "Thousands of tools to automate your business. Get real-time web data, track competitors, generate leads, and integrate your apps and AI agents.",
  "language": "en",
  "server": null,
  "poweredBy": null,
  "generator": null,
  "hsts": true,
  "csp": true,
  "htmlBytes": 497554,
  "technologies": [
    "HubSpot CMS",
    "Next.js",
    "Bootstrap"
  ],
  "techByCategory": {
    "cms": [
      "HubSpot CMS"
    ],
    "framework": [
      "Next.js"
    ],
    "library": [
      "Bootstrap"
    ],
    "analytics": [
      "Google Analytics",
      "Segment",
      "Mixpanel"
    ],
    "tag-manager": [
      "Google Tag Manager"
    ],
    "advertising": [
      "LinkedIn Insight",
      "TikTok Pixel",
      "Google Ads"
    ],
    "marketing": [
      "HubSpot"
    ],
    "chat": [
      "Intercom"
    ],
    "consent": [
      "OneTrust"
    ],
    "cdn": [
      "Cloudflare",
      "Amazon CloudFront"
    ],
    "monitoring": [
      "Sentry"
    ],
    "search": [
      "Algolia"
    ],
    "fonts": [
      "Google Fonts"
    ]
  },
  "ecommercePlatform": null,
  "cms": "HubSpot CMS",
  "technologyCount": 19,
  "robotsTxtFound": true,
  "robotsStatus": 200,
  "robotsSizeBytes": 174,
  "sitemaps": [
    "https://apify.com/sitemap.xml"
  ],
  "userAgentsListed": [
    "*"
  ],
  "disallowRuleCount": 0,
  "allBotsBlockedFromHome": false,
  "googlebotAllowedHome": true,
  "aiBotsBlocked": [],
  "aiBotsAllowed": [
    "GPTBot",
    "ChatGPT-User",
    "OAI-SearchBot"
  ],
  "blocksAllAiBots": false,
  "crawlDelay": null,
  "checkedAt": "2026-10-12T17:48:26.489Z"
}

Pricing

Pay per event: $0.002 per domain checked ($2 per 1,000). Unreachable domains and duplicates are free.

Notes

Detection is signature-based, from the homepage only. It can miss tools that load late or only on other pages. Sites behind heavy bot protection may return limited data, and those rows carry a warning.

Use it as an MCP tool (Claude, Cursor, ChatGPT, any AI agent)

This Actor works as a tool in the Apify MCP server, so an AI agent can call it directly and you pay per result, the same as a normal run. Add this server URL to any MCP client:

https://mcp.apify.com/?tools=appetizing_horsefly/tech-stack-robots-checker

Sign in with your Apify account (or pass your Apify API token as a Bearer header). The agent reads the input fields above from the Actor's input schema. Example request to the agent: "Which of these 20 competitor domains run Shopify, and which block GPTBot in robots.txt?"

Support

Questions, a bug, or a data source you need? Use this Actor's Issues tab or email gm-labs@mail.grokbot.com. Built and maintained by GM Labs.