🌐 Website Metadata Extractor – Analyze & Extract Site SEO, Tags, and Robots Data

This Apify actor crawls and extracts structured metadata from any website, including SEO tags, robot rules, sitemap references, headings, and technology fingerprints. It’s ideal for SEO analysis, data enrichment, domain audits, and competitive research — built using Crawlee + Cheerio for lightweight yet powerful web parsing.

---

🚀 What It Does

The actor processes each input URL and extracts:

• 🧠 Meta Information: Title, description, keywords, charset, viewport, theme color, etc.
• 🤖 Robots.txt Rules: Indexed by user-agent with all allow/disallow paths.
• 🗺️ Sitemaps: All discovered sitemap file URLs.
• 🧩 Social Metadata: Open Graph (og:), Twitter Cards, and Facebook verification tags.
• 🧱 Structured Data: Extracted JSON-LD schemas.
• 💬 Content Structure: First H1, all H1s & H2s, and total visible word count.
• 🕵️ Technologies: Detected frameworks, analytics tools, or CMS hints.
• 🌍 Hreflang Tags: Alternate language/country versions of the site.

---

💡 It Handles:

• ✅ Website metadata extraction from any URL
• ✅ Robots.txt and sitemap discovery
• ✅ JSON-LD and SEO tag parsing
• ✅ Automatic detection of CMS and tech stacks
• ✅ Fast mode (skip domain analysis) for large-scale scans
• ✅ Works with static and dynamic pages

---

⚙️ Input Configuration

Field	Type	Description	Default / Example
startUrls	Array	A list of website URLs to analyze.	["https://apify.com", "https://www.google.com"]
disableDomainAnalysis	Boolean	If true, skips robots.txt and sitemap fetching for faster runs (less data).	false

---

🧩 Example Input

{
  "startUrls": ["https://www.shopify.com", "https://www.wix.com"],
  "disableDomainAnalysis": false
}

---

📊 Example Output

{
  "url": "https://www.shopify.com",
  "robotsTxt": {
    "userAgents": {
      "GoogleDocs": {
        "allow": [],
        "disallow": ["/"]
      },
      "AdIdxBot": {
        "allow": ["*/ppc/*", "*utm_medium=cpc*"],
        "disallow": []
      },
      "Pinterestbot": {
        "allow": ["*/ppc/*", "*utm_medium=cpc*"],
        "disallow": []
      },
      "AdsBot-Google-Mobile": {
        "allow": ["*/ppc/*", "*utm_medium=cpc*"],
        "disallow": []
      },
      "AdsBot-Google": {
        "allow": ["*/ppc/*", "*utm_medium=cpc*"],
        "disallow": []
      },
      "Bingbot": {
        "allow": [],
        "disallow": ["/llms.txt"]
      },
      "Googlebot": {
        "allow": [],
        "disallow": ["/llms.txt"]
      },
      "*": {
        "allow": [],
        "disallow": [
          "*.data$",
          "*/account",
          "*/auth/callback",
          "*/authenticate",
          "*/authentication",
          "*/editor$",
          "*/finalize$",
          "*/loading$",
          "*/onboarding$",
          "*/ppc/*",
          "*/result$",
          "*/services/identity/login",
          "*/services/sa-appointments/login",
          "*/services/sa-appointments/redirect",
          "*/services/sa-appointments/stores.json",
          "*/stock-photos/*?*page=*?*page=*",
          "*/stock-photos/*?link_search=",
          "*/stock-photos/*?q=",
          "*/stock-photos/@*",
          "*/stock-photos/admin",
          "*/stock-photos/photos/search$",
          "*/stock-photos/photos/search?",
          "*/store/account",
          "*/store/admin",
          "*/store/cart",
          "*/store/carts",
          "*/store/checkout",
          "*/store/checkouts/",
          "*/store/orders",
          "*/tools/*/show*",
          "*country=*",
          "*itcat=*",
          "*itterm=*",
          "*lang=*",
          "*link_search=*",
          "*page=*",
          "*prev_msid=*",
          "*utm_medium=cpc*",
          "/*/*?*shpxid=*",
          "/*/*?services*",
          "/*/*digital_wallets/dialog",
          "/*/admin/",
          "/*/apple-app-site-association",
          "/*/blog-article-remove-faq-utms-*.js",
          "/*/blog.atom",
          "/*/blog/search$",
          "/*/blog/search?",
          "/*/blogs/blog.atom",
          "/*/blogs/technology.atom",
          "/*/blogsearch$",
          "/*/cannabis",
          "/*/cdn-cgi/challenge-platform*",
          "/*/email-validation",
          "/*/enterprise/blog/search$",
          "/*/enterprise/blog/search?",
          "/*/growth-tools-assets",
          "/*/landing/",
          "/*/retail/search$",
          "/*/retail/search?",
          "/*/step/",
          "/*/submit",
          "/*/submitted",
          "/*/technology.atom",
          "/*/technology/tagged/*page*",
          "/*/tools/business-name-generator/searchbutton*",
          "/*/tools/business-name-generator/searchpage*",
          "/*/tools/business-name-generator/searchutf8*",
          "/*/ventureone",
          "/*CampaignId*",
          "/*hashid=%subscriber_hash%",
          "/*hubs_content*",
          "/.s/assets/shopifycloud/collabs-community-widget/widget.js",
          "/.well-known/traffic-advice",
          "/500",
          "/__dux",
          "/__manifest",
          "/__pb/mm",
          "/__pb/rp",
          "/__pb/rr",
          "/authenticate",
          "/blog-article-remove-faq-utms-*.js",
          "/blog/search$",
          "/blog/search?",
          "/blogs/technology.atom",
          "/blogsearch",
          "/cannabis",
          "/careers/portal/*",
          "/careers/search?*",
          "/cdn-cgi/challenge-platform*",
          "/email-validation",
          "/enterprise/blog/search$",
          "/enterprise/blog/search?",
          "/meta.json",
          "/oauth",
          "/retail/search$",
          "/retail/search?"
        ]
      }
    }
  },
  "sitemapFileUrls": [
    "https://www.shopify.com/sitemaps_list.xml",
    "https://www.shopify.com/sitemap.xml"
  ],
  "metaTags": {
    "title": "Shopify: The All-in-One Commerce Platform for Businesses - Shopify",
    "viewport": "width=device-width,initial-scale=1",
    "description": "Try Shopify free and start a business or grow an existing one. Get more than ecommerce software with tools to manage every part of your business.",
    "fb:pages": "20409006880",
    "fb:app_id": "847460188612391",
    "og:type": "website",
    "og:site_name": "Shopify",
    "og:title": "Shopify: The All-in-One Commerce Platform for Businesses - Shopify",
    "og:description": "Try Shopify free and start a business or grow an existing one. Get more than ecommerce software with tools to manage every part of your business.",
    "og:image": "https://cdn.shopify.com/b/shopify-brochure2-assets/ad99387e77223a56caca1c7b18e31970.png",
    "twitter:image": "https://cdn.shopify.com/b/shopify-brochure2-assets/ad99387e77223a56caca1c7b18e31970.png",
    "og:url": "https://www.shopify.com/",
    "twitter:card": "summary_large_image",
    "twitter:site": "Shopify",
    "twitter:account_id": "17136315",
    "twitter:title": "Shopify: The All-in-One Commerce Platform for Businesses - Shopify",
    "twitter:description": "Try Shopify free and start a business or grow an existing one. Get more than ecommerce software with tools to manage every part of your business.",
    "ssr": "false",
    "canonical": "https://www.shopify.com/",
    "charset": "utf-8",
    "favicon": "https://cdn.shopify.com/shopifycloud/web/assets/v1/favicon-default-6cbad9de243dbae3.ico",
    "apple-touch-icon": "https://cdn.shopify.com/b/shopify-brochure2-assets/c97c60ca19c64a8b5378d9f9e971f7bd.png"
  },
  "jsonLd": [
    {
      "@context": "https://schema.org/",
      "@type": "Corporation",
      "name": "Shopify",
      "url": "https://www.shopify.com/",
      "logo": "https://cdn.shopify.com/shopifycloud/brochure/assets/brand-assets/shopify-logo-primary-logo-456baa801ee66a0a435671082365958316831c9960c480451dd0330bcdae304f.svg",
      "sameAs": [
        "https://github.com/Shopify",
        "https://en.wikipedia.org/wiki/Shopify",
        "https://www.youtube.com/shopify",
        "https://www.instagram.com/shopify/?hl=en",
        "https://x.com/Shopify",
        "https://www.linkedin.com/company/shopify/",
        "https://www.facebook.com/shopify/"
      ]
    }
  ],
  "hreflangLinks": {
    "es-AR": "https://www.shopify.com/ar",
    "en-AU": "https://www.shopify.com/au",
    "de-AT": "https://www.shopify.com/at",
    "ru": "https://www.shopify.com/by",
    "nl-BE": "https://www.shopify.com/be",
    "de-BE": "https://www.shopify.com/be-de",
    "fr-BE": "https://www.shopify.com/be-fr",
    "pt": "https://www.shopify.com/br",
    "bg": "https://www.shopify.com/bg",
    "en-CA": "https://www.shopify.com/ca",
    "fr-CA": "https://www.shopify.com/ca-fr",
    "es-CL": "https://www.shopify.com/cl",
    "es-CO": "https://www.shopify.com/co",
    "cs": "https://www.shopify.com/cz",
    "da": "https://www.shopify.com/dk",
    "fi": "https://www.shopify.com/fi",
    "fr": "https://www.shopify.com/fr",
    "de": "https://www.shopify.com/de",
    "el": "https://www.shopify.com/gr",
    "zh-HK": "https://www.shopify.com/hk",
    "en-HK": "https://www.shopify.com/hk-en",
    "hu": "https://www.shopify.com/hu",
    "hi": "https://www.shopify.com/in-hi",
    "en-IN": "https://www.shopify.com/in",
    "id": "https://www.shopify.com/id-id",
    "en-ID": "https://www.shopify.com/id",
    "en-IE": "https://www.shopify.com/ie",
    "it": "https://www.shopify.com/it",
    "ja-JP": "https://www.shopify.com/jp",
    "ko": "https://www.shopify.com/kr",
    "lt": "https://www.shopify.com/lt",
    "en-MY": "https://www.shopify.com/my",
    "es-MX": "https://www.shopify.com/mx",
    "nl": "https://www.shopify.com/nl",
    "en-NZ": "https://www.shopify.com/nz",
    "en-NG": "https://www.shopify.com/ng",
    "nb": "https://www.shopify.com/no",
    "en-NO": "https://www.shopify.com/no-en",
    "es-PE": "https://www.shopify.com/pe",
    "en-PH": "https://www.shopify.com/ph",
    "pl": "https://www.shopify.com/pl",
    "pt-PT": "https://www.shopify.com/pt",
    "ro": "https://www.shopify.com/ro",
    "en-SG": "https://www.shopify.com/sg",
    "en-ZA": "https://www.shopify.com/za",
    "es-ES": "https://www.shopify.com/es-es",
    "sv": "https://www.shopify.com/se",
    "de-CH": "https://www.shopify.com/ch",
    "zh-Hant-TW": "https://www.shopify.com/tw",
    "th": "https://www.shopify.com/th",
    "tr": "https://www.shopify.com/tr",
    "en-GB": "https://www.shopify.com/uk",
    "en": "https://www.shopify.com",
    "es": "https://www.shopify.com/es",
    "zh-Hans": "https://www.shopify.com/zh"
  },
  "socialLinks": {
    "facebook": "https://www.facebook.com/shopify",
    "twitter": "https://twitter.com/shopify",
    "youtube": "https://www.youtube.com/user/shopify",
    "instagram": "https://www.instagram.com/shopify/",
    "tiktok": "https://www.tiktok.com/@shopify",
    "linkedin": "https://www.linkedin.com/company/shopify",
    "pinterest": "https://www.pinterest.com/shopify/",
    "github": "https://github.com/Shopify",
    "x": "https://x.com/Shopify"
  },
  "h1": "Be the nextbig thing",
  "allH1s": ["Be the nextbig thing"],
  "allH2s": [
    "store they line up for",
    "The one commerce platform behind it all",
    "There’s no better place for you to build",
    "Start selling in no time"
  ],
  "wordCount": 969
}

{
	"url": "https://www.wix.com",
	"robotsTxt": {
		"userAgents": {
			"*": {
				"allow": [],
				"disallow": [
					"/api/",
					"/blogtemp",
					"/wixblog",
					"/bo/",
					"/editor.jsp",
					"/noflashhtml",
					"/siteBackHtml",
					"/wix/",
					"/wixpress/",
					"/wixdemo/",
					"/wix-editor/",
					"/editor2.jsp",
					"/flash/",
					"/flash-templates/",
					"/website-template/view/flash/",
					"/facebook-template/",
					"/facebook/templates/",
					"/website/templates/flash/",
					"/html5/",
					"/my-account",
					"/blog/*sitemap.xml$",
					"/velo/profile",
					"/velo/forum/search",
					"/velo/forum/main/comment",
					"/*velo-pt/profile",
					"/*velo-pt/forum/search",
					"/*velo-pt/forum/main/comment",
					"/app-market/search-result?query=",
					"/despiration/testseo",
					"/site-react-dropdown*",
					"/soundcloud-tpa*",
					"/*cacheKiller=",
					"/*hubs_content",
					"/*hubs_post",
					"/*hubs_signup-cta",
					"/lp-lang/mobilewebsite",
					"/mystunningwebsites/",
					"/dashboard",
					"/_api/albums-node-server",
					"/_partials/wix-vod",
					"/designers/events/fullscreen-page",
					"/photo-test-editor-x",
					"/thunderbolt/",
					"/meta-site/",
					"/learn/search-results",
					"/?criteria=",
					"/blog/*/search-results?q",
					"/blog/search-results?q",
					"/experts-arena/",
					"/createawebsite/",
					"/website/templates*?screen=",
					"/*?sort=",
					"/marketplace/hire*?serviceIds=",
					"/marketplace/hire*&serviceIds=",
					"/marketplace/brief/",
					"/wix-platform",
					"/blog-temp-ru",
					"/blog/search?*",
					"*/fullscreen-page",
					"/wearepages",
					"*/laboratory/conductAllInScope",
					"/studio/blog/*?filter=",
					"/studio/academy/search-results",
					"/studio/academy/search",
					"/studio/blog/search-results?q",
					"/studio/blog/*&filter=",
					"/seo/learn/assets*?topic=",
					"/seo/learn/assets*&topic=",
					"/seo/learn/assets*?resource-type=",
					"/seo/learn/assets*&resource-type=",
					"/studio-tech-desgin",
					"/_api/dealer-offer-events-service/proxy/v1/dealer-offer-events",
					"/website/builder?storyId=",
					"/studio-tech-design",
					"/domain/names/results?",
					"/business-name-generator/list",
					"/studio/templates?criteria",
					"/domains/results",
					"/_api/header-footer-service/content",
					"/wixel/templates/search?",
					"/wixel/templates/*?color",
					"/wixel/templates/*?style",
					"/wixel/templates/*?price",
					"/marketplace/templates?sort="
				]
			}
		}
	},
	"sitemapFileUrls": [
		"https://www.wix.com/sitemap.xml"
	],
	"metaTags": {
		"title": "Website Builder - Create a Free Website In Minutes | Wix.com",
		"viewport": "width=device-width, initial-scale=1",
		"generator": "Wix.com Website Builder",
		"format-detection": "telephone=no",
		"skype_toolbar": "skype_toolbar_parser_compatible",
		"description": "Get everything you need to create your website, your way. With a free easy-to-use website builder, integrated hosting, and essential business solutions.",
		"og:title": "Your vision. Your goals. Your website. | Wix.com",
		"og:description": "Get everything you need to create your website, your way. From an intuitive website builder to built-in business solutions and AI tools—try Wix for free.",
		"og:image": "https://static.wixstatic.com/media/0784b1_31d34d128f8b424cb8cb53607fe77c95~mv2.jpg/v1/fill/w_1200,h_630,al_c/0784b1_31d34d128f8b424cb8cb53607fe77c95~mv2.jpg",
		"og:image:width": "1200",
		"og:image:height": "630",
		"og:url": "https://www.wix.com",
		"og:site_name": "wix.com",
		"og:type": "website",
		"twitter:card": "summary_large_image",
		"twitter:title": "Your vision. Your goals. Your website. | Wix.com",
		"twitter:description": "Get everything you need to create your website, your way. From an intuitive website builder to built-in business solutions and AI tools—try Wix for free.",
		"twitter:image": "https://static.wixstatic.com/media/0784b1_31d34d128f8b424cb8cb53607fe77c95~mv2.jpg/v1/fill/w_1200,h_630,al_c/0784b1_31d34d128f8b424cb8cb53607fe77c95~mv2.jpg",
		"canonical": "https://www.wix.com",
		"charset": "utf-8",
		"favicon": "https://www.wix.com/favicon.ico",
		"apple-touch-icon": "https://www.wix.com/favicon.ico"
	},
	"jsonLd": [
		{
			"@context": "https://schema.org/",
			"@type": "HowTo",
			"name": "How to create a website for free",
			"description": "Follow these 7 simple steps to create a website today",
			"step": [
				{
					"@type": "HowToStep",
					"position": 1,
					"name": "Pick a platform",
					"text": "Sign up for a secure and reliable free website builder like Wix."
				},
				{
					"@type": "HowToStep",
					"position": 2,
					"name": "Plan out your website",
					"text": "Map out your goals, site structure and who your audience is."
				},
				{
					"@type": "HowToStep",
					"position": 3,
					"name": "Start creating",
					"text": "Choose from 900+ free templates or use the AI website builder."
				},
				{
					"@type": "HowToStep",
					"position": 4,
					"name": "Customize your website",
					"text": "Use the drag and drop editor and tailor your site to fit your brand."
				},
				{
					"@type": "HowToStep",
					"position": 5,
					"name": "Optimize for search engines",
					"text": "Increase your site’s visibility with a suite of built-in SEO tools."
				},
				{
					"@type": "HowToStep",
					"position": 6,
					"name": "Publish your website",
					"text": "Register and connect a custom domain name and go live."
				},
				{
					"@type": "HowToStep",
					"position": 7,
					"name": "Promote and drive traffic",
					"text": "Use built-in marketing tools to grow and expand your reach."
				}
			]
		},
		{
			"@context": "https://schema.org",
			"@type": "Organization",
			"name": "Wix.com",
			"legalName": "Wix.com Ltd",
			"url": "https://www.wix.com/",
			"logo": "https://static.wixstatic.com/media/9ab0d1_2ff5ca18550e405ea1844e52afaff120~mv2.jpg/v1/fill/w_333,h_154,al_c,lg_1,q_80,enc_auto/Wix%20logo%20white%20BG.jpg",
			"sameAs": [
				"https://www.facebook.com/wix",
				"https://twitter.com/wix",
				"https://www.instagram.com/wix",
				"https://www.youtube.com/user/Wix",
				"https://www.linkedin.com/company/wix-com/",
				"https://www.pinterest.com/wixcom/",
				"https://en.wikipedia.org/wiki/Wix.com"
			],
			"contactPoint": {
				"@type": "ContactPoint",
				"contactType": "Customer Support",
				"url": "https://www.wix.com/contact"
			}
		},
		{
			"@context": "https://schema.org",
			"@type": "FAQPage",
			"mainEntity": [
				{
					"@type": "Question",
					"name": "Is it easy to build a website?",
					"acceptedAnswer": {
						"@type": "Answer",
						"text": "Yes. Wix offers a few different ways to create your own free website, so you can choose the creation process that works best for you. Pick from 900+ designer-made templates, or use our AI website builder to create a business-ready site in no time using a conversational interface. You can also start from scratch using Wix’s drag and drop website builder. Whichever way you choose, you can always continue customizing in the Editor for total website design freedom."
					}
				},
				{
					"@type": "Question",
					"name": "How do I create a website?",
					"acceptedAnswer": {
						"@type": "Answer",
						"text": "1. Plan your website. First, think about the type of site you’re creating and your target audience. With that in mind, you can start mapping out the pages you want to incorporate like the “About” and “Contact” pages, perhaps a blog or a photo gallery, and a page for products or services. 2. Build with AI or choose a template. Chat with our AI website builder about your business and preferences to instantly get a fully functional and customizable website built for you. You can also start by choosing from a variety of templates, all professionally designed with the best site practices in mind. 3. Customize your website. Whether you start with AI or a template, you can use our intuitive drag and drop editor to customize your site to match your brand. Your site will also be optimized for mobile, but in the editor you have the option to make changes and customize your site’s mobile view. 4. Get a domain name. When trying to come up with the perfect domain name, you can use Wix to search and register available names, or connect an existing one to your new site. 5. Optimize for search engines. Use a suite of advanced SEO tools to help you optimize your site and increase organic traffic. 6. Publish and promote your website. Once you’re happy with your site, you’re ready to hit ‘publish’ and start gaining traffic. Now’s the time to promote your site with built-in marketing tools and streamline your customer management processes with a smart CRM system. By following these steps, you'll be able to build a powerful online presence that drives business growth."
					}
				},
				{
					"@type": "Question",
					"name": "How do I choose the best website builder?",
					"acceptedAnswer": {
						"@type": "Answer",
						"text": "The best website builder is one that will support your specific needs and business goals. As you research which builder is right for you, here are some important factors to consider: Ease of use: Look for a website maker with an intuitive interface. Using a drag and drop editor that includes AI tools can simplify the web design process and make it more efficient. Website templates: Look for a platform that offers a large selection of customizable templates tailored to a variety of industries and business types. Business solutions: Some website builders include built-in solutions to help you manage and grow your business seamlessly like eCommerce, scheduling, marketing tools and more. Security: It’s important that the website builder you choose offers reliable web hosting with data encryption to protect your and your users’ data. Customer support: Reliable support is crucial. Check what support options your website builder provides, and make sure you’ll be able to get the help you need 24/7. Wix fulfills all these requirements, giving you the tools you need to create a website and confidently run and grow your business all in one place."
					}
				},
				{
					"@type": "Question",
					"name": "How long does it take to build a website?",
					"acceptedAnswer": {
						"@type": "Answer",
						"text": "The amount of time it takes to build a website is now quicker than ever thanks to technological advancements in the field. In fact, you can build a functional and professional looking website in a matter of hours on a platform like Wix. To make the process even faster, you can chat with Wix’s AI website creator about the kind of site you want and you’ll have a draft of your website ready in minutes. Then you can customize it to make it your own with the drag and drop editor."
					}
				},
				{
					"@type": "Question",
					"name": "How much does it cost to build a website?",
					"acceptedAnswer": {
						"@type": "Answer",
						"text": "The cost of building a website varies depending on the features you need and whether you opt to build it yourself with a website builder or pay a developer to build it for you. On a website builder such as Wix, you can build as many websites as you want for free, however, you’ll have to upgrade to a Premium plan to connect a custom domain and get advanced business features. The cost of creating a website may be significantly higher if you don’t opt for an all-inclusive platform like Wix."
					}
				}
			]
		},
		{
			"@context": "https://schema.org",
			"@type": "VideoObject",
			"name": "Wix website builder",
			"description": "Have a conversation with our AI website builder to create your own personalized, business-ready site.",
			"thumbnailUrl": "https://static.wixstatic.com/media/0784b1_1c525438f02149c3b0489e0d789890f0f002.png",
			"uploadDate": "2024-09-24",
			"duration": "PT0M20S",
			"contentUrl": "https://video.wixstatic.com/video/0784b1_1c525438f02149c3b0489e0d789890f0/360p/mp4/file.mp4"
		}
	],
	"hreflangLinks": {
		"fr": "https://fr.wix.com",
		"uk": "https://uk.wix.com",
		"pt": "https://pt.wix.com",
		"cs": "https://cs.wix.com",
		"it": "https://it.wix.com",
		"vi": "https://vi.wix.com",
		"nl": "https://nl.wix.com",
		"ko": "https://ko.wix.com",
		"de": "https://de.wix.com",
		"ru": "https://ru.wix.com",
		"th": "https://th.wix.com",
		"id": "https://id.wix.com",
		"sv": "https://sv.wix.com",
		"tr": "https://tr.wix.com",
		"da": "https://da.wix.com",
		"en": "https://www.wix.com",
		"es": "https://es.wix.com",
		"ja": "https://ja.wix.com",
		"x-default": "https://www.wix.com",
		"no": "https://no.wix.com",
		"zh": "https://zh.wix.com",
		"pl": "https://pl.wix.com"
	},
	"socialLinks": {
		"facebook": "https://www.facebook.com/wix",
		"youtube": "https://www.youtube.com/user/Wix",
		"instagram": "https://www.instagram.com/wix",
		"tiktok": "https://www.tiktok.com/@wix",
		"pinterest": "https://www.pinterest.com/wixcom",
		"twitter": "https://twitter.com/wix",
		"linkedin": "https://www.linkedin.com/company/wix-com?trk=biz-companies-cym"
	},
	"detectedTechnologies": [
		"Wix.com Website Builder"
	],
	"h1": "Create a website without limits",
	"allH1s": [
		"Create a website without limits"
	],
	"allH2s": [
		"Create your site in minutes with our AI website builder",
		"Or choose a professionally designed template",
		"Customize to make it your own",
		"Add anything you need for your business as you go",
		"Make your website official with your own domain name",
		"Market your site from launch to scale",
		"Run your business from one dashboard",
		"Grow your website on a rock-solid foundation",
		"How to create a website for free",
		"Thriving with Wix",
		"Get inspired, gain new skills and see what’s trending",
		"Made on Wix",
		"We’re here for you 24/7",
		"Website builder FAQ",
		"Your vision. Your goals. Your website."
	],
	"wordCount": 9996
}

---

🧠 Features

• 🔍 Extracts full SEO metadata (title, description, keywords, social tags)
• 🤖 Parses robots.txt with detailed user-agent rules
• 🗺️ Finds all sitemap files for deeper crawling potential
• 📊 Counts visible words and extracts H1/H2 content
• 🧱 Detects frameworks and CMS technologies automatically
• ⚡ Supports Fast Mode to skip domain-level scans for large datasets
• 🛡️ Built with Crawlee + Apify SDK for reliability and scalability

---

💡 Use Cases

• SEO Audits: Evaluate site structure, metadata, and content depth
• Competitive Research: Identify frameworks and SEO strategies used by other sites
• Data Enrichment: Add structured metadata and robot rules to domain databases
• Web Intelligence: Analyze robots/sitemap behavior across industries

---

🧑‍💻 Developer Info

Author: CodeScraper

---

🏷️ Tags

website-metadata-extractor · seo-analyzer · metadata-scraper · robots-txt-parser · sitemap-analyzer · jsonld-extractor · seo-research · web-intelligence · apify · crawlee