Pricing

from $2.00 / 1,000 website enricheds

Try for free

Go to Apify Store

Company Website Enrichment: Contacts, Socials, Tech Stack

Try for free

Turn company domains into lead data: name, description, logo, generic emails (info@, sales@), phones, address, LinkedIn and socials, key pages and 130+ technologies. Input: domains, or another Actor's dataset such as Google Maps results. One row per company. Respects robots.txt.

Pricing

from $2.00 / 1,000 website enricheds

Rating

0.0

(0)

Developer

William Jin

Actor stats

Bookmarked

Total users

Monthly active users

4 hours ago

Last modified

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

446K

4.7

(603)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

261K

4.7

(175)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

354K

4.7

(370)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

660K

4.7

(1,818)

LinkedIn Profile Scraper + Email (No Cookies)

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

86K

4.6

(93)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

213K

4.6

(220)

Tweet Scraper V2 - X Scraper / Twitter Scraper / X.com Scraper

apidojo/tweet-scraper

Lightning-fast search, URL, list, and profile scraping, with customizable filters. At $0.40 per 1000 tweets, and 30-80 tweets per second, it is ideal for researchers, entrepreneurs, and businesses! Get comprehensive insights from Twitter (X) now!

API Dojo

113K

3.9

(196)

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract posts, videos, and engagement metrics from Facebook pages. Get text captions, reactions, video transcripts, images, external links, collaborators, and more from Facebook pages and profiles. Export ad data, schedule runs via API, and integrate with other tools or AI workflows.

Apify

130K

4.7

(236)

Instagram Post Scraper

apify/instagram-post-scraper

Scrape Instagram posts. Just add one or more Instagram usernames and get your data in seconds including caption, metrics, images, mentions, coauthors, recent comments, sponsored status, video duration, views. Export scraped data, schedule scraper via API, integrate with other tools or AI workflows.

Apify

147K

4.3

(138)

Web Fetch

apify/web-fetch

Download any web page by URL and convert it to clean Markdown, plain text, HTML, and other formats. Automatically renders JavaScript and works around bot blocking. The Actor provides a real-time API for quick response times.

Apify

37K

5.0

(3)

FROM apify/actor-node:20 COPY --chown=myuser package*.json ./ RUN npm --quiet set progress=false \ && npm install --omit=dev --omit=optional \ && echo "Installed NPM packages:" && (npm list --omit=dev --all || true) \ && echo "Node.js version:" && node --version COPY --chown=myuser . ./ CMD npm start --silent

{ "actorSpecification": 1, "name": "company-website-enrichment", "title": "Company Website Enrichment: Contacts, Socials, Tech Stack", "description": "Turn a list of company domains into lead data: company name, description, logo, generic contact emails (info@, sales@), phones, address, LinkedIn and social profiles, contact and careers pages, job board and 130+ technologies. Fast HTTP, respects robots.txt.", "version": "0.1", "buildTag": "latest", "meta": { "templateId": "js-empty" }, "input": "./input_schema.json", "storages": { "dataset": "./dataset_schema.json" }, "output": "./output_schema.json", "dockerfile": "./Dockerfile", "readme": "../README.md", "minMemoryMbytes": 256, "maxMemoryMbytes": 4096, "defaultMemoryMbytes": 512 }

{ "actorSpecification": 1, "fields": { "type": "object", "properties": { "input": { "title": "Input", "type": [ "string", "null" ], "description": "Website as you entered it", "examples": [ "stripe.com" ] }, "domain": { "title": "Domain", "type": [ "string", "null" ], "description": "Registrable domain, e.g. stripe.com", "examples": [ "stripe.com" ] }, "url": { "title": "Url", "type": [ "string", "null" ], "description": "Final homepage URL after redirects", "examples": [ "https://stripe.com/" ] }, "status": { "title": "Status", "type": [ "string", "null" ], "description": "ok, unreachable, http_error, blocked_by_robots_txt, not_html, invalid_input or error. Only 'ok' rows are charged.", "examples": [ "ok" ] }, "httpStatus": { "title": "Http Status", "type": [ "integer", "null" ], "description": "Homepage HTTP status", "examples": [ 200 ] }, "companyName": { "title": "Company Name", "type": [ "string", "null" ], "description": "Company name (schema.org, og:site_name or page title)", "examples": [ "Stripe" ] }, "legalName": { "title": "Legal Name", "type": [ "string", "null" ], "description": "Legal name (schema.org)", "examples": [ "Stripe, LLC" ] }, "description": { "title": "Description", "type": [ "string", "null" ], "description": "Meta description", "examples": [ "Stripe is a financial services platform that helps all types of businesses accept payments, build flexible billing models, and manage mon…" ] }, "language": { "title": "Language", "type": [ "string", "null" ], "description": "Site language (html lang)", "examples": [ "en-US" ] }, "logoUrl": { "title": "Logo URL", "type": [ "string", "null" ], "description": "Logo or icon URL", "examples": [ "https://images.stripeassets.com/fzn2n1nzq965/1hgcBNd12BfT9VLgbId7By/01d91920114b124fb4cf6d448f9f06eb/favicon.svg" ] }, "emails": { "title": "Emails", "type": [ "array", "null" ], "description": "Generic role-based emails on the company's own domain (info@, sales@, support@...). Personal addresses are never collected.", "examples": [ [] ] }, "phones": { "title": "Phones", "type": [ "array", "null" ], "description": "Company phone numbers (tel: links and schema.org)", "examples": [ [] ] }, "address": { "title": "Address", "type": [ "object", "null" ], "description": "Postal address from schema.org: street, city, region, postalCode, country" }, "socials": { "title": "Socials", "type": [ "object", "null" ], "description": "Company social profiles: linkedin, twitter, facebook, instagram, youtube, tiktok, github, pinterest", "examples": [ { "github": "https://github.com/stripe", "twitter": "https://x.com/stripe", "youtube": "https://www.youtube.com/@stripe", "linkedin": "https://www.linkedin.com/company/stripe", "facebook": "https://www.facebook.com/StripeHQ", "instagram": "https://www.instagram.com/stripehq" } ] }, "contactPageUrl": { "title": "Contact Page URL", "type": [ "string", "null" ], "description": "Contact page", "examples": [ "https://stripe.com/contact/sales" ] }, "aboutPageUrl": { "title": "About Page URL", "type": [ "string", "null" ], "description": "About page", "examples": [ "https://stripe.com/about" ] }, "careersPageUrl": { "title": "Careers Page URL", "type": [ "string", "null" ], "description": "Careers page", "examples": [ "https://stripe.com/careers" ] }, "legalPageUrl": { "title": "Legal Page URL", "type": [ "string", "null" ], "description": "Imprint / legal page", "examples": [ "https://stripe.com/legal/ssa" ] }, "jobBoard": { "title": "Job Board", "type": [ "object", "null" ], "description": "Detected job board: ats (greenhouse, lever, ashby, workable, recruitee, workday...), slug, url" }, "technologies": { "title": "Technologies", "type": [ "array", "null" ], "description": "Detected technologies: {name, category}", "examples": [ [ { "name": "Next.js", "category": "JavaScript framework" }, { "name": "React", "category": "JavaScript framework" } ] ] }, "foundingDate": { "title": "Founding Date", "type": [ "string", "null" ], "description": "Founding date (schema.org)" }, "numberOfEmployees": { "title": "Number Of Employees", "type": [ "string", "null" ], "description": "Employee count (schema.org, when published)" }, "pagesFetched": { "title": "Pages Fetched", "type": [ "array", "null" ], "description": "Pages read on the site", "examples": [ [ "https://stripe.com/", "https://stripe.com/contact/sales" ] ] }, "error": { "title": "Error", "type": [ "string", "null" ], "description": "Why a site was not enriched" }, "scrapedAt": { "title": "Scraped At", "type": [ "string", "null" ], "description": "Scrape time (ISO)", "examples": [ "2026-10-23T19:31:56.445Z" ] } }, "examples": [ { "input": "stripe.com", "domain": "stripe.com", "url": "https://stripe.com/", "status": "ok", "httpStatus": 200, "companyName": "Stripe", "legalName": "Stripe, LLC", "description": "Stripe is a financial services platform that helps all types of businesses accept payments, build flexible billing models, and manage mon…" } ] }, "views": { "overview": { "title": "Overview", "transformation": { "fields": [ "domain", "companyName", "status", "emails", "phones", "socials", "address", "contactPageUrl", "careersPageUrl", "jobBoard", "technologies", "description", "url" ] }, "display": { "component": "table", "properties": { "domain": { "label": "Domain" }, "companyName": { "label": "Company" }, "status": { "label": "Status" }, "emails": { "label": "Emails", "format": "array" }, "phones": { "label": "Phones", "format": "array" }, "socials": { "label": "Socials", "format": "object" }, "address": { "label": "Address", "format": "object" }, "contactPageUrl": { "label": "Contact page", "format": "link" }, "careersPageUrl": { "label": "Careers", "format": "link" }, "jobBoard": { "label": "Job board", "format": "object" }, "technologies": { "label": "Tech stack", "format": "array" }, "description": { "label": "Description" }, "url": { "label": "Website", "format": "link" } } } } } }

{ "title": "Company Website Enrichment input", "type": "object", "schemaVersion": 1, "properties": { "websites": { "title": "Websites or domains", "type": "array", "editor": "stringList", "description": "Array of strings, e.g. [\"stripe.com\",\"https://www.hubspot.com\"]. Domains or URLs; one row per company. Required unless you pass datasetId. Sites that fail get a free row with status and error instead of being dropped.", "prefill": [ "stripe.com", "hubspot.com", "zapier.com" ] }, "datasetId": { "title": "Or: dataset ID from another Actor", "type": "string", "editor": "textfield", "description": "Optional. Enrich the websites in another run's dataset, e.g. Google Maps, job or lead scraper results. Example: \"aBcD1234efGH5678\". Duplicates are removed. In an integration (run this after another Actor finishes) use {{resource.defaultDatasetId}}.", "sectionCaption": "Chain from another Actor", "resourceType": "dataset", "resourcePermissions": [ "READ" ] }, "websiteField": { "title": "Website field in that dataset", "type": "string", "editor": "textfield", "description": "Field holding the URL or domain, e.g. \"website\" (dot paths like \"company.url\" work). Leave empty to auto-detect: website, websiteUrl, companyWebsite, companyUrl, companyDomain, domain, homepage, url, loc." }, "datasetMaxItems": { "title": "Max websites from the dataset", "type": "integer", "editor": "number", "description": "Read at most this many websites from the dataset (default 1000), to keep costs predictable.", "default": 1000, "minimum": 1 }, "maxPagesPerSite": { "title": "Max pages per website", "type": "integer", "description": "Homepage plus contact, about, imprint and careers pages. More pages find more contacts; 4 is a good balance.", "default": 4, "minimum": 1, "maximum": 10 }, "includeTechStack": { "title": "Detect tech stack", "type": "boolean", "description": "Detect 130+ technologies (CMS, ecommerce, analytics, marketing automation, ad pixels, chat, frameworks, CDN, payments, consent).", "default": true }, "maxConcurrency": { "title": "Parallel websites", "type": "integer", "description": "How many websites to process at the same time.", "default": 10, "minimum": 1, "maximum": 50, "sectionCaption": "Advanced" }, "requestTimeoutSecs": { "title": "Request timeout (seconds)", "type": "integer", "default": 20, "minimum": 5, "maximum": 120, "description": "Seconds to wait for each page before giving up (default 20). Raise it for slow sites." }, "maxRetries": { "title": "Retries per page", "type": "integer", "default": 1, "minimum": 0, "maximum": 5, "description": "How many times to retry a failed page request (default 1)." }, "proxyConfiguration": { "title": "Proxy", "type": "object", "editor": "proxy", "description": "Not needed for most sites. Enable Apify Proxy only if many sites return HTTP 403 (bot protection).", "default": { "useApifyProxy": false } } } }

{ "actorOutputSchemaVersion": 1, "title": "Output of Company Website Enrichment", "description": "One row per website in the default dataset (failed sites included as free rows with status and error); a run summary in the OUTPUT record.", "properties": { "results": { "type": "string", "title": "Companies", "description": "One row per website. Example: {\"domain\":\"stripe.com\",\"companyName\":\"Stripe\",\"status\":\"ok\",\"emails\":[\"sales@stripe.com\"],\"phones\":[\"+1 888-926-2289\"],\"socials\":{\"linkedin\":\"https://www.linkedin.com/company/stripe\"},\"technologies\":[{\"name\":\"Next.js\",\"category\":\"JavaScript framework\"}]}. Failed: {\"input\":\"bad.example\",\"status\":\"unreachable\",\"error\":\"…\"}.", "template": "{{links.apiDefaultDatasetUrl}}/items?view=overview" }, "summary": { "type": "string", "title": "Run summary", "description": "JSON: websitesRequested, enriched, byStatus counts, chargeLimitReached, zeroResultsReason and request stats.", "template": "{{links.apiDefaultKeyValueStoreUrl}}/records/OUTPUT" } } }

{ "name": "company-website-enrichment", "version": "0.1.0", "type": "module", "description": "Enrich company websites with role-based contact emails, phones, address, social profiles, careers page and tech stack.", "main": "src/main.js", "scripts": { "start": "node src/main.js", "test": "node test/local.test.js" }, "dependencies": { "apify": "^3.4.0", "got-scraping": "^4.1.1", "cheerio": "1.0.0-rc.12", "robots-parser": "^3.0.1", "tldts": "^7.0.0" }, "engines": { "node": ">=20" }, "license": "MIT" }

1// Read website values from another Actor's dataset, in pages (never the whole dataset at once). 2// Field names other Actors commonly use for a company's site (e.g. Google Maps scrapers use "website"). 3export const WEBSITE_FIELDS = ['website', 'websiteUrl', 'companyWebsite', 'companyUrl', 'companyDomain', 'domain', 'homepage', 'url', 'loc']; 4// Map links are not company websites (Google Maps results carry both "url" and "website"). 5const NOT_A_COMPANY_SITE = /(google\.[a-z.]+\/maps|maps\.google\.|goo\.gl\/maps|maps\.app\.goo\.gl)/i; 6const getPath = (o, path) => path.split('.').reduce((v, k) => (v && typeof v === 'object' ? v[k] : undefined), o); 7export async function websitesFromDataset(client, id, field, maxItems) { 8 const out = []; let offset = 0; let scanned = 0; let chosen = field || null; 9 while (out.length < maxItems) { 10 let page; 11 try { page = await client.dataset(id).listItems({ offset, limit: 1000, clean: true }); } 12 catch (e) { const err = new Error(`Could not read dataset ${id}: ${e.message}. Use the dataset ID of a run you can access.`); err.isInputError = true; throw err; } 13 const items = page.items ?? []; 14 if (!items.length) break; 15 // Pick the highest-priority field present anywhere on the first page, so a first item without "website" does not select "url". 16 if (!chosen) chosen = WEBSITE_FIELDS.find((f) => items.some((it) => typeof getPath(it, f) === 'string' && getPath(it, f).trim())) ?? null; 17 for (const it of items) { 18 scanned++; 19 const v = chosen ? getPath(it, chosen) : null; 20 if (typeof v === 'string' && v.trim() && !NOT_A_COMPANY_SITE.test(v)) out.push(v.trim()); 21 if (out.length >= maxItems) break; 22 } 23 offset += items.length; 24 if (items.length < 1000) break; 25 } 26 return { websites: out, field: chosen, itemsScanned: scanned }; 27}

1import * as cheerio from 'cheerio'; 2import { getDomain } from 'tldts'; 3 4// Only generic, role-based mailboxes are kept. Personal addresses (e.g. jane.doe@) are never output. 5export const ROLE_EMAIL = /^(?:info|information|hello|hi|hey|contact|contactus|contact-us|kontakt|contacto|contato|sales|vertrieb|ventas|support|help|helpdesk|service|services|customerservice|customer-service|customercare|care|office|admin|press|media|pr|news|newsroom|marketing|partners?|partnerships|alliances|careers?|jobs|recruiting|recruitment|talent|hr|billing|accounts?|accounting|finance|invoices?|enquir(?:y|ies)|inquir(?:y|ies)|anfrage|team|business|bd|biz|legal|privacy|dpo|gdpr|compliance|security|abuse|webmaster|feedback|orders?|shop|store|booking|bookings|reservations?|investors?|ir|general|mail|post|welcome|bonjour|hola|ciao|studio|events|community|developers?|api|noreply|no-reply)(?:[._+-][a-z0-9]+)*$/i; 6const EMAIL_RE = /[a-z0-9][a-z0-9._%+-]{0,63}@[a-z0-9-]+(?:\.[a-z0-9-]+)*\.[a-z]{2,24}/gi; 7 8export const SOCIAL = [ 9 ['linkedin', /^https?:\/\/(?:[a-z]{2,3}\.)?linkedin\.com\/(?:company|school|showcase)\/[^/?#]+/i], 10 ['twitter', /^https?:\/\/(?:www\.|mobile\.)?(?:twitter|x)\.com\/(?!intent|share|home|search|hashtag|i\/)[A-Za-z0-9_]{1,15}\/?(?:[?#].*)?$/i], 11 ['facebook', /^https?:\/\/(?:www\.|m\.|[a-z]{2}-[a-z]{2}\.)?facebook\.com\/(?!sharer|share|dialog|plugins|tr\b|login|groups\/?$)[^?#]+/i], 12 ['instagram', /^https?:\/\/(?:www\.)?instagram\.com\/(?!p\/|reel\/|explore\/|stories\/)[A-Za-z0-9_.]+\/?(?:[?#].*)?$/i], 13 ['youtube', /^https?:\/\/(?:www\.|m\.)?youtube\.com\/(?:@[^/?#]+|channel\/[^/?#]+|c\/[^/?#]+|user\/[^/?#]+)/i], 14 ['tiktok', /^https?:\/\/(?:www\.)?tiktok\.com\/@[^/?#]+/i], 15 ['github', /^https?:\/\/(?:www\.)?github\.com\/(?!features|about|pricing|login|sponsors|marketplace|topics|orgs\/?$)[A-Za-z0-9-]+\/?(?:[?#].*)?$/i], 16 ['pinterest', /^https?:\/\/(?:[a-z]{2,3}\.)?pinterest\.[a-z.]+\/(?!pin\/)[A-Za-z0-9_]+\/?(?:[?#].*)?$/i], 17]; 18 19const PAGE_TYPES = [ 20 ['contact', /contact|kontakt|contacto|contato|contatti|get-in-touch|reach-us|talk-to|support\/contact/i, /^(contact|contact us|kontakt|contacto|contato|get in touch|talk to (?:us|sales))$/i], 21 ['about', /\/(?:about|about-us|company|who-we-are|our-story|ueber-uns|uber-uns|a-propos|quienes-somos|chi-siamo|over-ons)(?:\/|$|\.)/i, /^(about|about us|company|who we are|our story|über uns|a propos|quiénes somos)$/i], 22 ['legal', /\/(?:impressum|imprint|legal-notice|mentions-legales|aviso-legal|colofon|legal)(?:\/|$|\.)/i, /^(impressum|imprint|legal notice|mentions légales|aviso legal)$/i], 23 ['careers', /\/(?:careers?|jobs|join-us|work-with-us|karriere|vacatures|empleo|stellenangebote)(?:\/|$|\.)/i, /^(careers|jobs|join us|we're hiring|work with us|karriere)$/i], 24]; 25 26const ATS_LINKS = [ 27 ['greenhouse', /https?:\/\/(?:boards|job-boards)(?:\.eu)?\.greenhouse\.io\/(?:embed\/job_board\?for=)?([A-Za-z0-9_-]+)/i], 28 ['lever', /https?:\/\/jobs\.(?:eu\.)?lever\.co\/([A-Za-z0-9_.-]+)/i], 29 ['ashby', /https?:\/\/jobs\.ashbyhq\.com\/([A-Za-z0-9_.%-]+)/i], 30 ['workable', /https?:\/\/apply\.workable\.com\/([A-Za-z0-9_-]+)/i], 31 ['recruitee', /https?:\/\/([A-Za-z0-9-]+)\.recruitee\.com/i], 32 ['smartrecruiters', /https?:\/\/(?:jobs|careers)\.smartrecruiters\.com\/([A-Za-z0-9_-]+)/i], 33 ['workday', /https?:\/\/([a-z0-9-]+)\.wd\d+\.myworkdayjobs\.com/i], 34 ['bamboohr', /https?:\/\/([a-z0-9-]+)\.bamboohr\.com\/(?:careers|jobs)/i], 35 ['personio', /https?:\/\/([a-z0-9-]+)\.jobs\.personio\.(?:de|com)/i], 36 ['teamtailor', /https?:\/\/([a-z0-9-]+)\.teamtailor\.com/i], 37]; 38 39const ORG_TYPE = /Organization|Corporation|Business|Store|Company|Agency|Restaurant|Hotel|Clinic|Firm|Office|NGO|School|University/i; 40const clean = (s, max = 500) => { if (s == null) return null; const t = String(s).replace(/\s+/g, ' ').trim(); return t ? t.slice(0, max) : null; }; 41 42export function domainOf(host) { return getDomain(host, { allowPrivateDomains: false }) || host.replace(/^www\./, ''); } 43 44function flattenLd(node, out) { 45 if (!node || typeof node !== 'object') return; 46 if (Array.isArray(node)) { node.forEach((n) => flattenLd(n, out)); return; } 47 out.push(node); 48 if (node['@graph']) flattenLd(node['@graph'], out); 49 for (const k of ['publisher', 'author', 'provider', 'brand', 'parentOrganization', 'mainEntity', 'about']) if (node[k] && typeof node[k] === 'object') flattenLd(node[k], out); 50} 51const typeOf = (n) => [].concat(n['@type'] ?? []).join(' '); 52const str = (v) => (Array.isArray(v) ? v[0] : v); 53const urlOf = (v) => { v = str(v); if (!v) return null; if (typeof v === 'string') return v; return v.url ?? v.contentUrl ?? v['@id'] ?? null; }; 54 55export function parseLd($) { 56 const nodes = []; 57 $('script[type="application/ld+json"]').each((_, el) => { 58 const raw = $(el).contents().text().trim(); if (!raw) return; 59 try { flattenLd(JSON.parse(raw), nodes); } catch { try { flattenLd(JSON.parse(raw.replace(/[\u0000-\u001f]+/g, ' ')), nodes); } catch { /* ignore bad JSON-LD */ } } 60 }); 61 const orgs = nodes.filter((n) => ORG_TYPE.test(typeOf(n)) && !/Person/i.test(typeOf(n))); 62 if (!orgs.length) return null; 63 orgs.sort((a, b) => score(b) - score(a)); 64 const o = orgs[0]; 65 const addrNode = [].concat(o.address ?? []).find((a) => a && typeof a === 'object') ?? null; 66 const address = addrNode ? { 67 street: clean(addrNode.streetAddress, 200), city: clean(addrNode.addressLocality, 100), region: clean(addrNode.addressRegion, 100), 68 postalCode: clean(addrNode.postalCode, 30), country: clean(typeof addrNode.addressCountry === 'object' ? addrNode.addressCountry?.name : addrNode.addressCountry, 60), 69 } : (typeof o.address === 'string' ? { full: clean(o.address, 300) } : null); 70 return { 71 name: clean(str(o.name), 200), legalName: clean(str(o.legalName), 200), description: clean(str(o.description), 1000), 72 logo: urlOf(o.logo) ?? urlOf(o.image), telephone: [].concat(o.telephone ?? []).map(String), email: [].concat(o.email ?? []).map(String), 73 address, sameAs: [].concat(o.sameAs ?? []).filter((x) => typeof x === 'string'), foundingDate: clean(str(o.foundingDate), 30), 74 numberOfEmployees: clean(typeof o.numberOfEmployees === 'object' ? (o.numberOfEmployees.value ?? [o.numberOfEmployees.minValue, o.numberOfEmployees.maxValue].filter(Boolean).join('-')) : o.numberOfEmployees, 40), 75 type: typeOf(o), 76 }; 77 function score(n) { return ['address', 'logo', 'sameAs', 'telephone', 'legalName', 'url'].filter((k) => n[k]).length + (/^Organization$|Corporation/.test(typeOf(n)) ? 1 : 0); } 78} 79 80export function normPhone(p) { 81 let s = decodeURIComponent(String(p)).replace(/^tel:|^callto:/i, '').split(/[;,?]/)[0].trim(); 82 const digits = s.replace(/\D/g, ''); 83 if (digits.length < 7 || digits.length > 15) return null; 84 return (s.trim().startsWith('+') ? '+' : '') + s.replace(/^\+/, '').replace(/[^\d()\s.-]/g, '').replace(/\s+/g, ' ').trim(); 85} 86 87export function cleanEmail(e, siteDomains) { 88 e = String(e).replace(/^mailto:/i, '').split('?')[0].trim().toLowerCase(); 89 try { e = decodeURIComponent(e); } catch { /* keep */ } 90 const m = e.match(/^([a-z0-9._%+-]+)@([a-z0-9.-]+\.[a-z]{2,24})$/i); if (!m) return null; 91 const [, local, host] = m; 92 if (!siteDomains.has(domainOf(host))) return null; // only the company's own domain 93 if (!ROLE_EMAIL.test(local)) return null; // only role-based mailboxes, never personal ones 94 if (/^(?:no-?reply)/.test(local)) return null; 95 return `${local}@${host}`; 96} 97 98export function canonicalSocial(url) { 99 for (const [net, re] of SOCIAL) { 100 if (re.test(url)) { 101 let u = url.split('#')[0].split('?')[0].replace(/\/+$/, ''); 102 u = u.replace(/^http:/i, 'https:').replace(/\/\/(?:mobile|m)\./i, '//www.').replace(/\/\/twitter\.com/i, '//x.com').replace(/\/\/www\.twitter\.com/i, '//x.com'); 103 const uu = new URL(u); 104 if (['twitter', 'github'].includes(net)) uu.hostname = uu.hostname.replace(/^www\./, ''); 105 else if (!/^www\./.test(uu.hostname) && uu.hostname.split('.').length === 2) uu.hostname = `www.${uu.hostname}`; 106 u = uu.href.replace(/\/+$/, ''); 107 if (net === 'facebook' && /facebook\.com\/(?:pages\/category|profile\.php)/i.test(u) && !/id=/.test(url)) return null; 108 return [net, u]; 109 } 110 } 111 return null; 112} 113 114/** Extract everything from one HTML page. */ 115export function extractPage(html, pageUrl, siteDomains) { 116 const $ = cheerio.load(html); 117 const base = new URL(pageUrl); 118 const links = []; const emails = new Set(); const phones = new Map(); const socials = {}; const pages = {}; let jobBoard = null; 119 $('a[href]').each((_, el) => { 120 const href = ($(el).attr('href') || '').trim(); if (!href) return; 121 if (/^mailto:/i.test(href)) { const e = cleanEmail(href, siteDomains); if (e) emails.add(e); return; } 122 if (/^(?:tel|callto):/i.test(href)) { const p = normPhone(href); if (p) phones.set(p.replace(/\D/g, ''), p); return; } 123 let abs; try { abs = new URL(href, base); } catch { return; } 124 if (!/^https?:$/.test(abs.protocol)) return; 125 const s = abs.href; 126 const soc = canonicalSocial(s); if (soc && !socials[soc[0]]) socials[soc[0]] = soc[1]; 127 if (!jobBoard) for (const [ats, re] of ATS_LINKS) { const m = s.match(re); if (m) { jobBoard = { ats, slug: decodeURIComponent(m[1]), url: s.split('#')[0] }; break; } } 128 if (siteDomains.has(domainOf(abs.hostname))) { 129 const text = clean($(el).text(), 80) ?? ''; 130 const path = abs.pathname.toLowerCase(); 131 for (const [type, pathRe, textRe] of PAGE_TYPES) { 132 if (pathRe.test(path) || textRe.test(text)) { 133 const cand = abs.origin + abs.pathname; 134 if (cand.replace(/\/$/, '') === (base.origin + base.pathname).replace(/\/$/, '') && type !== 'careers') continue; 135 const prev = pages[type]; const len = abs.pathname.length + (textRe.test(text) ? 0 : 5); 136 if (!prev || len < prev.len) pages[type] = { url: cand, len }; 137 } 138 } 139 links.push(s); 140 } 141 }); 142 const ld = parseLd($); 143 if (ld) { 144 ld.email.forEach((e) => { const c = cleanEmail(e, siteDomains); if (c) emails.add(c); }); 145 ld.telephone.forEach((t) => { const p = normPhone(t); if (p) phones.set(p.replace(/\D/g, ''), p); }); 146 ld.sameAs.forEach((u) => { const soc = canonicalSocial(u); if (soc && !socials[soc[0]]) socials[soc[0]] = soc[1]; }); 147 } 148 // Plain-text emails (visible text only, scripts/styles removed) 149 $('script,style,noscript,svg').remove(); 150 const text = $('body').text().replace(/\s+/g, ' '); 151 for (const m of text.matchAll(EMAIL_RE)) { const e = cleanEmail(m[0], siteDomains); if (e) emails.add(e); } 152 const meta = (sel) => clean($(sel).attr('content'), 1000); 153 const abs = (u) => { try { return u ? new URL(u, base).href : null; } catch { return null; } }; 154 return { 155 title: clean($('title').first().text(), 300), 156 siteName: meta('meta[property="og:site_name"]'), 157 description: meta('meta[name="description"]') ?? meta('meta[property="og:description"]'), 158 ogImage: abs(meta('meta[property="og:image"]')), 159 icon: abs($('link[rel="apple-touch-icon"]').attr('href') || $('link[rel~="icon"]').attr('href')), 160 language: clean($('html').attr('lang'), 20), 161 ld, emails: [...emails], phones: [...phones.values()], socials, pages: Object.fromEntries(Object.entries(pages).map(([k, v]) => [k, v.url])), jobBoard, 162 }; 163} 164 165export function companyNameFrom(home, domain) { 166 const label = domain.split('.')[0].toLowerCase().replace(/[^a-z0-9]/g, ''); 167 const norm = (x) => x.toLowerCase().replace(/[^a-z0-9]/g, ''); 168 const titleParts = (home.title ?? '').split(/\s+[|–—·•:-]\s+/).map((p) => p.trim()).filter(Boolean); 169 const short = (x) => x && x.length <= 40 && x.split(/\s+/).length <= 5; 170 const cands = [home.ld?.name, home.ld?.legalName, home.siteName, ...titleParts].filter(Boolean); 171 // 1) a candidate that contains the domain label (e.g. "Hootsuite" for hootsuite.com) 172 const hit = cands.find((c) => short(c) && norm(c).includes(label)) ?? cands.find((c) => short(c) && label.includes(norm(c)) && norm(c).length >= 3); 173 if (hit) return hit; 174 // 2) structured names, if they look like names rather than slogans 175 for (const c of [home.ld?.name, home.siteName]) if (short(c) && !/sign up|log ?in|welcome|home ?page|free/i.test(c)) return c; 176 // 3) fall back to the domain label, capitalized 177 return label ? label.charAt(0).toUpperCase() + label.slice(1) : null; 178} 179 180export function normalizeWebsite(input) { 181 let s = String(input ?? '').trim(); if (!s) return null; 182 if (/@/.test(s) && !/^https?:/i.test(s)) s = s.split('@').pop(); 183 if (!/^https?:\/\//i.test(s)) s = `https://${s}`; 184 let u; try { u = new URL(s); } catch { return null; } 185 if (!/\./.test(u.hostname) || /^\d+\.\d+\.\d+\.\d+$/.test(u.hostname) || u.hostname === 'localhost') return null; 186 return { url: `${u.protocol}//${u.hostname}/`, host: u.hostname.toLowerCase(), domain: domainOf(u.hostname.toLowerCase()) }; 187}

1import { Actor, log } from 'apify'; 2import { websitesFromDataset, WEBSITE_FIELDS } from './dataset.js'; 3import { gotScraping } from 'got-scraping'; 4import robotsParser from 'robots-parser'; 5import { extractPage, companyNameFrom, normalizeWebsite, domainOf } from './extract.js'; 6import { detectTech } from './tech.js'; 7 8const EVENT = 'website'; 9const UA_TOKEN = 'CompanyWebsiteEnrichment'; 10const MAX_BODY = 2 * 1024 * 1024; 11// Shrink HTML before DOM parsing: drop inline script/style/svg bodies (keeps JSON-LD and script src attributes). Big DOMs x concurrency caused OOM at 512 MB. 12const slim = (html) => html.replace(/<script\b(?![^>]*ld\+json)([^>]*)>[\s\S]*?<\/script>/gi, '<script$1></script>').replace(/<style\b[^>]*>[\s\S]*?<\/style>/gi, '').replace(/<svg\b[\s\S]*?<\/svg>/gi, '').replace(//g, ''); 13const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); 14class InputError extends Error {} 15 16 17await Actor.init(); 18try { 19 const input = (await Actor.getInput()) ?? {}; 20 const { 21 websites = [], maxPagesPerSite = 4, includeTechStack = true, 22 maxConcurrency = 10, requestTimeoutSecs = 20, maxRetries = 1, 23 proxyConfiguration = { useApifyProxy: false }, 24 } = input; 25 const raw = (Array.isArray(websites) ? websites : String(websites).split(/[\n,]/)) 26 .map((w) => (typeof w === 'object' && w ? w.url : w)).map((w) => String(w ?? '').trim()).filter(Boolean); 27 // Chain from another Actor: read websites from a dataset (datasetId, or the dataset passed by an Apify integration). 28 const dsId = String(input.datasetId || input?.payload?.resource?.defaultDatasetId || input?.resource?.defaultDatasetId || '').trim(); 29 let fromDataset = null; 30 if (dsId) { 31 fromDataset = await websitesFromDataset(Actor.newClient(), dsId, String(input.websiteField || '').trim(), Math.max(1, Math.floor(Number(input.datasetMaxItems) || 1000))); 32 raw.push(...fromDataset.websites); 33 log.info(`Read ${fromDataset.websites.length} websites from dataset ${dsId} (field "${fromDataset.field ?? 'none found'}", ${fromDataset.itemsScanned} items scanned).`); 34 if (!fromDataset.websites.length && !raw.length) throw new InputError(`Dataset ${dsId} has no website values${fromDataset.field ? ` in field "${fromDataset.field}"` : ''}. Set "websiteField" to the field that holds the URL or domain (tried: ${WEBSITE_FIELDS.join(', ')}).`); 35 } 36 if (!raw.length) throw new InputError('Add at least one website or domain, e.g. "stripe.com" or "https://www.hubspot.com", or pass a datasetId from another Actor run.'); 37 const seen = new Set(); const targets = []; 38 for (const r of raw) { 39 const n = normalizeWebsite(r); 40 const key = n ? n.host.replace(/^www\./, '') : `invalid:${r}`; 41 if (seen.has(key)) continue; seen.add(key); targets.push({ input: r, n }); 42 } 43 const pagesCap = Math.max(1, Math.min(10, Math.floor(Number(maxPagesPerSite) || 4))); 44 const memMb = Number(process.env.ACTOR_MEMORY_MBYTES || process.env.APIFY_MEMORY_MBYTES) || 512; 45 const conc = Math.max(1, Math.min(50, Math.floor(memMb / 64), Math.floor(Number(maxConcurrency) || 10))); 46 const proxy = proxyConfiguration?.useApifyProxy || proxyConfiguration?.proxyUrls?.length ? await Actor.createProxyConfiguration(proxyConfiguration) : null; 47 const charging = Actor.getChargingManager(); 48 const stats = { requests: 0, retries: 0, robotsBlocked: 0 }; 49 const robotsCache = new Map(); 50 let charged = 0; let chargeLimitHit = false; 51 const counts = {}; 52 53 async function get(url, { accept = 'text/html,application/xhtml+xml;q=0.9,*/*;q=0.8' } = {}) { 54 let lastErr; 55 for (let a = 0; a <= maxRetries; a++) { 56 stats.requests++; 57 try { 58 const res = await gotScraping({ url, proxyUrl: proxy ? await proxy.newUrl() : undefined, timeout: { request: Number(requestTimeoutSecs) * 1000 || 20000 }, 59 throwHttpErrors: false, retry: { limit: 0 }, followRedirect: true, maxRedirects: 8, headers: { accept }, 60 headerGeneratorOptions: { browsers: ['chrome'], devices: ['desktop'] }, https: { rejectUnauthorized: false } }); 61 if ((res.statusCode === 429 || res.statusCode >= 500) && a < maxRetries) { stats.retries++; await sleep(1500 * (a + 1)); continue; } 62 return res; 63 } catch (e) { lastErr = e; if (a < maxRetries) { stats.retries++; await sleep(1000 * (a + 1)); } } 64 } 65 throw lastErr; 66 } 67 68 // robots.txt per origin. 4xx = no rules (allow); 5xx/unreachable = treat as disallow (Google's convention). 69 async function robotsFor(origin) { 70 if (robotsCache.has(origin)) return robotsCache.get(origin); 71 const p = (async () => { 72 try { 73 const res = await get(`${origin}/robots.txt`, { accept: 'text/plain,*/*;q=0.8' }); 74 if (res.statusCode >= 500) return { blockAll: true }; 75 if (res.statusCode >= 400) return { robots: robotsParser(`${origin}/robots.txt`, '') }; 76 const body = /html/i.test(String(res.headers['content-type'])) && /<html/i.test(res.body.slice(0, 500)) ? '' : res.body.slice(0, 500000); 77 return { robots: robotsParser(`${origin}/robots.txt`, body) }; 78 } catch { return { unreachable: true }; } 79 })(); 80 robotsCache.set(origin, p); return p; 81 } 82 async function allowed(url) { 83 const r = await robotsFor(new URL(url).origin); 84 if (r.unreachable) return 'unreachable'; 85 if (r.blockAll) return false; 86 return r.robots.isAllowed(url, UA_TOKEN) !== false; 87 } 88 async function crawlDelay(url) { const r = await robotsFor(new URL(url).origin); return Math.min(10, Number(r.robots?.getCrawlDelay(UA_TOKEN)) || 0) * 1000; } 89 90 async function fetchPage(url) { 91 const ok = await allowed(url); 92 if (ok === 'unreachable') return { error: 'unreachable' }; 93 if (!ok) { stats.robotsBlocked++; return { robotsBlocked: true }; } 94 const res = await get(url); 95 const finalUrl = res.url || url; 96 if (finalUrl !== url && new URL(finalUrl).origin !== new URL(url).origin) { 97 const ok2 = await allowed(finalUrl); 98 if (ok2 !== true) { stats.robotsBlocked++; return { robotsBlocked: true, finalUrl }; } 99 } 100 const ct = String(res.headers['content-type'] || ''); 101 let body = typeof res.body === 'string' ? res.body : String(res.body ?? ''); 102 if (body.length > MAX_BODY) body = body.slice(0, MAX_BODY); 103 const headerText = Object.entries(res.headers).map(([k, v]) => `${k}: ${[].concat(v).join(', ')}`).join('\n'); 104 return { status: res.statusCode, finalUrl, html: /html|xml/i.test(ct) || /<html/i.test(body.slice(0, 2000)) ? body : null, headerText }; 105 } 106 107 async function enrich(t) { 108 const now = new Date().toISOString(); 109 const row = { input: t.input, domain: t.n?.domain ?? null, url: null, status: 'ok', httpStatus: null, companyName: null, legalName: null, description: null, 110 language: null, logoUrl: null, emails: [], phones: [], address: null, socials: {}, contactPageUrl: null, aboutPageUrl: null, careersPageUrl: null, legalPageUrl: null, 111 jobBoard: null, technologies: includeTechStack ? [] : undefined, foundingDate: null, numberOfEmployees: null, pagesFetched: [], error: null, scrapedAt: now }; 112 if (!t.n) { row.status = 'invalid_input'; row.error = 'Not a valid website or domain'; return row; } 113 let home; 114 const tries = [t.n.url]; 115 if (!/^www\./.test(t.n.host)) tries.push(`https://www.${t.n.host}/`); 116 tries.push(t.n.url.replace(/^https:/, 'http:')); 117 for (const u of tries) { 118 try { 119 home = await fetchPage(u); 120 if (home.error === 'unreachable') { home = null; continue; } 121 break; 122 } catch (e) { row.error = e.code || e.message; home = null; } 123 } 124 if (!home) { row.status = 'unreachable'; row.error ??= 'Website could not be reached'; return row; } 125 if (home.robotsBlocked) { row.status = 'blocked_by_robots_txt'; row.error = 'The site\'s robots.txt does not allow automated access to the homepage'; return row; } 126 row.httpStatus = home.status; row.url = home.finalUrl; 127 if (home.status >= 400) { row.status = 'http_error'; row.error = `Homepage returned HTTP ${home.status}${home.status === 403 ? ' (bot protection; try enabling a proxy)' : ''}`; return row; } 128 if (!home.html) { row.status = 'not_html'; row.error = 'Homepage is not an HTML page'; return row; } 129 130 const finalHost = new URL(home.finalUrl).hostname.toLowerCase(); 131 const siteDomains = new Set([t.n.domain, domainOf(finalHost)]); 132 row.domain = domainOf(finalHost); 133 const tech = includeTechStack ? detectTech(home.html, home.headerText) : null; 134 const h = extractPage(slim(home.html), home.finalUrl, siteDomains); 135 home.html = null; 136 row.pagesFetched.push(home.finalUrl); 137 const emails = new Set(h.emails); const phones = new Map(h.phones.map((p) => [p.replace(/\D/g, ''), p])); const socials = { ...h.socials }; 138 let jobBoard = h.jobBoard; let ld = h.ld; 139 const pages = { ...h.pages }; 140 const queue = ['contact', 'about', 'legal', 'careers'].filter((k) => pages[k]).map((k) => [k, pages[k]]); 141 const delay = Math.max(300, await crawlDelay(home.finalUrl)); 142 for (const [kind, url] of queue) { 143 if (row.pagesFetched.length >= pagesCap) break; 144 if (kind === 'careers' && jobBoard) continue; 145 await sleep(delay); 146 try { 147 const p = await fetchPage(url); 148 if (!p.html || p.status >= 400) continue; 149 row.pagesFetched.push(p.finalUrl); 150 const x = extractPage(slim(p.html), p.finalUrl, siteDomains); 151 x.emails.forEach((e) => emails.add(e)); x.phones.forEach((ph) => { const k = ph.replace(/\D/g, ''); if (!phones.has(k)) phones.set(k, ph); }); 152 for (const [k, v] of Object.entries(x.socials)) socials[k] ??= v; 153 jobBoard ??= x.jobBoard; ld ??= x.ld; 154 for (const [k, v] of Object.entries(x.pages)) pages[k] ??= v; 155 } catch { /* subpage failures are non-fatal */ } 156 } 157 row.companyName = companyNameFrom({ ...h, ld }, row.domain); 158 row.legalName = ld?.legalName ?? null; 159 row.description = h.description ?? ld?.description ?? null; 160 row.language = h.language; row.logoUrl = ld?.logo ? new URL(ld.logo, home.finalUrl).href : (h.icon ?? h.ogImage); 161 row.emails = [...emails].slice(0, 25); row.phones = [...phones.values()].slice(0, 10); 162 row.address = ld?.address && Object.values(ld.address).some(Boolean) ? ld.address : null; 163 row.socials = socials; row.contactPageUrl = pages.contact ?? null; row.aboutPageUrl = pages.about ?? null; 164 row.careersPageUrl = pages.careers ?? null; row.legalPageUrl = pages.legal ?? null; row.jobBoard = jobBoard ?? null; 165 row.foundingDate = ld?.foundingDate ?? null; row.numberOfEmployees = ld?.numberOfEmployees ?? null; 166 if (includeTechStack) row.technologies = tech; 167 return row; 168 } 169 170 async function save(row) { 171 counts[row.status] = (counts[row.status] ?? 0) + 1; 172 if (row.status !== 'ok') { await Actor.pushData(row); return; } // failed sites are free 173 if (chargeLimitHit) { counts.ok--; skipped.push(row.input); return; } 174 if (charging) { 175 const allowedCount = charging.calculateMaxEventChargeCountWithinLimit(EVENT); 176 if (Number.isFinite(allowedCount) && allowedCount < 1) { chargeLimitHit = true; counts.ok--; skipped.push(row.input); log.warning('Reached the maximum cost you set for this run. Stopping; results so far are saved.'); return; } 177 } 178 const r = await Actor.pushData(row, EVENT); charged++; 179 if (r?.eventChargeLimitReached) chargeLimitHit = true; 180 } 181 182 let idx = 0; let done = 0; const skipped = []; 183 const worker = async () => { 184 while (idx < targets.length && !chargeLimitHit) { 185 const t = targets[idx++]; 186 let row; 187 try { row = await enrich(t); } catch (e) { row = { input: t.input, domain: t.n?.domain ?? null, status: 'error', error: e.message, scrapedAt: new Date().toISOString() }; } 188 await save(row); 189 if (++done % 25 === 0) log.info(`Processed ${done}/${targets.length} websites`); 190 } 191 }; 192 await Promise.all(Array.from({ length: Math.min(conc, targets.length) }, worker)); 193 194 // Never silent: websites left out because the run hit your maximum cost get a free status row each. 195 for (const t of targets.slice(idx)) skipped.push(t.input); 196 for (const inp of skipped) await Actor.pushData({ input: inp, domain: null, status: 'skipped_charge_limit', error: 'Not processed: the run reached the maximum cost you set. Raise the limit and run these again.', scrapedAt: new Date().toISOString() }); 197 if (skipped.length) counts.skipped_charge_limit = skipped.length; 198 const summary = { websitesRequested: targets.length, enriched: counts.ok ?? 0, byStatus: counts, chargeLimitReached: chargeLimitHit, sourceDatasetId: dsId || null, 199 zeroResultsReason: counts.ok ? null : `No website could be enriched: ${Object.entries(counts).map(([k, v]) => `${v} ${k}`).join(', ')}. See the error field of each row.`, stats, finishedAt: new Date().toISOString() }; 200 if (counts.ok) summary.nextSteps = [`Export to Google Sheets: run swillz/google-sheets-import-export with {"spreadsheet": "<your sheet URL>", "datasetId": "${Actor.getEnv().defaultDatasetId}"}`]; 201 await Actor.setValue('OUTPUT', summary); 202 const valid = targets.filter((t) => t.n).length; 203 // Silent-failure guard: many valid sites and none reachable means our network is broken, not the sites. 204 if (valid >= 3 && !(counts.ok) && (counts.unreachable ?? 0) === valid) throw new Error(`None of the ${valid} websites could be reached. This looks like a network problem; try again or enable a proxy.`); 205 await Actor.exit(`${counts.ok ? '' : '0 results: '}Enriched ${counts.ok ?? 0} of ${targets.length} websites${counts.ok === targets.length ? '' : ` (${Object.entries(counts).filter(([k]) => k !== 'ok').map(([k, v]) => `${v} ${k}`).join(', ')}; not charged)`}.`); 206} catch (e) { 207 if (e instanceof InputError || e?.isInputError) { log.error(e.message); await Actor.fail(`Input error: ${e.message}`); } 208 else { log.exception(e, 'Run failed'); await Actor.fail(e.message); } 209}

1// Signature-based technology detection. h = HTML regex, hd = response-header regex (headers serialized as "name: value" lines). 2const S = (n, c, h, hd) => ({ n, c, h, hd }); 3export const SIGNATURES = [ 4 // CMS & site builders 5 S('WordPress', 'CMS', /\/wp-content\/|\/wp-includes\/|wp-json/i, /link:.*wp-json|x-powered-by:.*wordpress/i), 6 S('Drupal', 'CMS', /drupal-settings-json|\/sites\/default\/files\/|Drupal\.settings/i, /x-drupal|x-generator: drupal/i), 7 S('Joomla', 'CMS', /\/media\/jui\/|content="Joomla/i), 8 S('Webflow', 'CMS', /data-wf-site|webflow\.js|assets\.website-files\.com|cdn\.prod\.website-files\.com/i), 9 S('Wix', 'CMS', /static\.wixstatic\.com|_wixCIDX|wix-bolt/i, /x-wix-request-id/i), 10 S('Squarespace', 'CMS', /static1\.squarespace\.com|Squarespace\.Constants/i), 11 S('Ghost', 'CMS', /content="Ghost \d|ghost-(?:portal|sdk)/i), 12 S('HubSpot CMS', 'CMS', /\.hubspotusercontent(?:-na1)?\.net|hs-sites\.com/i, /x-hs-hub-id|x-powered-by: hubspot/i), 13 S('Framer', 'CMS', /framerusercontent\.com|data-framer-/i), 14 S('Contentful', 'CMS', /ctfassets\.net/i), 15 S('Sanity', 'CMS', /cdn\.sanity\.io/i), 16 S('Weebly', 'CMS', /editmysite\.com|weebly\.com\/weebly/i), 17 S('GoDaddy Website Builder', 'CMS', /img1\.wsimg\.com\/(?:isteam|blobby)/i), 18 S('TYPO3', 'CMS', /typo3(?:conf|temp)\//i), 19 S('Adobe Experience Manager', 'CMS', /\/etc\.clientlibs\/|\/content\/dam\//i), 20 S('Storyblok', 'CMS', /a\.storyblok\.com/i), 21 // Ecommerce 22 S('Shopify', 'Ecommerce', /cdn\.shopify\.com|Shopify\.theme/i, /x-shopify|x-shopid/i), 23 S('WooCommerce', 'Ecommerce', /wp-content\/plugins\/woocommerce|woocommerce-(?:no-js|page|js)|wc-ajax=|wc_add_to_cart_params/i), 24 S('Magento', 'Ecommerce', /Mage\.Cookies|mage\/cookies|\/static\/version\d+\/frontend\//i), 25 S('BigCommerce', 'Ecommerce', /cdn\d*\.bigcommerce\.com/i), 26 S('PrestaShop', 'Ecommerce', /var prestashop\s*=|prestashop\.(?:static_token|urls)|content="PrestaShop/i), 27 S('Salesforce Commerce Cloud', 'Ecommerce', /demandware\.static|\/on\/demandware\.store/i), 28 S('Ecwid', 'Ecommerce', /app\.ecwid\.com/i), 29 // Analytics 30 S('Google Analytics', 'Analytics', /google-analytics\.com\/(?:analytics|ga)\.js|gtag\/js\?id=(?:G|UA)-/i), 31 S('Google Tag Manager', 'Tag manager', /googletagmanager\.com\/gtm\.js|GTM-[A-Z0-9]{5,8}\b/), 32 S('Adobe Analytics', 'Analytics', /omtrdc\.net|AppMeasurement|s_code\.js/i), 33 S('Segment', 'Analytics', /cdn\.segment\.com|analytics\.load\(/i), 34 S('Mixpanel', 'Analytics', /cdn\.mxpnl\.com|mixpanel\.init/i), 35 S('Amplitude', 'Analytics', /cdn\.amplitude\.com|amplitude\.getInstance|@amplitude/i), 36 S('Heap', 'Analytics', /heapanalytics\.com|heap\.load\(/i), 37 S('Hotjar', 'Analytics', /static\.hotjar\.com|hotjar\.com\/c\/hotjar/i), 38 S('Microsoft Clarity', 'Analytics', /clarity\.ms\/tag/i), 39 S('Plausible', 'Analytics', /plausible\.io\/js/i), 40 S('Matomo', 'Analytics', /matomo\.js|piwik\.js|_paq\.push/i), 41 S('FullStory', 'Analytics', /fullstory\.com\/s\/fs\.js|edge\.fullstory\.com/i), 42 S('PostHog', 'Analytics', /posthog\.com\/static|posthog\.init|i\.posthog\.com/i), 43 S('Yandex Metrica', 'Analytics', /mc\.yandex\.ru\/metrika/i), 44 S('Fathom', 'Analytics', /cdn\.usefathom\.com/i), 45 S('Pendo', 'Analytics', /cdn\.pendo\.io|pendo\.initialize/i), 46 // Marketing automation, CRM, sales tools 47 S('HubSpot', 'Marketing automation', /js\.hs-scripts\.com|js\.hsforms\.net|js\.hs-analytics\.net|_hsq\.push|js\.hs-banner\.com/i), 48 S('Marketo', 'Marketing automation', /munchkin\.marketo\.net|mktoForms2|marketo\.com\/js/i), 49 S('Pardot', 'Marketing automation', /pi\.pardot\.com|pardot\.com\/pd\.js|piAId/i), 50 S('Salesforce', 'CRM', /\.my\.salesforce\.com|service\.force\.com|servlet\.WebToLead|webto(?:lead|case)/i), 51 S('Mailchimp', 'Email marketing', /chimpstatic\.com|list-manage\.com/i), 52 S('Klaviyo', 'Email marketing', /static\.klaviyo\.com|klaviyo\.com\/onsite/i), 53 S('ActiveCampaign', 'Marketing automation', /trackcmp\.net|activehosted\.com/i), 54 S('Brevo', 'Email marketing', /sibforms\.com|sendinblue\.com|brevo\.com\/js/i), 55 S('6sense', 'Account-based marketing', /j\.6sc\.co|6sc\.co\/j\//i), 56 S('Clearbit', 'Account-based marketing', /tag\.clearbitscripts\.com|x\.clearbitjs\.com/i), 57 S('ZoomInfo', 'Account-based marketing', /ws\.zoominfo\.com|js\.zi-scripts\.com/i), 58 S('Demandbase', 'Account-based marketing', /tag\.demandbase\.com|scripts\.demandbase\.com/i), 59 S('Calendly', 'Scheduling', /assets\.calendly\.com|calendly\.com\/assets/i), 60 S('Chili Piper', 'Scheduling', /js\.chilipiper\.com/i), 61 S('Typeform', 'Forms', /embed\.typeform\.com/i), 62 S('Qualified', 'Live chat', /js\.qualified\.com/i), 63 S('Unbounce', 'Landing pages', /unbounce\.com|ubembed\.com/i), 64 // A/B testing 65 S('Optimizely', 'A/B testing', /cdn\.optimizely\.com/i), 66 S('VWO', 'A/B testing', /dev\.visualwebsiteoptimizer\.com/i), 67 S('AB Tasty', 'A/B testing', /try\.abtasty\.com/i), 68 // Advertising pixels 69 S('Meta Pixel', 'Advertising', /connect\.facebook\.net\/[^"']*fbevents\.js|fbq\(\s*['"]init/i), 70 S('LinkedIn Insight Tag', 'Advertising', /snap\.licdn\.com\/li\.lms-analytics|_linkedin_partner_id/i), 71 S('Google Ads', 'Advertising', /googleadservices\.com\/pagead\/conversion|gtag\/js\?id=AW-|googleads\.g\.doubleclick\.net/i), 72 S('X (Twitter) Pixel', 'Advertising', /static\.ads-twitter\.com|twq\(\s*['"]init/i), 73 S('TikTok Pixel', 'Advertising', /analytics\.tiktok\.com/i), 74 S('Pinterest Tag', 'Advertising', /s\.pinimg\.com\/ct\/core\.js|pintrk\(/i), 75 S('Microsoft Advertising', 'Advertising', /bat\.bing\.com/i), 76 S('Reddit Pixel', 'Advertising', /redditstatic\.com\/ads|rdt\(\s*['"]init/i), 77 S('Criteo', 'Advertising', /static\.criteo\.net/i), 78 S('Taboola', 'Advertising', /cdn\.taboola\.com/i), 79 // Live chat & support 80 S('Intercom', 'Live chat', /widget\.intercom\.io|js\.intercomcdn\.com|intercomSettings/i), 81 S('Drift', 'Live chat', /js\.driftt\.com|drift\.com\/include/i), 82 S('Zendesk', 'Customer support', /static\.zdassets\.com|zopim|\.zendesk\.com/i), 83 S('Freshworks', 'Customer support', /wchat\.freshchat\.com|freshdesk\.com|fw-cdn\.com/i), 84 S('LiveChat', 'Live chat', /cdn\.livechatinc\.com/i), 85 S('Crisp', 'Live chat', /client\.crisp\.chat/i), 86 S('Tawk.to', 'Live chat', /embed\.tawk\.to/i), 87 S('Tidio', 'Live chat', /code\.tidio\.co/i), 88 S('Gorgias', 'Customer support', /config\.gorgias\.chat/i), 89 S('Help Scout', 'Customer support', /beacon-v2\.helpscout\.net/i), 90 // JavaScript frameworks & libraries 91 S('Next.js', 'JavaScript framework', /__NEXT_DATA__|\/_next\/static\//, /x-powered-by: next\.js/i), 92 S('Nuxt', 'JavaScript framework', /__NUXT__|\/_nuxt\//), 93 S('Gatsby', 'JavaScript framework', /___gatsby|\/page-data\/app-data\.json/), 94 S('Astro', 'JavaScript framework', /astro-island|data-astro-cid/), 95 S('Remix', 'JavaScript framework', /__remixContext/), 96 S('SvelteKit', 'JavaScript framework', /__sveltekit|data-sveltekit/), 97 S('Angular', 'JavaScript framework', /ng-version="/), 98 S('Vue.js', 'JavaScript framework', /data-v-[0-9a-f]{8}\b|vue(?:\.runtime)?(?:\.global)?(?:\.min)?\.js/), 99 S('React', 'JavaScript framework', /data-reactroot|react-dom(?:\.production)?(?:\.min)?\.js|__NEXT_DATA__|___gatsby/), 100 S('jQuery', 'JavaScript library', /jquery(?:[.-]\d[\d.]*)?(?:\.min)?\.js/i), 101 S('Bootstrap', 'UI framework', /bootstrap(?:\.bundle)?(?:\.min)?\.(?:css|js)/i), 102 S('Font Awesome', 'Font', /font-?awesome|kit\.fontawesome\.com/i), 103 S('Google Fonts', 'Font', /fonts\.googleapis\.com/i), 104 S('Adobe Fonts', 'Font', /use\.typekit\.net/i), 105 // Hosting, CDN, servers 106 S('Cloudflare', 'CDN', null, /^server: cloudflare|^cf-ray:/im), 107 S('Fastly', 'CDN', null, /^x-served-by: cache-|^fastly-|^x-fastly/im), 108 S('Akamai', 'CDN', null, /^server: akamai|^x-akamai|akamaighost/im), 109 S('Amazon CloudFront', 'CDN', null, /^x-amz-cf-id|^via:.*cloudfront/im), 110 S('Vercel', 'Hosting', null, /^x-vercel-id|^server: vercel/im), 111 S('Netlify', 'Hosting', null, /^x-nf-request-id|^server: netlify/im), 112 S('Heroku', 'Hosting', null, /^via:.*vegur/im), 113 S('GitHub Pages', 'Hosting', null, /^server: github\.com/im), 114 S('WP Engine', 'Hosting', null, /^x-powered-by: wp engine|^wpe-backend/im), 115 S('Kinsta', 'Hosting', null, /^x-kinsta/im), 116 S('Amazon S3', 'Hosting', null, /^server: amazons3/im), 117 S('Nginx', 'Web server', null, /^server: nginx/im), 118 S('Apache', 'Web server', null, /^server: apache/im), 119 S('Microsoft IIS', 'Web server', null, /^server: microsoft-iis/im), 120 S('LiteSpeed', 'Web server', null, /^server: litespeed/im), 121 S('PHP', 'Programming language', null, /^x-powered-by: php|PHPSESSID/im), 122 S('ASP.NET', 'Web framework', /__VIEWSTATE/, /^x-aspnet-version|^x-powered-by: asp\.net|ASP\.NET_SessionId/im), 123 S('Laravel', 'Web framework', null, /laravel_session/i), 124 // Payments 125 S('Stripe', 'Payments', /js\.stripe\.com/i), 126 S('PayPal', 'Payments', /paypal\.com\/sdk\/js|paypalobjects\.com/i), 127 S('Klarna', 'Payments', /klarnaservices\.com|js\.klarna\.com/i), 128 S('Afterpay', 'Payments', /afterpay\.com|static\.afterpay/i), 129 S('Braintree', 'Payments', /js\.braintreegateway\.com/i), 130 // Cookie consent 131 S('OneTrust', 'Cookie consent', /cdn\.cookielaw\.org|optanon/i), 132 S('Cookiebot', 'Cookie consent', /consent\.cookiebot\.com/i), 133 S('CookieYes', 'Cookie consent', /cdn-cookieyes\.com/i), 134 S('Osano', 'Cookie consent', /cmp\.osano\.com/i), 135 S('TrustArc', 'Cookie consent', /consent\.trustarc\.com/i), 136 S('Usercentrics', 'Cookie consent', /usercentrics\.eu/i), 137 S('Didomi', 'Cookie consent', /sdk\.privacy-center\.org/i), 138 // Monitoring 139 S('Sentry', 'Error monitoring', /browser\.sentry-cdn\.com|Sentry\.init|ingest\.sentry\.io/i), 140 S('Datadog RUM', 'Monitoring', /datadoghq-browser-agent|DD_RUM/i), 141 S('New Relic', 'Monitoring', /js-agent\.newrelic\.com|NREUM/), 142 S('LogRocket', 'Monitoring', /cdn\.logrocket\.io|cdn\.lr-ingest\.io|LogRocket\.init/i), 143 // Security & other widgets 144 S('reCAPTCHA', 'Security', /google\.com\/recaptcha|recaptcha\/api\.js/i), 145 S('hCaptcha', 'Security', /hcaptcha\.com\/1\/api\.js/i), 146 S('Cloudflare Turnstile', 'Security', /challenges\.cloudflare\.com\/turnstile/i), 147 S('Algolia', 'Search', /algolianet\.com|algoliasearch|\.algolia\.net/i), 148 S('Google Maps', 'Maps', /maps\.googleapis\.com|google\.com\/maps\/embed/i), 149 S('Wistia', 'Video', /fast\.wistia\.(?:com|net)/i), 150 S('Vimeo', 'Video', /player\.vimeo\.com/i), 151 S('YouTube', 'Video', /youtube(?:-nocookie)?\.com\/embed\//i), 152 S('Trustpilot', 'Reviews', /widget\.trustpilot\.com/i), 153 S('Yotpo', 'Reviews', /staticw2\.yotpo\.com|cdn-widgetsrepository\.yotpo\.com/i), 154]; 155 156export function detectTech(html, headerText) { 157 const out = []; 158 for (const s of SIGNATURES) { 159 if ((s.h && s.h.test(html)) || (s.hd && s.hd.test(headerText))) out.push({ name: s.n, category: s.c }); 160 } 161 return out; 162}