BuiltWith Technology Scraper - Apify Actor

This Apify actor scrapes BuiltWith.com to extract all technologies used by a specific company domain.

📁 Project Structure

.
├── main.js              # Main actor code with multi-method extraction
├── package.json         # Dependencies and project config
├── INPUT_SCHEMA.json    # Input schema for the actor
├── .actor/
│   └── actor.json       # Actor configuration
└── README.md           # This file

🚀 Setup Instructions

1. Install Node.js (if not already installed)

Download and install Node.js from nodejs.org (version 18 or higher recommended).

2. Install Dependencies

$npm install

This will install:

• apify - Apify SDK for building actors
• playwright - Browser automation for scraping

3. Install Playwright Browsers

Playwright needs to download browser binaries:

$npx playwright install chromium

4. Test Locally

Test the actor with a sample domain:

$apify run -i '{"domain": "kendakehoney.co.uk"}'

Or with an input file:

echo '{"domain": "kendakehoney.co.uk"}' > input.json
apify run -i input.json

5. Deploy to Apify

1. Login to Apify:

   $apify login
   

2. Push your actor to Apify:

   $apify push
   

3. Go to Apify Console to see your actor and run it!

6. View Output Locally

After running the actor, view the results in a readable format:

$node view-output.js

Or view the raw JSON file:

$cat storage/datasets/default/000000001.json | python3 -m json.tool

The output file is located at: storage/datasets/default/000000001.json

📝 Usage

Input Format

{
  "domain": "example.com"
}

The domain can be provided with or without:

• Protocol (http:// or https://)
• www prefix
• Trailing slashes

Examples:

• kendakehoney.co.uk
• www.example.com
• https://example.com

Output Format

{
  "results": [
    {
      "domain": "example.com",
      "Mobile": "IPhone / Mobile Compatible, Apple Mobile Web Clips Icon",
      "Payment": "PayPal, Visa, MasterCard",
      "eCommerce": "Shopify",
      "Analytics and Tracking": "Google Analytics",
      ...
    }
  ]
}

🔧 How It Works

The actor uses 4 different extraction methods (A/B testing approach) and automatically selects the one that works best:

1. Method 1: Standard BuiltWith CSS selectors
2. Method 2: Data attributes and structured divs
3. Method 3: Text parsing from page content
4. Method 4: Comprehensive DOM traversal

The actor will try each method until one successfully extracts the technology data.

📦 Technologies Extracted

The actor extracts technologies across various categories including:

• Mobile technologies
• Payment systems
• Widgets and plugins
• Languages
• eCommerce platforms
• Frameworks
• Advertising networks
• Name servers
• SSL certificates
• Shipping providers
• Audio/Video media
• Web hosting providers
• Analytics and tracking
• Email hosting
• Content delivery networks
• JavaScript libraries
• And more...

⚠️ Notes

• BuiltWith.com may have rate limits or require authentication for some features
• The actor includes delays to respect the website
• Results may vary based on BuiltWith's current page structure
• Make sure to comply with BuiltWith's Terms of Service

🐛 Troubleshooting

If the actor doesn't extract data:

1. Check if the domain exists on BuiltWith.com
2. Try running it again (BuiltWith may have temporary issues)
3. Check the actor logs to see which extraction method was used

📄 License

This actor is provided as-is for scraping BuiltWith.com technology data.