Domain Info Scraper

A powerful Apify actor that scrapes comprehensive domain information including WHOIS data, DNS records, IP addresses, ASN information, and more.

🚀 Quick Start

1. Input: Provide a domain, list of domains, or URLs
2. Run: Execute the actor on Apify
3. Output: Get detailed domain analysis in the dataset

✨ Features

• WHOIS Information: Creation date, expiration date, registrar details, name servers
• DNS Records: A records, MX records, TXT records, SPF records
• IP Analysis: IP addresses, ASN numbers, ASN country, ASN description
• Registrar Information: Accredited registrar details from ICANN
• Flexible Input: Accepts domains, URLs, or lists of either
• Error Handling: Graceful handling of invalid domains with detailed error reporting

📥 Input Schema

The actor accepts three types of input formats:

1. Single Domain

{
  "domain": "example.com"
}

2. List of Domains

{
  "domains": [
    "example.com",
    "google.com",
    "github.com"
  ]
}

3. Start URLs (to extract domains from)

{
  "startUrls": [
    {
      "url": "https://example.com"
    },
    {
      "url": "https://google.com"
    }
  ]
}

📤 Output Format

Each domain processed will generate a result with the following structure:

{
  "domain": "example.com",
  "creation_date": "1995-08-14T04:00:00Z",
  "expiration_date": "2024-08-13T04:00:00Z",
  "updated_date": "2023-08-14T04:00:00Z",
  "dnssec": false,
  "emails": "dns-admin@example.com",
  "name_servers": "a.iana-servers.net / b.iana-servers.net",
  "registrar": "ICANN",
  "registrar_country": "United States",
  "ip_addresses": "93.184.216.34",
  "ASN": "15133",
  "ASN_Country": "US",
  "ASN_Description": "EDGECAST - EdgeCast Inc.",
  "TXT": "v=spf1 -all",
  "SPF": "v=spf1 -all",
  "MX": "mail.example.com",
  "MX_IP": "93.184.216.34",
  "MX_ASN#": "15133",
  "MX_ASN_Country": "US",
  "MX_ASN_Description": "EDGECAST - EdgeCast Inc.",
  "processed_at": "2024-01-15T10:30:00.000Z",
  "actor_run_id": "abc123def456"
}

⚠️ Error Handling

If a domain cannot be processed, an error result will be generated:

{
  "domain": "invalid-domain",
  "error": "Error message describing the issue",
  "processed_at": "2024-01-15T10:30:00.000Z",
  "actor_run_id": "abc123def456"
}

🔧 Requirements

• Python 3.7+
• Google Drive API credentials (for data file downloads)
• Internet connection for domain lookups

📦 Dependencies

• apify-client: Apify platform integration
• dnspython: DNS record lookups
• whois: WHOIS information retrieval
• tldextract: Domain parsing
• google-api-python-client: Google Drive integration

💡 Usage Examples

Basic Domain Analysis

{
  "domain": "example.com"
}

Batch Domain Processing

{
  "domains": [
    "example.com",
    "google.com",
    "github.com"
  ]
}

URL-based Domain Extraction

{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ]
}

📊 Performance & Limits

• Memory: 512MB allocated
• Timeout: 5 minutes per run
• Rate Limiting: Built-in delays to respect DNS and WHOIS server limits
• Batch Processing: Efficiently handles multiple domains in sequence

🔍 Troubleshooting

Common Issues

1. "No valid domains provided": Check your input format matches the schema
2. "WHOIS Error": Domain may have restricted access or rate limiting
3. "DNS Error": Domain may be unreachable or have DNS issues
4. Timeout errors: Reduce the number of domains per run

Best Practices

• Start with 1-5 domains to test
• Use valid domain formats (e.g., "example.com" not "http://example.com")
• Check logs for detailed error information
• For large batches, consider splitting into multiple runs

📝 Notes

• The actor automatically downloads required data files (accredited registrars, IP-to-ASN mappings)
• Processing time depends on the number of domains and network conditions
• Some domains may have restricted WHOIS information
• The actor handles rate limiting and provides detailed logging
• Results are automatically saved to the Apify dataset

🆘 Support

For issues or questions:

1. Check the Apify platform documentation
2. Review the actor logs for error details
3. Contact support through the Apify console
4. Ensure your input matches the documented schema

📄 License

This actor is provided as-is for use on the Apify platform.