Malaysia Healthcare Directory Scraper

A professional-grade web scraping solution for extracting comprehensive healthcare specialist data from Malaysia's National Specialist Register (NSR). Built for healthcare analytics, market research, medical directory services, and healthcare business intelligence.

🌟 Key Features

🔍 Comprehensive Data Extraction

• 20+ data fields per specialist including qualifications, addresses, and contact details
• Geocoding integration with latitude/longitude coordinates
• Multi-format parsing handles various website layouts automatically
• Quality validation with completeness reporting

🏥 Healthcare-Specific Capabilities

• NSR Number extraction with format validation
• Qualification tracking including awarding bodies and years
• Practice addresses with geocoding to coordinates
• Specialty classification and subspecialty details
• Registration dates and renewal information

⚡ Enterprise-Grade Reliability

• Retry logic with exponential backoff
• Rate limiting to respect website policies
• Error recovery and detailed logging
• Pagination safety prevents excessive scraping
• Progress monitoring with real-time updates

📊 Business Intelligence Ready

• Structured JSON output compatible with analytics platforms
• Dataset views optimized for different analysis needs
• CSV export for spreadsheet analysis
• API-ready for integration with healthcare systems

🚀 Quick Start

Apify Platform (Recommended)

# Install Apify CLI
npm install -g apify-cli
apify login

# Push and run the Actor
apify push
apify run malaysia-healthcare-directory-scraper

Local Development

# Clone and setup
cd malaysia-healthcare-directory-scraper
pip install -r requirements.txt

# Run locally
python src/main.py

📋 Input Parameters

Parameter	Type	Default	Description
states	Array	[]	Malaysian states to scrape (empty = all states)
delay	Number	1.5	Delay between requests in seconds
fetch_all_pages	Boolean	true	Scrape all pages per state
extractDetails	Boolean	true	Extract detailed profile information
geocodeAddresses	Boolean	true	Convert addresses to coordinates
maxPagesPerState	Integer	500	Safety limit for pages per state
maxConcurrency	Integer	10	Concurrent detail page requests

Example Input

{
  "states": ["johor", "selangor", "kuala lumpur"],
  "extractDetails": true,
  "geocodeAddresses": true,
  "maxPagesPerState": 500
}

📊 Output Data Structure

Core Fields

• nsrNo: National Specialist Register number
• name: Specialist full name
• jobTitle: Professional title (Dr., Prof., etc.)
• gender: Male/Female
• specialty: Primary medical specialty
• subspecialty: Subspecialty classification

Professional Details

• qualifications: Array of qualification objects
• qualificationsStructured: Detailed qualification data
• registrationDate: Date of NSR registration
• years_of_experience: Calculated experience years
• sector: Public/Private sector

Location & Contact

• state: Malaysian state
• city: City/town
• address: Full practice address
• establishment: Healthcare facility name
• latitude/longitude: GPS coordinates (when geocoding enabled)

Metadata

• profileUrl: Source URL
• state_category: Regular/Special state classification

🎯 Use Cases

🏥 Healthcare Market Research

• Analyze specialist distribution by specialty and location
• Identify market gaps and saturation areas
• Track qualification trends and specialties growth

📈 Business Intelligence

• Healthcare facility analysis and competitor mapping
• Geographic distribution of medical specialties
• Practice location optimization

🔬 Medical Research

• Epidemiological studies using specialist distribution
• Healthcare access analysis by region
• Specialty-specific research and surveys

🏢 Healthcare Administration

• Workforce planning and resource allocation
• Specialist registry maintenance and verification
• Healthcare policy and planning support

📈 Performance & Limits

Scraping Limits

• Max 500 pages per state (configurable safety limit)
• 1.5 second delays between requests (respectful crawling)
• Automatic retry with exponential backoff
• Rate limiting prevents server overload

Data Quality

• 100% NSR number validation with format checking
• Address geocoding with fallback handling
• Completeness validation with detailed reporting
• Duplicate detection and data integrity checks

Processing Speed

• ~20 specialists per page (varies by state)
• ~40 pages/hour with detail extraction
• ~800 specialists/hour processing rate
• Scales with Apify platform resources

🛠️ Technical Architecture

Multi-Format Parsing

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Table Format  │ -> │  Detail Parser   │ -> │  Structured     │
│                 │    │                  │    │  JSON Output    │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
                    ┌──────────────────┐
                    │   Geocoding      │
                    │   Service        │
                    └──────────────────┘

Error Handling Strategy

• Network failures: Automatic retry with backoff
• Parse errors: Fallback to alternative parsing methods
• Rate limiting: Adaptive delays and queue management
• Data validation: Post-processing quality checks

📋 Requirements

• Python 3.8+
• Apify account (for cloud deployment)
• Internet connection for NSR website access
• Optional: API keys for enhanced geocoding

🔧 Configuration

Environment Variables

# Apify platform (auto-configured)
ACTOR_DEFAULT_DATASET_ID=default

# Local development
PYTHONPATH=/path/to/actor/src

Custom Geocoding

The Actor includes OpenStreetMap integration by default. For enhanced geocoding accuracy, consider:

• Google Maps API integration
• HERE Maps API
• Mapbox Geocoding API

📊 Dataset Views

Overview (Default)

Complete specialist profiles with all fields for comprehensive analysis.

By State

Group specialists by Malaysian state for regional analysis.

By Specialty

Organize by medical specialty for specialty-specific insights.

🤝 Contributing

This is a commercial healthcare data extraction tool. For customizations or enhancements:

1. Feature Requests: Open issues with detailed requirements
2. Data Quality: Report parsing issues or missing fields
3. Performance: Suggest optimizations for large-scale scraping

📜 License & Compliance

• Commercial Use: Licensed for business and research applications
• Data Usage: Subject to NSR terms of service and Malaysian data protection laws
• Ethical Scraping: Respects website policies and implements rate limiting
• GDPR Compliance: Handles personal data appropriately for healthcare context

🆘 Support

Common Issues

• Rate limiting: Increase delay between requests
• Geocoding failures: Check address format or disable geocoding
• Parse errors: Enable debug logging for troubleshooting

Performance Optimization

• Reduce maxConcurrency for slower networks
• Increase delay for high-traffic periods
• Use maxPagesPerState to limit data volume

---

Built with ❤️ for healthcare analytics and medical research in Malaysia

Extracting healthcare intelligence from Malaysia's National Specialist Register 🏥🇲🇾