🌱 ESG Scraper: Sustainability Reports, Articles & PDF Disclosures Extractor

Enterprise-grade ESG web scraper that automatically extracts sustainability articles, corporate reports, climate news, and PDF disclosures — clean, structured, and ready for investors, compliance teams, or AI training.

High-quality ESG & Sustainability Web Scraper for Investors, Analysts, and AI Teams

Automatically collects ESG articles, sustainability reports, corporate disclosures, climate news, and PDF reports from any website — clean, structured, ready for analysis or AI.

Built for:

• Sustainable investors & analysts
• Compliance and risk teams
• AI/ML engineers building ESG models
• Researchers and NGOs tracking climate & governance trends

✅ Smart ESG keyword filtering ✅ Full clean article text extraction ✅ PDF sustainability reports parsing ✅ Rich metadata (date, author, description) ✅ ESG-relevant images and related links ✅ AI-ready dataset splitting (overview / full-text / images)

👉 Runs on Apify • No code required • Pay only for compute used

---

🚀 Why This Scraper

✔ Purpose-Built for ESG Data Intelligently filters pages using custom ESG keywords (climate, emissions, governance, CSR, net zero, etc.).

✔ Excellent PDF Handling Full text extraction from sustainability and ESG reports (PDF) with metadata where available.

✔ Clean & Noise-Free Output Removes ads, navigation, scripts — only meaningful content remains.

✔ Rich Structured Data Title, publication date, author, description, ESG keywords, internal links, relevant images.

✔ AI & ML Ready Optional splitting into specialized datasets for RAG, LLM fine-tuning, or training.

✔ Fast & Efficient
Powered by Crawlee + Cheerio — excellent for static and content-heavy sites (news, corporate pages, PDFs). For heavily JavaScript-rendered sites, results may vary.

✔ Safe & Controlled Crawling Automatic domain restriction, depth limit (max 3 levels), request limits.

---

💼 Use Cases

• ESG portfolio screening and risk monitoring
• Training ESG-focused LLMs or RAG systems
• Regulatory compliance and disclosure tracking
• Competitive intelligence on corporate sustainability
• Academic research on climate and governance trends

---

📊 Supported Sources

• ESG news sections (Reuters, Bloomberg, FT, Guardian, etc.)
• Corporate sustainability / ESG pages
• Annual sustainability reports (PDF)
• Climate, emissions, governance disclosures

---

⚙️ How It Works

1. Provide start URLs (news sections, corporate pages, PDF links)
2. Set custom ESG keywords and limits
3. Run the Actor
4. Download clean, structured ESG datasets

---

🧩 Input Configuration

Example JSON Input

{
"startUrls": [
  { "url": "https://www.reuters.com/sustainability/" },
  { "url": "https://www.weforum.org/stories/technological-innovation/" }
],
  "allowedDomains": ["reuters.com"],
  "useApifyProxy": false,
  "maxRequestsPerCrawl": 500,
  "esgKeywords": [
    "ESG",
    "sustainability",
    "climate",
    "emissions",
    "net zero",
    "governance"
  ],
  "extractContent": true,
  "extractMetadata": true,
  "followLinks": true,
  "useSeparateDatasets": true,
  "cleanDefaultDataset": true,
  "proxyUrls": [
    {
      "url": "http://user:pass@host:port"
    }
  ]
}

Key Options

• startUrls — one or more starting pages or direct PDF links (required)
• allowedDomains — restrict crawling to specific domains. If empty, automatically limited to domains from startUrls
• maxRequestsPerCrawl — control cost and crawl size
• esgKeywords — custom list for relevance filtering (default includes common ESG terms)
• extractContent / extractMetadata — toggle full text or metadata extraction
• followLinks — enable internal crawling (limited to depth 3 for safety)
• useSeparateDatasets — recommended for large runs and AI workflows
• cleanDefaultDataset — clear previous run data

---

📂 Output Datasets

When useSeparateDatasets: true (recommended):

• esg-overview (primary) — lightweight metadata for fast analysis
• esg-full-content — long articles (>5000 characters)
• esg-images — ESG-relevant images with context
• Default dataset — minimal preview records (for Apify UI visibility)

When useSeparateDatasets: false

• Single dataset with full detailed records

---

Example Output Record (Full Mode)

{
  "url": "https://www.reuters.com/sustainability/example",
  "title": "Companies strengthen climate commitments",
  "scrapedAt": "2025-12-15T10:30:45Z",
  "publishedDate": "2025-12-10",
  "author": "Jane Doe",
  "description": "Major firms enhance ESG targets...",
  "content": "Full clean article text...\n\nParagraphs preserved...",
  "esgKeywords": ["climate", "emissions", "sustainability"],
  "relatedLinks": [
    {
      "url": "https://www.reuters.com/sustainability/esg-guide",
      "text": "ESG Explained"
    }
  ],
  "images": [
    {
      "url": "https://reuters.com/chart-netzero.jpg",
      "alt": "Net zero emissions progress"
    }
  ]
}

PDF Example

{
  "url": "https://company.com/sustainability-2024.pdf",
  "title": "Annual Sustainability Report 2024",
  "content": "Full extracted report text...",
  "esgKeywords": ["sustainability", "carbon", "governance"],
  "type": "PDF",
  "author": "Corporate Sustainability Team",
  "publishedDate": "2024-03-15"
}

---

🏁 Getting Started

1. Click “Try for free” on Apify
2. Paste ESG/sustainability URLs or direct PDF links
3. Customize keywords and limits
4. Run and download your dataset

---

📧 Support

• Email: kidaxxb@gmail.com
• Response within 24 hours
• Issues: Use Apify Issues tab

Tags: ESG, sustainability, web scraping, PDF extraction, climate data, corporate governance, RAG, LLM training, sustainable investing, compliance monitoring

Built with ❤️ on Apify