Instagram Advanced Scraper (Rate Limit Bypass)

An advanced Instagram scraper for Apify that bypasses rate limits using residential proxy rotation, user-agent rotation, and anti-bot detection techniques. Scrapes profiles, posts, stories, and engagement metrics.

Features

• Profile Scraping: Extract username, full name, bio, follower count, following count, post count, and verified status
• Post Scraping: Extract post URLs, captions, likes, comments, and media URLs
• Story Scraping: Optionally scrape Instagram Stories from profile pages
• Comment Extraction: Optionally scrape comments on each post
• Rate Limit Bypass:
  • Residential proxy rotation (Apify RESIDENTIAL group)
  • 5 modern user-agent rotation
  • Anti-bot detection via navigator.webdriver override and Chrome runtime spoofing
  • Random delays between requests (2-5s)
  • Max 3 requests per profile to avoid triggering rate limits
  • Retry on failure (up to 3 retries)

Input

Field	Type	Default	Description
profileUrls	array[string]	["https://www.instagram.com/"]	Instagram URLs (profiles, posts, or stories)
maxPosts	integer	50	Max posts to scrape per profile (max 500)
includeStories	boolean	false	Also scrape Instagram Stories
includeComments	boolean	false	Also scrape comments on posts
proxyConfiguration	object	{ useApifyProxy: true, apifyProxyGroups: ["RESIDENTIAL"] }	Proxy config — RESIDENTIAL proxies strongly recommended

Output

Each dataset item has a type field: "profile", "post", "story", or "error".

Profile

{
  "type": "profile",
  "username": "example",
  "fullName": "Example Account",
  "bio": "This is a bio",
  "followerCount": 1234,
  "followingCount": 567,
  "postCount": 89,
  "isVerified": false,
  "profileUrl": "https://www.instagram.com/example/",
  "scrapedAt": "2025-06-06T12:00:00.000Z"
}

Post

{
  "type": "post",
  "postUrl": "https://www.instagram.com/p/ABC123/",
  "caption": "Post caption here",
  "likesCount": 150,
  "commentsCount": 12,
  "mediaUrl": "https://...",
  "username": "example",
  "profileUrl": "https://www.instagram.com/example/",
  "comments": [],
  "scrapedAt": "2025-06-06T12:00:00.000Z"
}

Rate Limit Bypass Strategy

Instagram significantly increased rate limiting in Q1 2025, breaking many existing scrapers. This actor implements:

1. Residential Proxies: Essential for avoiding blocks. Datacenter proxies are frequently blocked.
2. User-Agent Rotation: 5 modern browser user-agents rotated per request
3. Anti-Detection Scripts: navigator.webdriver set to false, Chrome runtime spoofing, valid plugin and language values
4. Random Delays: 2-5 second delay after each page load, plus 1-3 second pre-scrape delay
5. Request Limits: Max 3 requests per profile URL pattern
6. Retry Logic: Failed requests retried up to 3 times

Local Development

# Install dependencies
npm install

# Run syntax check
node --check src/main.js

# Run locally (requires Apify token and proxy)
APIFY_TOKEN=your_token_here node src/main.js

Technical Notes

• Built with Crawlee PlaywrightCrawler for robust browser automation
• ES module format ("type": "module")
• Compatible with Apify platform's apify/actor-node-playwright-chrome Docker image
• Selectors target multiple DOM patterns for resilience against Instagram's dynamic class names