Instagram Followers/Following Scraper

A powerful Apify Actor for scraping Instagram followers and following lists using Playwright. This actor intercepts Instagram's internal API responses to efficiently collect user data with advanced authentication handling and rate limit detection.

Features

• Dual Scraping Modes: Scrape either followers or following lists
• Advanced Authentication: Cookie-based authentication with comprehensive validation
• Proxy Support: Built-in support for residential proxies to avoid rate limits
• Real-time Data Streaming: Results are pushed to the dataset in real-time as they're scraped
• Smart Rate Limit Handling: Automatically detects and handles Instagram rate limit modals
• Headless Browser: Runs in headless mode for efficient scraping
• API Interception: Captures Instagram's internal API responses for fast, accurate data collection
• Error Handling: Robust error detection with screenshots for debugging
• Session Validation: Detects invalid cookies, expired sessions, and automated behavior warnings

How to use:

▶️ Watch an overview/demo video here:
https://youtu.be/aCtfAj5PCvI

Input Parameters

Parameter	Type	Required	Default	Description
instagram_url	String	Yes	-	Instagram profile URL to scrape (e.g., https://www.instagram.com/username/)
scraping_type	String	No	followers	Type of data to scrape: followers or following
max_number	Integer	No	100	Maximum number of public users to scrape (1-10000)
max_page	Integer	No	10	Maximum number of API pages to scroll through (1-100)
cookies	Array	Yes	-	Instagram cookies for authentication (see below)
proxy_url	String	No	-	Residential proxy URL (format: http://username:password@proxy.example.com:8000)

Important Notes

• max_number: Only counts public profiles. Private profiles are also saved but not counted toward this limit.
• max_page: Each scroll triggers an API request. More pages = more data but longer runtime.
• Cookies are required: Instagram requires authentication to view followers/following lists.

How to Get Instagram Cookies

Instagram cookies are required for authentication. Follow these steps to export your cookies:

Using Cookie Export Extensions

Use browser extensions like:

• EditThisCookie (Chrome/Edge)
• Cookie-Editor (Firefox/Chrome)
• Get cookies.txt LOCALLY (Chrome)

Export cookies and format them as JSON array:

[
  {
    "name": "sessionid",
    "value": "YOUR_SESSION_ID_HERE",
    "domain": ".instagram.com",
    "path": "/",
    "httpOnly": true,
    "secure": true,
    "sameSite": "Lax"
  },
  {
    "name": "csrftoken",
    "value": "YOUR_CSRF_TOKEN_HERE",
    "domain": ".instagram.com",
    "path": "/",
    "secure": true,
    "sameSite": "Lax"
  },
  {
    "name": "ds_user_id",
    "value": "YOUR_USER_ID_HERE",
    "domain": ".instagram.com",
    "path": "/",
    "secure": true,
    "sameSite": "Lax"
  }
]

Output Format

Each scraped user is saved with the following fields:

{
  "pk": "1234567890",
  "username": "example_user",
  "profile_url": "https://www.instagram.com/example_user/",
  "full_name": "Example User",
  "is_private": false,
  "profile_pic_url": "https://instagram.com/path/to/profile_pic.jpg",
  "curr_page": 1
}

Output Fields

Field	Type	Description
pk	String	Instagram user ID (unique identifier)
username	String	Instagram username/handle
profile_url	String	Direct link to the user's Instagram profile
full_name	String	User's display name
is_private	Boolean	Whether the profile is private or public
profile_pic_url	String	URL to the user's profile picture
curr_page	Integer	The API page number where this user was found

Usage Example

Basic Usage (Followers)

{
  "instagram_url": "https://www.instagram.com/instagram/",
  "scraping_type": "followers",
  "max_number": 500,
  "max_page": 20,
  "cookies": [
    {
      "name": "sessionid",
      "value": "your_session_id",
      "domain": ".instagram.com",
      "path": "/",
      "httpOnly": true,
      "secure": true
    }
  ]
}

With Proxy (Following)

{
  "instagram_url": "https://www.instagram.com/username/",
  "scraping_type": "following",
  "max_number": 1000,
  "max_page": 50,
  "cookies": [...],
  "proxy_url": "http://user:pass@proxy.example.com:8000"
}

Rate Limiting & Best Practices

Instagram actively monitors and rate-limits automated activity. Follow these best practices:

1. Use Residential Proxies

• Datacenter proxies are often blocked by Instagram
• Residential proxies are highly recommended for reliable scraping
• The actor supports authenticated proxy URLs

2. Scrape Responsibly

• Don't scrape too aggressively (keep max_page reasonable)
• Add random delays between scrolls (built-in: 3-6 seconds)
• The actor includes automatic rate limit detection

3. Cookie Management

• Cookies expire after some time
• Use cookies from an actively logged-in session
• Avoid using cookies from accounts with suspicious activity
• The actor validates cookies and provides clear error messages

4. Automated Behavior Warnings

The actor automatically detects and reports:

• Invalid/expired cookies
• Login modal appearances
• Challenge pages
• Automated behavior warnings
• Rate limit modals (and dismisses them automatically)

Error Handling

The actor provides detailed error messages for common issues:

Authentication Errors

{
  "error": "authentication failed: Invalid or expired cookies"
}

Solution: Update your cookies with fresh ones from an active Instagram session.

Automated Behavior Detection

{
  "error": "automated behavior detected: Instagram is showing a challenge page"
}

Solution: Wait a few hours/days before scraping again. Use a different account or proxy.

No Followers/Following

{
  "error": "no followers / following"
}

Info: The target profile has 0 followers or following (depending on scraping type).

Navigation Timeout

{
  "error": "navigation timeout after retries"
}

Solution: Check your internet connection or try using a proxy.

Technical Details

Scroll Behavior

• Random delays between 3-6 seconds per scroll
• Detects when no new data is loaded (3 consecutive failures)
• Automatically handles rate limit modals
• Uses mouse wheel scrolling for natural behavior

Limitations

• Authentication Required: Instagram requires valid session cookies
• Rate Limits: Instagram may temporarily block excessive requests
• Private Profiles: Cannot scrape followers/following of private profiles (unless you follow them)
• Large Lists: Very large follower counts (millions) may take hours to scrape
• Bot Detection: Instagram actively detects automated behavior

Support & Troubleshooting

Common Issues

Issue: "Login modal appeared"

• Cause: Cookies are invalid or expired
• Fix: Export fresh cookies from an active session

Issue: "Modal not found"

• Cause: Profile is private or you don't have permission
• Fix: Make sure you follow the private profile with the account whose cookies you're using

Issue: "Rate limit modal"

• Cause: Too many requests in a short time
• Fix: Wait a few hours, use proxies, or reduce max_page

Issue: Slow scraping

• Cause: Random delays prevent bot detection
• Fix: This is intentional. Faster scraping increases ban risk.

License

This actor is for educational and research purposes. Please respect Instagram's Terms of Service and robots.txt. Use responsibly and don't spam or harass users.

Resources

• Apify Documentation
• Apify SDK for Python
• Playwright Documentation
• Instagram Terms of Service

Changelog

Version 0.1

• Initial release
• Followers/following scraping
• Cookie authentication
• Proxy support
• Rate limit detection
• Real-time data streaming
• Error handling with screenshots

---

Note: This tool is for educational purposes. Always respect website terms of service and rate limits. Use responsibly and ethically.