Instagram Post/Reels Scraper

A high-performance Instagram post and reels metadata scraper. Features concurrent scraping, single-session authentication, and comprehensive error handling.

Features

• Single Authentication: Logs in once and reuses the session across all posts
• Concurrent Scraping: Process multiple posts simultaneously for faster execution
• Comprehensive Metadata: Extracts detailed post and reels information
• Error Resilient: Failed posts don't stop the entire scraping process
• Rate Limit Aware: Configurable batch sizes to avoid Instagram rate limits
• Cookie-based Authentication: Secure login without storing passwords

Input Configuration

{
  "postUrls": [
    "https://www.instagram.com/p/ABC123/",
    "https://www.instagram.com/reel/XYZ789/"
  ],
  "cookies": [
    {
      "name": "sessionid",
      "value": "your_session_id",
      "domain": ".instagram.com",
      "path": "/"
    },
    {
      "name": "csrftoken",
      "value": "your_csrf_token",
      "domain": ".instagram.com",
      "path": "/"
    },
    {
      "name": "ds_user_id",
      "value": "your_user_id",
      "domain": ".instagram.com",
      "path": "/"
    }
  ]
}

Parameters

Parameter	Type	Required	Default	Description
postUrls	Array	Yes	-	List of Instagram post/reel URLs to scrape
cookies	Array	Yes	-	Instagram session cookies for authentication

Supported URL Formats

• Regular posts: https://www.instagram.com/p/{shortcode}/
• Reels: https://www.instagram.com/reel/{shortcode}/
• Reels (alternate): https://www.instagram.com/reels/{shortcode}/
• IGTV: https://www.instagram.com/tv/{shortcode}/

Getting Instagram Cookies

Method 1: Browser Developer Tools

1. Open Instagram in your browser and log in
2. Open Developer Tools (F12)
3. Go to the "Application" or "Storage" tab
4. Click on "Cookies" → "https://www.instagram.com"
5. Copy the following cookies:
   • sessionid
   • csrftoken
   • ds_user_id

Method 2: Browser Extensions

Use cookie export extensions like:

• EditThisCookie (Chrome/Edge)
• Cookie-Editor (Firefox)

Export cookies in JSON format and extract the required values.

Required Cookie Format

{
  "name": "sessionid",
  "value": "123456789...",
  "domain": ".instagram.com",
  "path": "/",
  "expires": 1234567890,
  "httpOnly": true,
  "secure": true
}

Output Data

Output Schema

The scraper returns an array of objects with the following structure:

{
  "scraped_at": "2024-12-29T10:36:34.817Z",
  "post": {
    "shortcode": "DSvAQq8CeH_",
    "url": "https://www.instagram.com/p/DSvAQq8CeH_/",
    "type": "video",
    "caption": "Post caption text...",
    "date_posted": "2024-12-28T15:30:00",
    "likes": 15420,
    "comments": 342,
    "is_video": true,
    "video_url": "https://...",
    "video_view_count": 50000,
    "video_duration": 30.5,
    "thumbnail_url": "https://...",
    "media_count": 1,
    "tagged_users": ["user1", "user2"],
    "location": "New York, NY",
    "accessibility_caption": "Image description"
  },
  "user": {
    "username": "example_user",
    "user_id": "123456789",
    "full_name": "Example User",
    "biography": "Bio text...",
    "external_url": "https://example.com",
    "followers": 50000,
    "followees": 500,
    "total_posts": 1250,
    "is_verified": true,
    "is_private": false,
    "is_business_account": true,
    "profile_pic_url": "https://..."
  }
}

Post Metadata Fields

Field	Type	Description
shortcode	String	Unique Instagram post identifier
url	String	Full URL to the post
type	String	Post type: "photo" or "video"
caption	String	Post caption text
date_posted	String	ISO 8601 timestamp of post creation
likes	Integer	Number of likes
comments	Integer	Number of comments
is_video	Boolean	Whether the post is a video
video_url	String	Direct video URL (if applicable)
video_view_count	Integer	Video view count (if applicable)
video_duration	Float	Video duration in seconds (if applicable)
thumbnail_url	String	Post thumbnail/image URL
media_count	Integer	Number of media items in carousel posts
tagged_users	Array	List of tagged usernames
location	String	Location name (if available)
accessibility_caption	String	Alt text for images

User Metadata Fields

Field	Type	Description
username	String	Instagram username
user_id	String	Instagram user ID
full_name	String	Display name
biography	String	Profile bio text
external_url	String	Website link in bio
followers	Integer	Follower count
followees	Integer	Following count
total_posts	Integer	Total number of posts
is_verified	Boolean	Verified badge status
is_private	Boolean	Private account status
is_business_account	Boolean	Business account status
profile_pic_url	String	Profile picture URL

Performance and Limits

Rate Limits

Instagram enforces rate limits to prevent abuse:

• Approximate limit: 200-500 requests per hour per account
• Each post scrape: 2-3 requests (post + user profile)
• Safe limit: 50-100 posts per run
• Conservative limit: 20-30 posts per run

Best Practices

1. Start small: Test with 5-10 posts first
2. Monitor errors: Watch for 403/429 HTTP errors (rate limiting)
3. Use delays: If scraping large datasets, split into multiple runs with 10-15 minute gaps
4. Fresh cookies: Update cookies if you get authentication errors

Error Handling

Common Errors

403 Forbidden

• Cause: Invalid or expired cookies
• Solution: Get fresh cookies from your browser

Invalid URL

• Cause: Unsupported URL format
• Solution: Use supported formats (see "Supported URL Formats")

Connection Timeout

• Cause: Network issues or Instagram downtime
• Solution: Retry after a few minutes

Handling Failed Posts

The scraper continues processing even if some posts fail. Check the logs for:

Failed: 2
Failed URLs: ['https://instagram.com/p/ABC/', 'https://instagram.com/p/XYZ/']

Security Considerations

1. Rotate cookies: Update cookies periodically
2. Use secondary accounts: Don't use your main Instagram account for scraping
3. Respect rate limits: Excessive scraping can lead to account bans
4. Follow Instagram's ToS: Use responsibly and for legitimate purposes

License

This project is provided as-is for educational and research purposes. Users are responsible for complying with Instagram's Terms of Service and applicable laws.

Support

For issues and questions:

• Check the logs for detailed error messages
• Review the "Common Errors" section
• Verify your cookies are fresh and valid
• Test with a small dataset first