Instagram Hashtag Super Scraper

Extract comprehensive post data from Instagram hashtags including captions, locations, likes, comments, images, timestamps, hashtags, mentions, and owner information.

Features

• Scrape posts and reels from Instagram hashtags
• Extract detailed post metadata (likes, comments, timestamps)
• Get caption text with hashtags and mentions
• Extract image URLs and dimensions
• Support for carousel posts (multiple images)
• Get owner/author information
• Location data (if available)
• Music information for reels
• Export data in JSON, CSV, Excel formats
• Fast and efficient scraping with automatic pagination
• Rate limiting and retry logic

Output Format

Each post is extracted with the following data structure:

{
  "inputUrl": "https://www.instagram.com/explore/tags/Interior_design_saudi",
  "id": "3756182696478136272",
  "type": "Image",
  "shortCode": "DQgpELnDFfQ",
  "caption": "من الفكرة إلى التنفيذ...",
  "hashtags": ["Interior_design_saudi", "ديكور_فاخر"],
  "mentions": [],
  "url": "https://www.instagram.com/p/DQgpELnDFfQ/",
  "commentsCount": 0,
  "firstComment": "",
  "latestComments": [],
  "dimensionsHeight": 1594,
  "dimensionsWidth": 1277,
  "displayUrl": "https://scontent-...",
  "images": [],
  "likesCount": 0,
  "timestamp": "2025-11-01T10:11:32.000Z",
  "childPosts": [],
  "ownerFullName": "SAK",
  "ownerUsername": "sakco.sa",
  "ownerId": "78029144673",
  "productType": "feed",
  "isSponsored": true,
  "musicInfo": {
    "audio_canonical_id": "0",
    "audio_type": null,
    "music_info": null,
    "original_sound_info": null,
    "pinned_media_ids": null
  }
}

Setup

1. Install dependencies:

   $npm install
   

2. Configure Instagram cookies:

   Edit cookies.json with your Instagram session cookies. For better performance, add multiple sessions:

   {
     "sessions": [
       "sessionid=YOUR_SESSION_ID_1; csrftoken=YOUR_CSRF_TOKEN_1; ds_user_id=YOUR_USER_ID_1",
       "sessionid=YOUR_SESSION_ID_2; csrftoken=YOUR_CSRF_TOKEN_2; ds_user_id=YOUR_USER_ID_2",
       "sessionid=YOUR_SESSION_ID_3; csrftoken=YOUR_CSRF_TOKEN_3; ds_user_id=YOUR_USER_ID_3"
     ]
   }
   

   🚀 Performance Tip: The scraper automatically uses parallel processing based on the number of sessions:

   • 1 session = slower, sequential processing
   • 3 sessions = 3x faster with parallel comment fetching
   • 5 sessions = 5x faster (optimal)

   To get these cookies:

   1. Login to Instagram in your browser
   2. Open Developer Tools (F12)
   3. Go to Application/Storage → Cookies → https://www.instagram.com
   4. Copy the values for sessionid, csrftoken, and ds_user_id
   5. For multiple sessions: Use different browser profiles or browsers (Chrome, Firefox, Edge)

3. Configure input:

   Edit test_input.json or pass input when running:

   {
     "hashtags": ["Interior_design_saudi", "webscraping"],
     "resultsLimit": 50,
     "resultsType": "posts"
   }
   

Usage

Local Development

$npm start

Apify Platform

1. Push to Apify:

   $apify push
   

2. Run on Apify platform with custom input

Input Parameters

Parameter	Type	Description	Default
hashtags	Array	Instagram hashtags to scrape (with or without # symbol)	Required
resultsLimit	Integer	Max posts per hashtag (0 = unlimited)	50
resultsType	String	Content type: "posts" or "reels"	"posts"
maxComments	Integer	Max comments to fetch per post (0 = skip comments for faster scraping)	0

Example Input

{
  "hashtags": [
    "Interior_design_saudi",
    "webscraping",
    "Coffee"
  ],
  "resultsLimit": 100,
  "resultsType": "posts"
}

Output

Results are saved to the dataset and can be exported in multiple formats:

• JSON
• CSV
• Excel
• HTML

Post Types

The scraper handles three types of Instagram posts:

• Image: Single image posts
• Sidecar: Carousel posts with multiple images
• Video: Video posts and reels

Important Notes

Cookie Requirements:

• Valid Instagram session cookies are required
• Cookies expire periodically and need to be refreshed
• Use cookies from an account that can view the target hashtags

Rate Limiting:

• The scraper includes automatic delays to respect Instagram's rate limits
• Scraping too aggressively may result in temporary blocks
• Default delay: 200ms between requests (reduced from 2s due to session rotation)
• With multiple sessions, rate limits are distributed across accounts

Performance Optimization:

• Session Rotation: Automatically rotates between multiple Instagram sessions
• Parallel Comment Fetching: Uses Promise.all to fetch comments concurrently
• Staggered Requests: Adds small delays (100ms) between parallel requests to avoid bursts
• Smart Retry: If a request fails, automatically switches to a different session for retry

Legal & Ethical Use:

• Only scrape public data
• Respect Instagram's Terms of Service
• Don't scrape private accounts without permission
• Use for legitimate purposes only

Troubleshooting

No posts extracted:

• Check that your cookies are valid and not expired
• Verify the hashtags exist and have public posts
• Ensure the hashtags are spelled correctly

API Errors:

• Refresh your session cookies
• Reduce the resultsLimit to avoid rate limiting
• Add delays between requests
• Check if Instagram has changed their API

Empty fields:

• Some fields may not be available for all posts
• ownerFullName may be empty (Instagram API limitation)
• Location data is only available if the poster added it

Support

For issues or questions, please open an issue on the repository.

License

ISC