Instagram Profile Scraper

A powerful Instagram profile scraper that extracts comprehensive profile data including bio, followers, posts, IGTV videos, and related profiles. This scraper mimics the functionality of Apify's apify/instagram-profile-scraper.

Features

✅ Complete Profile Data Extraction

• Profile information (username, ID, full name, bio)
• Follower and following counts
• Post counts and statistics
• Verification status
• Business account information
• Profile pictures (standard and HD)
• External URLs from bio

✅ Latest Posts & Media

• Extract up to 12 latest posts per profile
• Post captions with hashtags and mentions
• Image and video URLs
• Engagement metrics (likes, comments)
• Post types (Image, Video, Carousel/Sidecar)
• Tagged users
• Timestamp information

✅ IGTV Videos

• Extract up to 12 latest IGTV videos
• Video metadata and statistics
• Video URLs and thumbnails

✅ Related Profiles

• Discover related accounts
• Profile suggestions based on Instagram's algorithm

✅ Optional About Section (Paid Add-on)

• Account join date
• Country information (if available)
• Verification date
• Username change history

Output Format

The scraper outputs data in the exact same format as Apify's Instagram Profile Scraper:

{
  "inputUrl": "https://www.instagram.com/ahmdasofficial",
  "id": "55079635587",
  "username": "ahmdasofficial",
  "url": "https://www.instagram.com/ahmdasofficial",
  "fullName": "احمد - Ahmed",
  "biography": "𝗙𝗼𝘂𝗻𝗱𝗲𝗿 @labshkw...",
  "externalUrls": [...],
  "externalUrl": "https://reach.link/ahmedabsofficial",
  "externalUrlShimmed": "https://l.instagram.com/?u=...",
  "followersCount": 40657,
  "followsCount": 617,
  "hasChannel": false,
  "highlightReelCount": 80,
  "isBusinessAccount": false,
  "joinedRecently": false,
  "businessCategoryName": "Digital creator",
  "private": false,
  "verified": true,
  "profilePicUrl": "https://...",
  "profilePicUrlHD": "https://...",
  "igtvVideoCount": 0,
  "relatedProfiles": [...],
  "latestIgtvVideos": [...],
  "postsCount": 921,
  "latestPosts": [...]
}

Installation

1. Clone the repository:

   git clone <repository-url>
   cd instagram_profile_scraper
   

2. Install dependencies:

   $npm install
   

3. Configure cookies:

   • Edit cookies.json and add your Instagram session cookies
   • You can add multiple sessions for rotation

Configuration

cookies.json

{
  "sessions": [
    "sessionid=YOUR_SESSION_ID; csrftoken=YOUR_CSRF_TOKEN; ...",
    "sessionid=ANOTHER_SESSION; csrftoken=ANOTHER_TOKEN; ..."
  ]
}

test_input.json

{
  "usernames": ["username1", "username2", "https://www.instagram.com/username3/"],
  "includeAboutSection": false
}

Input Schema

The scraper accepts the following input parameters:

Parameter	Type	Required	Description
usernames	Array	Yes	List of Instagram usernames, profile URLs, or profile IDs
includeAboutSection	Boolean	No	Extract about account information (default: false)

Input Examples

Single username:

{
  "usernames": ["nasa"]
}

Multiple usernames:

{
  "usernames": ["nasa", "spacex", "elonmusk"]
}

Profile URLs:

{
  "usernames": [
    "https://www.instagram.com/nasa/",
    "https://www.instagram.com/spacex/"
  ]
}

With about section:

{
  "usernames": ["nasa"],
  "includeAboutSection": true
}

Usage

Local Development

Run the scraper with test input:

$npm start

Custom Input

1. Modify test_input.json with your desired usernames
2. Run npm start

Programmatic Usage

import { Actor } from 'apify';

await Actor.init();

const input = {
    usernames: ['nasa', 'spacex'],
    includeAboutSection: false
};

await Actor.setInput(input);
// Run your scraper logic here

await Actor.exit();

Output Data

The scraper saves data to the default dataset. You can access it:

• In storage folder: ./storage/datasets/default/
• Via Apify API: If running on Apify platform
• Programmatically: Using Dataset.getData()

Export Formats

The data can be exported in various formats:

• JSON
• CSV
• Excel
• XML
• RSS
• HTML Table

How It Works

1. Session Management:

   • Uses rotating session cookies to avoid rate limits
   • Supports multiple sessions for better performance

2. API Calls:

   • Fetches profile data from Instagram's web API
   • Retrieves posts using GraphQL queries
   • Gets related profiles and IGTV videos

3. Data Transformation:

   • Converts raw API data to standardized format
   • Extracts hashtags and mentions from captions
   • Processes media URLs and metadata

4. Rate Limiting:

   • Implements delays between requests
   • Rotates sessions to distribute load
   • Handles rate limit responses gracefully

Features Comparison with Apify Actor

Feature	This Scraper	Apify Actor
Profile Information	✅	✅
Latest Posts (12)	✅	✅
IGTV Videos (12)	✅	✅
Related Profiles	✅	✅
External URLs	✅	✅
About Section	✅	✅
Multiple Profiles	✅	✅
Session Rotation	✅	✅

Troubleshooting

No data returned

• Check cookies: Ensure your session cookies are valid
• Verify username: Make sure the username exists
• Check rate limits: You may be rate-limited, try using more sessions

Rate limiting

• Add more sessions to cookies.json
• Increase delays in the code
• Reduce number of profiles scraped at once

Missing data fields

• Some profiles may have privacy settings
• IGTV videos only available if account has them
• Related profiles depend on Instagram's algorithm

Getting Instagram Session Cookies

1. Open Instagram in your browser
2. Log in to your account
3. Open Developer Tools (F12)
4. Go to Application/Storage → Cookies → instagram.com
5. Copy the entire cookie string
6. Paste into cookies.json

Important cookies:

• sessionid - Your session ID
• csrftoken - CSRF token
• ds_user_id - Your user ID

Best Practices

1. Use multiple sessions for better performance
2. Don't scrape too fast to avoid rate limits
3. Rotate sessions between requests
4. Monitor for errors and adjust delays
5. Respect Instagram's ToS and privacy policies

Legal & Ethical Considerations

• Only scrape public data
• Respect user privacy
• Follow Instagram's Terms of Service
• Don't use for spam or harassment
• Comply with GDPR and data protection laws

Dependencies

• apify - Apify SDK for web scraping
• crawlee - Web crawling and scraping library
• cheerio - HTML parsing (for fallback)
• node-fetch - HTTP client

License

Apache-2.0

Author

Nomanriaz786

Support

For issues and questions:

• Create an issue in the repository
• Check existing issues for solutions
• Review the code documentation

Changelog

Version 1.0.0

• Initial release
• Profile data extraction
• Latest posts extraction
• IGTV videos extraction
• Related profiles extraction
• Session rotation
• Multiple input formats support

Roadmap

• Add support for stories
• Add support for highlights
• Add follower/following list extraction
• Add post comments extraction
• Improve error handling
• Add proxy support
• Add more detailed logging

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

Acknowledgments

• Inspired by Apify's Instagram Profile Scraper
• Built with Apify SDK and Crawlee
• Thanks to the open-source community