Instagram Profile Crawler

Recursively discover and scrape Instagram profiles starting from seed accounts. The crawler explores Instagram's "related profiles" to build a network of connected accounts.

How It Works

1. Start with one or more seed profiles
2. Scrape each profile's data including their related profiles
3. Add related profiles to the queue (respecting depth limits)
4. Continue until maxProfiles is reached
5. Each result includes crawl metadata showing how it was discovered

instagram (seed, depth 0)
├── creator_a (depth 1, from: instagram)
│   ├── creator_x (depth 2, from: creator_a)
│   └── creator_y (depth 2, from: creator_a)
├── creator_b (depth 1, from: instagram)
└── creator_c (depth 1, from: instagram)

Features

• Recursive discovery - Automatically finds new profiles via related profiles
• BFS traversal - Explores breadth-first to get diverse results
• Deduplication - Never scrapes the same profile twice
• Crawl metadata - Track depth and discovery path for each profile
• Configurable limits - Set max profiles and max depth
• Filters - Skip private profiles, filter by minimum followers
• Date filtering - Only include recent posts
• Proxy support - Built-in Apify Proxy integration

Input

Field	Type	Description
seedUsernames	array	Required. Starting Instagram usernames to crawl from
maxProfiles	integer	Maximum total profiles to scrape (default: 100)
maxDepth	integer	Maximum recursion depth from seeds (default: 3)
proxyConfiguration	object	Proxy settings. Highly recommended for crawling
maxRetries	integer	Max retry attempts per profile (default: 3)
delayBetweenRequests	integer	Delay in ms between requests (default: 2000)
onlyPostsNewerThan	string	Filter posts by date (30 days, 6 months, YYYY-MM-DD)
minFollowers	integer	Skip profiles with fewer followers (default: 0)
skipPrivateProfiles	boolean	Skip private profiles entirely (default: true)

Example Input

{
  "seedUsernames": ["instagram", "cristiano"],
  "maxProfiles": 500,
  "maxDepth": 2,
  "minFollowers": 10000,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  },
  "onlyPostsNewerThan": "6 months"
}

Output

Each profile includes standard data plus crawl metadata:

{
  "username": "creator_account",
  "userId": "12345678",
  "fullName": "Creator Name",
  "biography": "Content creator...",
  "followerCount": 150000,
  "followingCount": 500,
  "mediaCount": 200,
  "isVerified": true,
  "isPrivate": false,
  "isBusiness": false,
  "isProfessional": true,
  "profilePicUrl": "https://...",
  "profilePicUrlHd": "https://...",
  "externalUrl": "https://linktr.ee/creator",
  "bioLinks": [...],
  "pronouns": ["she", "her"],
  "categoryName": "Digital Creator",
  "highlightReelCount": 5,
  "posts": [...],
  "relatedProfiles": [...],
  "engagementRate": 3.2,
  "scrapedAt": "2025-01-13T12:00:00.000Z",
  "crawlDepth": 1,
  "discoveredFrom": "instagram"
}

Crawl Metadata

Field	Description
crawlDepth	How many hops from seed (0 = seed profile)
discoveredFrom	Which profile's related profiles led to this one (null for seeds)

Error Response

{
  "username": "private_user",
  "error": "private",
  "message": "Profile is private: private_user",
  "crawlDepth": 2,
  "discoveredFrom": "some_creator"
}

Depth Explained

Depth	Description
0	Seed profiles you provided
1	Profiles related to seeds
2	Profiles related to depth-1 profiles
3	Profiles related to depth-2 profiles

Setting maxDepth: 1 only scrapes seeds and their immediate related profiles.

Proxy Configuration

Strongly recommended for crawling multiple profiles:

{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Residential proxies are recommended for better success rates when scraping many profiles.

Performance Tips

1. Start small - Test with maxProfiles: 10 first
2. Use delays - Keep delayBetweenRequests at 2000ms or higher
3. Use proxies - Essential for crawling more than a few profiles
4. Filter early - Use minFollowers to skip irrelevant profiles
5. Limit depth - maxDepth: 2 is usually sufficient

Cost Estimation

• ~1 request per profile
• Time: ~3 seconds per profile (including delays)
• Memory: 256MB-512MB depending on maxProfiles

maxProfiles	Estimated Time	Estimated Cost
100	~5 minutes	~$0.10
500	~25 minutes	~$0.50
1000	~50 minutes	~$1.00

Use Cases

• Influencer discovery - Find creators in a niche starting from known accounts
• Competitor analysis - Map related accounts in your industry
• Network analysis - Understand connections between accounts
• Lead generation - Build lists of relevant profiles

Limitations

• Only public profiles can be fully scraped
• Related profiles may not be available for all accounts
• Maximum 12 recent posts per profile
• Instagram may limit related profiles shown

Local Development

# Install dependencies
pnpm install

# Run locally
echo '{"seedUsernames": ["instagram"], "maxProfiles": 5}' > storage/key_value_stores/default/INPUT.json
pnpm run start:dev

# Build
pnpm run build

Deploy to Apify

$apify push

License

ISC