📸 Instagram Related Profiles Scraper

Discover public Instagram profiles related to seed accounts and enrich them with a controlled, production-oriented pipeline — without Instagram login credentials.

✨ What this Actor does

• 🔎 Related-profile discovery from public Instagram information
• 🧭 Supports search depth 1 and 2
• 🧹 Global deduplication before enrichment
• 📏 Hard output cap through maxCountExpansion
• 👤 Profile identity, bio, followers, following, verification and profile picture URL
• 📊 Explicit/confirmed total post count when Instagram exposes one
• 🎬 Has Reels presence only; no Reel total is fetched
• 🔗 Bio Links (URLs) and Bio Link Titles plus the first usable External URL
• 📧 Public email and 📞 public phone where exposed
• 📍 Public physical address and tagged location where exposed
• 📗 Excel output with a fixed 21-column Profiles schema
• 🚫 No Instagram login, username/password or user-supplied cookies required

💰 Proxy-cost optimized acquisition

The current build is designed around the fact that Residential proxy data transfer is the dominant variable cost.

Normal path

Seed
 ↓
Browser-first related-profile discovery
 ↓
Deduplicate + cap
 ↓
Lightweight HTTP profile enrichment
 ↓
Normalize
 ↓
Dataset / Excel

Cost-capped enrichment fallback

One anonymous structured profile-JSON request
 ↓ only if it fails or lacks useful fields
One Playwright APIRequestContext request to the same JSON endpoint
 ↓
Normalize available fields; keep unavailable fields as N/A

The enrichment fallback is a request API, not a browser page: it does not open a profile URL, render a profile, execute page JavaScript, scroll, or download profile HTML. The enrichment ceiling is two structured requests per profile. If both are gated, discovery-backed identity fields are retained and the run summary reports enrichment failure separately.

Discovery is browser-based: the xig_user_by_igid_v2 response is the primary source and the DOM carousel is the supplement. Its discovery behavior is unchanged by this enrichment-only optimization.

Cost-saving measures in this build

• Browser-only related-profile discovery
• One structured HTTP profile request plus one Playwright API-request fallback only when needed
• No profile-document HTML downloads, page rendering, or scrolling during enrichment
• Compact persistent cache without duplicate raw-profile JSON blobs
• One reusable Residential proxy session per run
• DOM carousel exploration skipped when xig_user_by_igid_v2 already satisfies a capped request
• gzip/Brotli response compression negotiation
• Browser startup deferred until fallback is needed
• Browser discovery blocks images, media, fonts, stylesheets and text tracks
• No post crawling or media downloads for the requested schema
• No Reel pagination; has_reels remains presence-only
• One response can satisfy both discovery and later profile enrichment during the same run
• Global deduplication/capping remains before enrichment
• Minimal retries for transient failures only
• Compact in-run and optional cross-run caches
• No crawling of external bio-link websites
• Run-level proxy request/byte/cache instrumentation

🚀 Setup

$python -m venv .venv

Windows

.venv\Scripts\activate

macOS / Linux

$source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt
playwright install chromium

▶️ Run locally

Using the Apify CLI:

apify run --input-file .\examples\input.json

Or:

python -m src.main --input-file .\examples\input.json

📝 Input

The current input field names are:

{
  "startUsernames": ["virat.kohli"],
  "searchDepth": "1",
  "maxCountExpansion": 70,
  "enrichmentConcurrency": 12,
  "discoveryConcurrency": 5,
  "clearSavedData": true,
  "profileCacheTtlSeconds": 21600,
  "relatedCacheTtlSeconds": 3600,
  "cacheMaxEntries": 1000,
  "httpMaxRetries": 1
}

Key controls

Input	Default	Purpose
startUsernames	required	1–5 seed usernames
searchDepth	1	1 = direct related profiles, 2 = expand Layer 1 profiles
maxCountExpansion	0	Hard ceiling on discovered/enriched profiles; 0 means no explicit cap
enrichmentConcurrency	12	Concurrent profile enrichment workers
discoveryConcurrency	5	Concurrent discovery workers
clearSavedData	true	Clears normal actor checkpoints/lists for a clean run
profileCacheTtlSeconds	21600	Cross-run compact profile-cache lifetime; 0 disables it
relatedCacheTtlSeconds	3600	Cross-run related-profile cache lifetime; 0 disables it
cacheMaxEntries	1000	Maximum cached profiles and relation sets
httpMaxRetries	1	Attempts for transient HTTP failures; capped at 2

🧪 Proxy-cost benchmark

A ready-to-run fresh benchmark input is included:

apify run --input-file .\examples\proxy_cost_benchmark_fresh.json

That benchmark disables cross-run cache reuse so that the run measures the browser-discovery and lightweight-enrichment network behavior rather than getting an artificially low value from previously cached accounts.

For your earlier 70-result comparison, keep the seed, searchDepth, cap and concurrency settings identical to the old run.

Target

Your previous benchmark was about:

70 results
0.17 GB Residential proxy data
$1.363 Residential proxy cost
$1.409 total platform usage

At the same proxy price, a $5/1,000 total-cost target implies roughly:

≤ ~0.54 MB Residential proxy traffic per successful result

The new actor reports diagnostic efficiency in RUN_SUMMARY.proxy_metrics.

📈 Run Summary proxy metrics

The RUN_SUMMARY object now contains a proxy_metrics section with:

http_requests
http_retries
http_bytes_estimate
http_cache_hits
http_related_hits
http_profile_hits
browser_fallbacks
browser_requests
browser_blocked_requests
browser_bytes_estimate
browser_navigation_attempts
cross_run_cache_hits
cross_run_cache_misses
proxy_requests_estimate
proxy_bytes_estimate
proxy_mb_estimate
proxy_mb_per_result_estimate
proxy_requests_per_result_estimate

These are diagnostics, not billing statements. The Apify Usage page is the source of truth for billed Residential GB and platform cost.

📊 Output

The Profiles sheet is a fixed 21-field contract:

1. Account
2. Username
3. Profile URL
4. Source
5. User ID
6. Full Name
7. Biography
8. Profile Picture
9. Followers Count
10. Following Count
11. Bio Links (URLs)
12. Bio Link Titles
13. Total Posts (Confirmed)
14. Has Reels
15. Is Verified
16. Account Type
17. Email
18. Phone
19. External URL
20. Physical Address
21. Tagged Location

The workbook contains:

• 📊 Profiles
• 📋 Run Summary
• ⏭️ Skipped Accounts

🔗 Bio Links vs External URL

Bio Links (URLs) lists every bio link URL exposed by the public profile response in one cell, separated by | (a comma is not used because URLs and titles can contain commas). Bio Link Titles lists the matching titles in the same order, with an empty slot for a link that has no title. Both are N/A when the profile has no bio links. Instagram's other per-link properties (redirect URL, link ID, thumbnail, colour, ...) are intentionally not exported; they previously caused ~50 flattened bio_links/N/... columns in Apify's table export.

External URL contains the first usable structured external URL.

No external website is opened merely to enrich these fields.

📊 Confirmed posts and Reels

Total Posts (Confirmed) is not calculated from the number of sampled media objects.

Has Reels is presence-only. The actor does not enumerate Reel pages or calculate a Reel total.

🔐 Login and privacy boundary

The actor is designed around public Instagram information and does not require Instagram login credentials. Contact and location fields are populated only when the corresponding public information is exposed by the source data.

❓ Troubleshooting

HTTP enrichment requests are gated

This can happen when Instagram restricts anonymous structured endpoints. The Actor tries one low-bandwidth HTTP JSON request and then one Playwright APIRequestContext request to the same endpoint. It does not download profile HTML or navigate to the discovered profile as an enrichment fallback. Check RUN_SUMMARY.proxy_metrics.enrichment_profiles_failed, enrichment_profiles_playwright_fallback, and enrichment_field_counts to distinguish request failures from fields Instagram did expose.

Residential proxy usage is still high

Check these together:

1. Apify Usage → Residential proxy GB
2. RUN_SUMMARY.proxy_metrics.proxy_mb_estimate
3. proxy_mb_per_result_estimate
4. browser_fallbacks
5. browser_requests
6. http_retries

Browser navigation is expected during related-profile discovery. A high proxy_mb_per_result_estimate indicates that the browser discovery or HTTP enrichment payloads are larger than expected.

📄 More detail

See PROXY_OPTIMIZATION_NOTES.md for the optimization matrix and a controlled benchmarking procedure.