# Instagram Related Profiles Scraper (`trend_wizard/instagram-related-profiles-scraper`) Actor

Discovers Instagram profiles related to seed accounts via the public suggested-profiles surface, enriches them with public data only, filters them, and exports a stable dataset and XLSX report.

- **URL**: https://apify.com/trend_wizard/instagram-related-profiles-scraper.md
- **Developed by:** [Yashwardhan](https://apify.com/trend_wizard) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 📸 Instagram Related Profiles Scraper

> Discover public Instagram profiles related to seed accounts and enrich them with a controlled, production-oriented pipeline — without Instagram login credentials.

### ✨ What this Actor does

- 🔎 Related-profile discovery from public Instagram information
- 🧭 Supports search depth 1 and 2
- 🧹 Global deduplication before enrichment
- 📏 Hard output cap through `maxCountExpansion`
- 👤 Profile identity, bio, followers, following, verification and profile picture URL
- 📊 Explicit/confirmed total post count when Instagram exposes one
- 🎬 `Has Reels` presence only; no Reel total is fetched
- 🔗 `Bio Links (URLs)` and `Bio Link Titles` plus the first usable `External URL`
- 📧 Public email and 📞 public phone where exposed
- 📍 Public physical address and tagged location where exposed
- 📗 Excel output with a fixed 21-column Profiles schema
- 🚫 No Instagram login, username/password or user-supplied cookies required

### 💰 Proxy-cost optimized acquisition

The current build is designed around the fact that Residential proxy data transfer is the dominant variable cost.

#### Normal path

```text
Seed
 ↓
Browser-first related-profile discovery
 ↓
Deduplicate + cap
 ↓
Lightweight HTTP profile enrichment
 ↓
Normalize
 ↓
Dataset / Excel
```

#### Cost-capped enrichment fallback

```text
One anonymous structured profile-JSON request
 ↓ only if it fails or lacks useful fields
One Playwright APIRequestContext request to the same JSON endpoint
 ↓
Normalize available fields; keep unavailable fields as N/A
```

The enrichment fallback is a request API, not a browser page: it does not open a profile URL, render a profile, execute page JavaScript, scroll, or download profile HTML. The enrichment ceiling is two structured requests per profile. If both are gated, discovery-backed identity fields are retained and the run summary reports enrichment failure separately.

Discovery is browser-based: the `xig_user_by_igid_v2` response is the primary source and the DOM carousel is the supplement. Its discovery behavior is unchanged by this enrichment-only optimization.

#### Cost-saving measures in this build

- Browser-only related-profile discovery
- One structured HTTP profile request plus one Playwright API-request fallback only when needed
- No profile-document HTML downloads, page rendering, or scrolling during enrichment
- Compact persistent cache without duplicate raw-profile JSON blobs
- One reusable Residential proxy session per run
- DOM carousel exploration skipped when `xig_user_by_igid_v2` already satisfies a capped request
- gzip/Brotli response compression negotiation
- Browser startup deferred until fallback is needed
- Browser discovery blocks images, media, fonts, stylesheets and text tracks
- No post crawling or media downloads for the requested schema
- No Reel pagination; `has_reels` remains presence-only
- One response can satisfy both discovery and later profile enrichment during the same run
- Global deduplication/capping remains before enrichment
- Minimal retries for transient failures only
- Compact in-run and optional cross-run caches
- No crawling of external bio-link websites
- Run-level proxy request/byte/cache instrumentation

### 🚀 Setup

```bash
python -m venv .venv
```

#### Windows

```powershell
.venv\Scripts\activate
```

#### macOS / Linux

```bash
source .venv/bin/activate
```

Install dependencies:

```bash
pip install -r requirements.txt
playwright install chromium
```

### ▶️ Run locally

Using the Apify CLI:

```powershell
apify run --input-file .\examples\input.json
```

Or:

```powershell
python -m src.main --input-file .\examples\input.json
```

### 📝 Input

The current input field names are:

```json
{
  "startUsernames": ["virat.kohli"],
  "searchDepth": "1",
  "maxCountExpansion": 70,
  "enrichmentConcurrency": 12,
  "discoveryConcurrency": 5,
  "clearSavedData": true,
  "profileCacheTtlSeconds": 21600,
  "relatedCacheTtlSeconds": 3600,
  "cacheMaxEntries": 1000,
  "httpMaxRetries": 1
}
```

#### Key controls

| Input | Default | Purpose |
|---|---:|---|
| `startUsernames` | required | 1–5 seed usernames |
| `searchDepth` | `1` | `1` = direct related profiles, `2` = expand Layer 1 profiles |
| `maxCountExpansion` | `0` | Hard ceiling on discovered/enriched profiles; `0` means no explicit cap |
| `enrichmentConcurrency` | `12` | Concurrent profile enrichment workers |
| `discoveryConcurrency` | `5` | Concurrent discovery workers |
| `clearSavedData` | `true` | Clears normal actor checkpoints/lists for a clean run |
| `profileCacheTtlSeconds` | `21600` | Cross-run compact profile-cache lifetime; `0` disables it |
| `relatedCacheTtlSeconds` | `3600` | Cross-run related-profile cache lifetime; `0` disables it |
| `cacheMaxEntries` | `1000` | Maximum cached profiles and relation sets |
| `httpMaxRetries` | `1` | Attempts for transient HTTP failures; capped at `2` |

### 🧪 Proxy-cost benchmark

A ready-to-run fresh benchmark input is included:

```powershell
apify run --input-file .\examples\proxy_cost_benchmark_fresh.json
```

That benchmark disables cross-run cache reuse so that the run measures the browser-discovery and lightweight-enrichment network behavior rather than getting an artificially low value from previously cached accounts.

For your earlier 70-result comparison, keep the seed, `searchDepth`, cap and concurrency settings identical to the old run.

#### Target

Your previous benchmark was about:

```text
70 results
0.17 GB Residential proxy data
$1.363 Residential proxy cost
$1.409 total platform usage
```

At the same proxy price, a $5/1,000 total-cost target implies roughly:

```text
≤ ~0.54 MB Residential proxy traffic per successful result
```

The new actor reports diagnostic efficiency in `RUN_SUMMARY.proxy_metrics`.

### 📈 Run Summary proxy metrics

The `RUN_SUMMARY` object now contains a `proxy_metrics` section with:

```text
http_requests
http_retries
http_bytes_estimate
http_cache_hits
http_related_hits
http_profile_hits
browser_fallbacks
browser_requests
browser_blocked_requests
browser_bytes_estimate
browser_navigation_attempts
cross_run_cache_hits
cross_run_cache_misses
proxy_requests_estimate
proxy_bytes_estimate
proxy_mb_estimate
proxy_mb_per_result_estimate
proxy_requests_per_result_estimate
```

These are diagnostics, not billing statements. The Apify Usage page is the source of truth for billed Residential GB and platform cost.

### 📊 Output

The Profiles sheet is a fixed 21-field contract:

1. Account
2. Username
3. Profile URL
4. Source
5. User ID
6. Full Name
7. Biography
8. Profile Picture
9. Followers Count
10. Following Count
11. Bio Links (URLs)
12. Bio Link Titles
13. Total Posts (Confirmed)
14. Has Reels
15. Is Verified
16. Account Type
17. Email
18. Phone
19. External URL
20. Physical Address
21. Tagged Location

The workbook contains:

- 📊 `Profiles`
- 📋 `Run Summary`
- ⏭️ `Skipped Accounts`

### 🔗 Bio Links vs External URL

`Bio Links (URLs)` lists every bio link URL exposed by the public profile response in one cell, separated by `|` (a comma is not used because URLs and titles can contain commas). `Bio Link Titles` lists the matching titles in the same order, with an empty slot for a link that has no title. Both are `N/A` when the profile has no bio links. Instagram's other per-link properties (redirect URL, link ID, thumbnail, colour, ...) are intentionally not exported; they previously caused ~50 flattened `bio_links/N/...` columns in Apify's table export.

`External URL` contains the first usable structured external URL.

No external website is opened merely to enrich these fields.

### 📊 Confirmed posts and Reels

`Total Posts (Confirmed)` is not calculated from the number of sampled media objects.

`Has Reels` is presence-only. The actor does not enumerate Reel pages or calculate a Reel total.

### 🔐 Login and privacy boundary

The actor is designed around public Instagram information and does not require Instagram login credentials. Contact and location fields are populated only when the corresponding public information is exposed by the source data.

### ❓ Troubleshooting

#### HTTP enrichment requests are gated

This can happen when Instagram restricts anonymous structured endpoints. The Actor tries one low-bandwidth HTTP JSON request and then one Playwright APIRequestContext request to the same endpoint. It does not download profile HTML or navigate to the discovered profile as an enrichment fallback. Check `RUN_SUMMARY.proxy_metrics.enrichment_profiles_failed`, `enrichment_profiles_playwright_fallback`, and `enrichment_field_counts` to distinguish request failures from fields Instagram did expose.

#### Residential proxy usage is still high

Check these together:

1. Apify Usage → Residential proxy GB
2. `RUN_SUMMARY.proxy_metrics.proxy_mb_estimate`
3. `proxy_mb_per_result_estimate`
4. `browser_fallbacks`
5. `browser_requests`
6. `http_retries`

Browser navigation is expected during related-profile discovery. A high `proxy_mb_per_result_estimate` indicates that the browser discovery or HTTP enrichment payloads are larger than expected.

### 📄 More detail

See `PROXY_OPTIMIZATION_NOTES.md` for the optimization matrix and a controlled benchmarking procedure.

# Actor input Schema

## `startUsernames` (type: `array`):

1 to 5 Instagram usernames or public profile URLs.

## `searchDepth` (type: `string`):

1 = direct related profiles. 2 = also expand related profiles discovered at depth 1.

## `maxCountExpansion` (type: `integer`):

Hard ceiling on discovered/enriched profiles. 0 = no explicit cap.

## `enrichmentConcurrency` (type: `integer`):

Number of public profile pages enriched concurrently.

## `discoveryConcurrency` (type: `integer`):

Number of related-profile discovery requests run concurrently.

## `profileCacheTtlSeconds` (type: `integer`):

Reuse compact public profile results from previous runs for this many seconds. 0 disables the cross-run profile cache.

## `relatedCacheTtlSeconds` (type: `integer`):

Reuse compact related-profile usernames from previous runs for this many seconds. 0 disables the cross-run related-profile cache.

## `cacheMaxEntries` (type: `integer`):

Maximum number of compact profile and related-profile entries kept in the persistent optimization cache.

## `httpMaxRetries` (type: `integer`):

Maximum attempts for transient HTTP failures. Keep at 1 to minimize proxy traffic; the actor never blindly retries login/challenge/404/429 responses.

## `clearSavedData` (type: `boolean`):

Clear previously saved actor data before starting the run.

## `enableOfflineMode` (type: `boolean`):

Reuse previously saved enrichment records without making new Instagram requests.

## `proxySettings` (type: `object`):

Select Apify Proxy or provide custom proxy URLs for Instagram requests.

## Actor input object example

```json
{
  "startUsernames": [
    "instagram"
  ],
  "searchDepth": "1",
  "maxCountExpansion": 0,
  "enrichmentConcurrency": 12,
  "discoveryConcurrency": 5,
  "profileCacheTtlSeconds": 21600,
  "relatedCacheTtlSeconds": 3600,
  "cacheMaxEntries": 1000,
  "httpMaxRetries": 1,
  "clearSavedData": true,
  "enableOfflineMode": false,
  "proxySettings": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset containing the scraped profile records.

## `excel` (type: `string`):

Excel workbook containing profiles, run summary, and skipped accounts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUsernames": [
        "instagram"
    ],
    "proxySettings": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("trend_wizard/instagram-related-profiles-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUsernames": ["instagram"],
    "proxySettings": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("trend_wizard/instagram-related-profiles-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUsernames": [
    "instagram"
  ],
  "proxySettings": {
    "useApifyProxy": true
  }
}' |
apify call trend_wizard/instagram-related-profiles-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,trend_wizard/instagram-related-profiles-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/no0cYkJyvHTjrMMvD/builds/HZp4bPdLHat7qrDTe/openapi.json