Instagram Profile Scraper

Extract public Instagram profile metadata and recent posts for creator research, influencer lists, and competitor social intelligence. Built with TypeScript, Crawlee CheerioCrawler (HTTP-first, no browser), and multi-strategy extraction (meta tags, JSON-LD, embedded GraphQL/_sharedData, DOM).

Run it on the Apify platform for scheduling, API access, residential proxy rotation, monitoring, and dataset exports (JSON, CSV, Excel).

Legal / ToS notice: This Actor is intended for publicly available data only. Respect Instagram Terms of Use, robots rules, and applicable privacy laws (GDPR/CCPA). Do not use this tool to scrape private content, bypass authentication, or harass users. You are responsible for how you use the data.

What does Instagram Profile Scraper do?

Given usernames and/or profile URLs, this Actor:

1. Deduplicates usernames (from usernames + profileUrls)
2. Loads each public profile page over HTTP (Cheerio)
3. Extracts profile details: username, full name, bio, followers, following, posts count, verified/business/private flags, external URL, profile picture
4. Optionally extracts recent posts: shortcode, URL, caption, likes, comments, timestamp, video flag, display image
5. Handles private / blocked / not-found profiles gracefully (dataset row with error)
6. Pushes dataset items (type: "profile" and type: "post")
7. Exports PROFILES.csv and SUMMARY.md to the key-value store

Why use this Actor?

• Influencer research — build creator lists with follower counts and bios
• Competitor social intel — track public brand accounts and posting patterns
• Lead generation — collect public business profile metadata (external links)
• Automation-ready — JSON dataset + CSV export + Apify API / webhooks

Proxy necessity (important)

Instagram aggressively blocks scrapers, especially datacenter IPs. For production runs:

Setting	Recommendation
Proxy	Apify Proxy RESIDENTIAL (default in input schema)
maxConcurrency	1–2
requestDelayMs	1500–3000+
maxRequestRetries	3+

Without residential proxies you will often see login walls, empty shells, or HTTP 429/403. The Actor detects these and writes an error on the profile record instead of crashing.

How to scrape Instagram profiles

1. Open the Actor in Apify Console
2. Enter usernames (e.g. natgeo) and/or full profile URLs
3. Choose results type: details, posts, or details_and_posts
4. Keep residential proxy enabled
5. Click Start
6. Download results from the Dataset tab or key-value store (PROFILES.csv, SUMMARY.md)

Input

Field	Type	Default	Description
usernames	string[]	["natgeo","instagram"]	Usernames without @
profileUrls	string[]	[]	Full profile URLs
resultsType	string	"details"	details | posts | details_and_posts
resultsLimit	integer	12	Max recent posts per profile
maxConcurrency	integer	2	Concurrent requests (keep low)
maxRequestRetries	integer	3	HTTP retry count
requestDelayMs	integer	1500	Used for rate limiting
proxyConfiguration	object	Residential on	Apify Proxy settings

Example:

{
    "usernames": ["natgeo", "instagram"],
    "profileUrls": [],
    "resultsType": "details",
    "resultsLimit": 12,
    "maxConcurrency": 2,
    "maxRequestRetries": 3,
    "requestDelayMs": 1500,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}

See examples/input.json.

Output

Profile record

{
    "type": "profile",
    "username": "natgeo",
    "fullName": "National Geographic",
    "biography": "Experience the world through the eyes of National Geographic photographers.",
    "followers": 281000000,
    "following": 140,
    "postsCount": 28500,
    "isVerified": true,
    "isBusiness": true,
    "isPrivate": false,
    "externalUrl": "https://www.nationalgeographic.com/",
    "profilePicUrl": "https://…/profile.jpg",
    "profileUrl": "https://www.instagram.com/natgeo/",
    "scrapedAt": "2026-07-13T12:00:00.000Z",
    "error": null
}

Post record

{
    "type": "post",
    "username": "natgeo",
    "shortcode": "CxAbc12345",
    "url": "https://www.instagram.com/p/CxAbc12345/",
    "caption": "A lion rests on the savanna at sunset. #wildlife",
    "likes": 450000,
    "comments": 3200,
    "timestamp": "2023-09-18T02:40:00.000Z",
    "isVideo": false,
    "displayUrl": "https://…/media.jpg",
    "profileUrl": "https://www.instagram.com/natgeo/",
    "scrapedAt": "2026-07-13T12:00:00.000Z",
    "error": null
}

Key-value store exports

Key	Description
PROFILES.csv	CSV of all type=profile rows
SUMMARY.md	Markdown summary of profiles + sample posts

Extraction strategy

Instagram does not offer a stable public HTML API. This Actor tries, in order:

1. Open Graph / meta description — often includes follower/following/posts counts
2. JSON-LD (ProfilePage / Person) when present
3. Embedded page JSON — classic window._sharedData, GraphQL-like user objects, timeline media edges
4. DOM fallbacks — private badge, verified icon, /p/{shortcode} anchors

Optional JSON endpoints such as ?__a=1&__d=dis are intentionally not required; they are frequently blocked. The HTML/embedded-JSON approach is more resilient for public profiles when combined with residential proxies.

Limitations

• Heavy blocking — Instagram may return login walls, challenges, or empty app shells even with proxies
• Public data only — private accounts yield limited metadata and no posts
• No login / no cookies — this Actor does not authenticate as a user
• No stories, reels feed, DMs, or full archive — recent posts from the profile page only
• Counts may be approximate — meta descriptions sometimes use abbreviated figures (281M)
• Layout changes — Instagram changes markup frequently; extraction methods may need updates
• Rate limits — high concurrency will increase failures; prefer slow + residential
• Compliance — bulk scraping may violate platform ToS; use responsibly and for legitimate purposes

Pricing

Charged via Pay-per-event on the primary dataset item (apify-default-dataset-item) at $0.005 per profile or post result, plus a small actor-start fee.

Development

cd factory/instagram-profile-scraper
npm install
npm test
npm run build
# local run (needs APIFY_TOKEN for residential proxy):
apify run

Fixture-based unit and integration tests live in test/ (no live Instagram calls required for CI).

Changelog

1.0.0

• Initial release: profile details + optional posts, CSV/Markdown export, residential proxy default, graceful private/block handling

License

Apache-2.0