One Instagram scraper for everything — profiles, contacts (emails & phones), posts, reels, comments, likers, followers, following, tagged, stories, highlights, hashtags, locations & search. No code: paste usernames or URLs, export to Excel, CSV, JSON or API. Public data needs no login.
All notable changes to this Actor will be documented in this file. Format
follows Keep a Changelog .
[2.0.84] - 2026-09-03
Fixed — the gated surface no longer fills the breaker's window
2.0.83 latched web_profile_info off once the run had seen it refuse, and said
plainly what it had not finished: 33 requests still reached a surface answering
0% of them, and throttle_circuit_open still fired. Two levers were named
there. Both are pulled here, and the second one is what mattered.
Rotation is not spent on a closed door. A retry inside request takes a
fresh exit IP, device and TLS bundle — the right answer to a blocked address,
the wrong one to a gated endpoint, and each rotation's 429 is counted as a
block by the breaker. request now takes a per-call max_attempts, and the
ladder passes 1 once this run has already watched the surface refuse. The first
attempt of a run keeps its full rotation, so an endpoint having a bad minute is
still absorbed rather than written off. 33 → 26.
The first look happens alone. The remaining 24 of those 26 were a single
concurrent wave: eight targets each spending a rotation before any verdict
existed to read. get_limited already documents this exact failure for
documents — "workers start together, so with the default concurrency of 8 every
one of them asked for the same unproven document before the first answer came
back" — and serialises the first read of an unproven surface. That discipline is
now available to any caller as IgClient.first_probe_of, and the ladder uses it.
A verdict either way opens the gate and it stays open, so a healthy run pays one
serialised lookup at its start and nothing after.
Measured across the same eight-profile job, four builds:
build web_profile_info attempts throttle circuit
2.0.6596 opened
2.0.6748opened(early exit)
2.0.6933opened(attempt-level latch)
2.0.7026opened(no rotation)
2.0.716**closed**(first look alone)
Sixteen times fewer requests to a surface that answers none of them, the same
eight records delivered, the same wall-clock (20.8s against 20.4s — serialising
one lookup costs nothing when it is one), and proxy down from $0.0267 to
$0.0179.
The circuit staying shut is the part that matters. It is what closed production
run bWCq3a6k5OgFPqHEl before its work began, and a run whose breaker never trips
over an address problem it does not have is a run that goes on to do what the
user asked.
[2.0.83] - 2026-09-03
Measured — web_profile_info is gated, not throttled, and the ladder argued with it
2.0.82 called this a throttle and cut the ladder short on that basis. The word
was wrong, and the difference matters.
Probed directly on 2026-09-03: users/web_profile_info answers HTTP 429 with
an empty body to everything. Two hosts, two usernames, cookie and none, and
every header combination tried — app-id, referer, X-ASBD-ID,
X-Requested-With, a full browser set. From Apify residential exits in two
countries and from an unrelated home connection. In the same second from the
same machine, topsearch, friendships/followers and get_badge_count all
answered 200, so it is neither the network nor the credential.
And it is not a queue that clears: on 2026-09-02 the same endpoint answered
29 of 38 (run fBYngOPYe3cFIFhmc). Something changed on Instagram's side
overnight. A 429 that every caller gets, always, is a gate wearing a rate
limit's status code.
The ladder therefore was not being unlucky — it was re-asking a closed door
once per target, and each 429 also counts as a block in the throttle circuit's
window. That is what closed production run bWCq3a6k5OgFPqHEl before its work
began.
The run now latches the surface off after _PROFILE_INFO_MISS_LIMIT refused
attempts, through the same surface_is_dead mechanism the tagged API already
uses. Per run, so it is a memory rather than a verdict: the next run tests
the surface again and picks it straight back up when the gate lifts.
The limit counts attempts, not targets, and that correction came from a
measurement rather than a design. The first version counted targets, and dev run
tqNm2OUC0QorGiIhu recorded missStreaks {web_profile_info: 8} beside
savedCalls {} — eight concurrent targets had all read the latch before any of
them finished to set it, so the first wave paid in full and one wave was the
whole run.
Measured across the same eight-profile job:
build web_profile_info attempts savedCalls
2.0.65 96 —
2.0.67 480(early exit only)
2.0.69 335(attempt-level latch)
Not finished. 33 requests still reach a surface that answers 0% of them, and
throttle_circuit_open still fires on that job: each ladder attempt carries
three in-request retries, so the floor of this design is the miss limit times
three. Declining to retry a 429 on a surface already known to be gated is the
next lever, and it is a change to shared retry policy that wants its own
measurement rather than another guess.
[2.0.82] - 2026-09-03
Fixed — the lookup ladder spent the run's block budget before the work began
Production run bWCq3a6k5OgFPqHEl, a paying customer scraping tagged posts for
two accounts. 24 of the run's 29 requests were web_profile_info being
throttled — the bootstrap ladder retrying two hosts across two rounds, three
in-request retries each. That is 24 blocks inside the throttle circuit's
25-sample window, so the circuit opened before the tagged walk had made a
single request. The API path never ran, the HTML fallback found nothing, and
the run reported "0 records — the user may have none". They ran it three times
at three different limits, then gave up on the mode and scraped ordinary posts
instead.
The ladder exempted throttles from its early exit on the stated grounds that
blocks "can differ per IP" — rounds after the first do rotate the exit IP. The
counter added in 2.0.78 put a number on that belief: zero recoveries in nine
round-1 double throttles (dev run PcxFz69KTkB2y4LJ8 over eight profiles, plus a
production smoke). The fresh IP did not once turn a double throttle into an
answer.
So a first round throttled on both hosts now ends the ladder, and the run falls
through to the rungs below it. Throttles only — blocks keep their full
ladder, because one observation is not a measurement, and the counter keeps
watching them. The skip is itself recorded as stoppedEarly, so it cannot read
as "this never happens" to whoever looks next, and afterFirstRound no longer
books a recovery that was never attempted.
Verified on the customer's own input, their third attempt, 2.0.21 against 2.0.66:
records 0->1000
web_profile_info attempts 24->12(still all throttled)
usertags API attempts 0->48(all successful)
throttle circuit opened -> never opened
ladder full -> stoppedEarly {"rate_limited":2}
The lookup is throttled just as hard. What changed is that it stops arguing with
Instagram about it and leaves the block budget for the work the user asked for —
the search rung answers with the pk, which is all the usertags feed needs.
[2.0.81] - 2026-09-03
Fixed — an emoji hashtag killed its own target with a Python traceback
Production run AFqINACf4ddA2EGGI, a paying customer. A search for
#SmirnoffMintyJamun came back from Instagram with three matches, one of them
smirnoffmintyjamun🍇. Two scraped, 23 records. The third died:
UnicodeEncodeError: 'latin-1' codec can't encode character '🍇'
HTTP headers are latin-1. hashtag_url(tag) builds the feed URL and it is sent
as the Referer, so a hashtag Instagram itself returned could not be requested.
Every referer this Actor sends is built the same way — from a hashtag, a
username, a location, a highlight id — none of them values it chose.
Fixed at the boundary rather than in the builder: _ig_headers puts every
referer through header_safe_url, which percent-encodes only what latin-1
cannot carry and leaves URL syntax alone. A builder added later cannot
reintroduce the crash by forgetting.
hashtag_url itself deliberately keeps the readable form. It also feeds a
target's raw_url, which reaches the user as their own input echoed back, and
.../tags/%D0%BA%D0%BE%D1%88%D0%BA%D0%B8/ is a worse thing to hand someone than
.../tags/кошки/ — a contract tests/test_line_reading has pinned since before
this bug existed, and the first version of this fix broke it.
[2.0.80] - 2026-09-03
Security — a live Instagram session cookie was committed in 18 load reports
tools/run_apify_load_suite.py wrote each scenario's input to
load_reports/<run>/<scenario>/input.json and then called the Actor with that
same file. It was therefore both the call input and the committed record, so
sessionCookies — a real sessionid for the account the suite runs as — went
into git in plaintext, in 18 files. The directory was not ignored.
Three layers, because scrubbing alone would only last until the next run:
The 18 files are scrubbed. The cookie names are kept and the values
dropped: which cookies a run carried is evidence about the scenario, the token
is not. They were re-redacted through the runner's own function, so a
historical record and a new one read identically.
The source is closed. The runner now writes two files: input.local.json,
the real input it calls the Actor with, and input.json, the record, with
secret values dropped. A future report cannot carry the token even if it is
committed.
load_reports/ is ignored. Committing a report stays possible on purpose
— git add -f <report dir> — because these reports are the measurement base
this codebase cites throughout. The redaction above is what makes doing so
safe. Already-tracked reports are unaffected: git ignores only untracked
files, so the 711 committed before this stay.
This does not un-leak the token. It remains in the repository's history — one
commit, 82b52a2, reachable by anyone with the repo. The only remedies are
invalidating that session (log out of it from Instagram's own security settings,
which is what actually kills a sessionid) or rewriting history. Both are the
owner's call; nothing here does either.
[2.0.79] - 2026-09-03
Retracted — "an identity-only object is accepted as the profile" was a misreading
Reported here earlier today as a defect: _profile_user_score gives
{username, id, full_name, profile_pic_url, is_private} a 7 against a threshold
of 4, so a five-field object is accepted as a profile and produces a hollow row.
It does not. _PROFILE_USER_SIGNAL_KEYS holds only substance fields
(edge_followed_by, biography, edge_owner_to_timeline_media, bio links,
external url, category, highlight count), and _profile_user_score returns 0
outright when a candidate carries none of them — below any threshold. username
is not a signal key; counting it as one is where the 7 came from. Checked by
running it, not by reading it again: identity-only scores 0 and
walk_for_profile_user refuses it.
The instrument added in 2.0.76 had already said so, and this is the point of
having built it: across every dev run of the day, documentOutcomes recorded
profile and id_only and never onceidentity_only. The thin rows all
came from the regex-id exit, which 2.0.76 fixed. No behaviour change here — the
defect was mine, not the code's.
Two guards left behind, because the property is load-bearing and was implicit:
the rejection of an identity-only candidate is now pinned, so lowering the
threshold or adding an identity field to the signal keys fails a test instead
of quietly reviving the bug this session chased twice;
the asymmetry between the two substance lists is pinned as deliberate. Writing
that test found the narrow door the report should have described: a candidate
carrying only a peripheral signal — external_url, highlight_reel_count,
bio_links, category_name, edge_related_profiles,
edge_felix_video_timeline — is accepted by the walk while
user_carries_profile calls it an address. That is not a hole. Such a lookup
is labelled identity_only in resolvedBy and record_is_chargeable does not
bill the row, so the system degrades correctly; widening either list to make
them agree would undo the protection added for the nine hollow rows of run
fBYngOPYe3cFIFhmc. The test fails if a future edit widens the wrong one, and
says which.
The two lists now name each other in the source, with the difference stated,
since mistaking one for the other is what produced the phantom in the first
place.
[2.0.78] - 2026-09-03
Added — the lookup ladder now reports whether its extra rounds earn their bytes
web_profile_info is retried over two hosts for up to _PROFILE_BOOTSTRAP_ROUNDS
rounds. One early exit exists: both hosts answering HTTP 400 in round 1 is a
serialisation failure no exit IP can fix, measured and commented as such. Blocks
and throttles deliberately keep the full ladder, on the stated grounds that
"those can differ per IP" — a belief that had never been measured and is the most
expensive thing the ladder does.
It could not be measured, because the throttle branch had no telemetry at all:
it caught LOAD_ERRORS, logged at debug level and continued. Dev run
7HdJAszP08UbXIaIr spent 36 of 36 attempts there and reported
refusalKinds: {} — the branch that ate the run was invisible in the record
meant to explain the ladder. Every refusal kind is now counted, and every round
contributes to afterFirstRound, which answers the conditional question totals
cannot: given both hosts refused this way in round 1, did a later round ever
answer?
Eight of eight: both hosts throttled in round 1, and the round after it — which
does rotate the proxy session — never recovered. 12 HTTP attempts per profile
(the ladder's 4, times the client's own in-request retries), 306 KB of wire per
profile spent entirely on 429 bodies.
No behaviour changed. One run at one moment through one proxy pool is not enough
to overturn a documented decision, and this session has already been wrong twice
from acting on a single reading. What is now true is that the question has a
counter: a few more runs of this shape, and extending the early exit to throttles
is arithmetic rather than argument.
Measured — the profile document is worse with a cookie, not better
The same reading corrects something recorded here on 2026-09-03. Comparing the
document rung across login states, after the needs_profile fix of 2.0.76:
2.0.63 signed out,3 accounts documentOutcomes {"signed_out":{"profile":3}} rows 10-12/46
2.0.65 signed in,8 accounts documentOutcomes {"signed_in":{"id_only":8}} rows thin
Signed out, the public profile page carries a parseable profile object. Signed
in, on the same accounts, it carries no user object at all — only an extractable
id — and all 8 paid for the full page (documentRefetches: 8) to find that out.
The earlier note that login state had been "ruled out" was drawn from pre-fix
runs, where both states stopped at the first id and so looked identical. They are
not identical, and the difference runs the counter-intuitive way: for Profile
details, a cookie makes this rung useless. Left as a measurement for now — acting
on it means choosing to read a signed-out document from a signed-in run, which
is a real behaviour change and wants its own evidence.
[2.0.77] - 2026-09-03
Fixed — the cost meter priced its largest transfers as if uncompressed
A byte-capped read is aborted mid-transfer, so libcurl never reports its wire
size, and those reads are the biggest transfers this Actor makes. The ledger
converted them with the run's own decompressed / wire ratio and fell back to
1.0 when that was unusable — documented as "an over-estimate, which is the
safe direction for a spend guard".
The spend guard is gone. This module's own first line says so: "This is a
meter. It measures; it does not decide." For a meter an over-estimate is not
caution, it is a wrong number, and it was wrong by a lot:
dev run 7HdJAszP08UbXIaIr priced 66% of its bytes at 1.0, while
tools/cost_repro priced the same pages at 5.9x — two cost models in one
repository disagreeing about the same page by ~6x;
the ledger's own docstring already recorded three real measurements — 4.39x,
4.83x, 5.81x — which the 1.0 fallback threw away.
Worse, the fallback fired exactly when it did most damage. decompressed / wire
was computed over all counted responses, so on a throttled run the sample is
dozens of small error bodies whose wire cost exceeds their content — a ratio
below 1, which was then floored to 1.0 and used. A body cannot expand: such a
sample is measuring per-request overhead, not compression, and it is now
rejected rather than floored.
Three tiers, best first, and the one used is reported as compressionSource
beside the number, because 1.0 used to mean both "measured, no compression"
and "nothing to measure":
capped_reads — a capped page read that finished inside its cap, or the full
re-read of a document whose slice missed its payload. Same population as the
aborted transfers, measured rather than assumed. That refetch is now marked as
the sample it always was, which is what makes this tier reachable on the runs
whose cost was mis-stated.
mixed — no such read, but the run's overall ratio is above 1.0.
measured_default — neither; MEASURED_COMPRESSION (4.39, the lowest of the
three real measurements, so the estimate errs towards a higher cost) stands in.
tools/cost_repro now reads that same constant instead of keeping its own 5.9.
One physical quantity had two constants in one repository, which is how the
disagreement arose; the tool's proxy figures rise ~34% as a result, which is the
conservative direction and the one the ledger claims for its numbers.
Verified on 2.0.64, three profiles, document-heavy:
compressionSource capped_reads
observedCompression 6.16(from 139896 wire bytes of comparable reads)
proxyUsd $0.0036
the same run under the old 1.0fallback: $0.0141
So that run's proxy cost had been over-stated 3.9x. Every margin figure taken
from a gated run's ledger was pessimistic by roughly the compression ratio.
[2.0.76] - 2026-09-03
Added — a profile row can say which rung produced it
Run fBYngOPYe3cFIFhmc saved 38 Profile-details rows and nine carried five fields
of forty-six. Nothing in the run said why, so finding out meant re-deriving the
lookup ladder by hand from endpoint counters — done twice, on two different
days, reaching a different conclusion each time. The second attempt ended in two
explanations that totals cannot separate: Instagram serving a shell to a
signed-in session, or the run being throttled into shells.
Two counters end that. profileLookup.resolvedBy names the rung that answered
(web_profile_info, search, document, or unresolved) and whether what it
answered with was a profile or only an identity — an address for one.
profileLookup.documentOutcomes says what each public-document read came back
as (profile, identity_only, id_only, wall, nothing), split by login
state, which is the axis that could not be settled from totals.
html_extract.user_carries_profile is the single definition of the difference,
so the measurement and any rule that acts on it cannot drift apart. It asks
presence, not truth: an account with 0 posts still reports 0, and reporting 0 is
what makes it a profile.
Fixed — the document rung stopped reading at the first id
The instrument answered on its first use, and both earlier guesses were wrong.
Dev run 7HdJAszP08UbXIaIr (signed in) and its signed-out control, three
business-category accounts each, both read:
Identical signed in and signed out, no wall, and the control had
skipReasons: {} — so neither a shell nor throttling. The document was a
full-size page and the stop was ours: _fetch_capped_document(has_payload=…)
was given a predicate that accepted a bare user id as "the payload has
arrived". The id appears at 393 216 / 327 680 bytes, the read stopped there, and
the extractor then found no user object to build a record from.
That is the third instance of one root cause, after the search rung and
search_answer_is_enough in 2.0.70: a sufficiency predicate that does not know
what the caller needs. An id is the whole answer for a followers walk and half a
record for Profile details. The caller's own flag is now threaded one rung
further down as needs_profile, and the same predicate that measures the
difference decides it.
Verified on the identical input, 2.0.62 against 2.0.63:
before resolvedBy {"document":{"identity_only":3}} rows 3/46 fields, followers None
after resolvedBy {"document":{"profile":3}} rows 10-12/46, followers 60589/18106/13892
documentRefetches: 2 is the price: two of the three needed the full page once
the capped slice no longer counted an id as enough. That is the cost this buys
the data with, and it is now visible in the same record.
[2.0.75] - 2026-09-03
Fixed — the capped-list warning was built on a signal that goes missing
2.0.72 reported a short follower list by comparing the rows against the
account's follower count. Dev run gsbtaefY5aXkzMLne measured that silent on the
exact case it was written for: @aisultan, 50 rows of the 100 asked, an account
with 245 861 followers — and skipReasons: {}, an unqualified
✅ Done — saved 50 records from 1URL
. The production bug, reproduced, with the fix for it
shipped and doing nothing.
The count was missing. web_profile_info answered HTTP 429 twelve times out of
twelve, the profile resolved through the cheap search rung, and a topsearch
user carries no follower count at all — only a rendered social_context of
"245K followers", which src/text_numbers refuses to turn into a number on
purpose ("expanding it would be a guess about a value we were not given"). So
the check asked how long the list was, got None, and said nothing. The signal
it depended on disappears precisely when the run is degraded, which is when the
warning matters.
Instagram states the cap itself. Measured live on 2026-09-03 with a cookie, on
the accounts of run 2C6aWIYxOWbBM5ivX:
account rows has_more should_limit next_max_id
elenapetitti (222)25 True False yes
adrien.armanet (2259)25 True False yes
gibsonhazard (182501)46 False True no
aisultan (245861)50 False True no
The two that stop early are exactly the two that came back short on that run.
should_limit_list_of_followers names them, needs no follower count, and lives
in a response the walk is already reading, so it cannot go missing the way the
count did. _paginate_friendships now returns it alongside the row count, and
the warning fires on Instagram's own word rather than on an inference.
Verified on 2.0.61:
⚠️ 👥 aisultan: Instagram capped this account's followers list at 50forthis session
, skipReasons: {relationship_list_truncated: 1},
and the cut-short sentence in the final message. The report gloss no longer
suggests a re-run reaches further, because measurement says it does not.
The search rung also normalises its user object now, like every other rung —
correct on its own terms, and explicitly not a fix for the above: the comment
there records that topsearch has no count to normalise.
[2.0.74] - 2026-09-03
Fixed — the run told a signed-in user to sign in
Dev run DYk8KKTpiEVNdlJYp installed eleven Instagram cookies, probed the session
successfully, hit sustained throttling, and then said, three times over:
🧯 Throttle circuit opened(22/25 recent requests throttled) … Add an
Instagram cookie for block-free runs.
⚠️ Instagram throttling wound this run down early … A cookie, or a re-run,
usually gets it.
⚠️ Throttling wound the run down early … add a cookie for full runs.
plus a zero-result line offering "hidden from logged-out users" as the
explanation. Every one of those was printed by code that could see
is_logged_in and did not look.
It is the same defect as the unbilled-records sentence fixed in 2.0.70 — a
message asserting something the run knows to be false — and it lands on the user
least able to act on it: someone who has already done the thing being suggested
concludes the advice is boilerplate, and stops reading the line that would have
told them something real.
All four now branch on whether the run has a working cookie. is_logged_in
goes false the moment a cookie proves dead, which is exactly when advising a
fresh one becomes right again, so the condition needs no second case for it.
_cut_short_sentence takes logged_out as a required argument rather than
defaulting it — the same reasoning its docstring already gives for taking
readings rather than the client they come from.
[2.0.73] - 2026-09-03
Fixed — a logged-out profile line walked the feed for posts it had already paid for
Found by running 2.0.58 on the dev Actor, which is the only reason it was found
at all: every offline test passed.
Logged out, Comments on a profile line could not list a single post.
Run aRNtg70aFdXL7x16l (@nasa, 15 comments asked for) ended with three fresh-IP
requeues and nothing saved. Its own diagnostics say where the money went:
The profile document answered in full, and the posts were in it. But
_recent_shortcodes only looked at the user object's embedded timeline — which
a logged-out document does not carry — and then went to the feed, the one API
surface Instagram gates hardest for a session with no cookie. 25 requests,
pagination_no_progress twice, $0.0074, zero rows.
That is the same shape _scrape_profile_from_public_html already records as
profile_feed_walk_not_repeated: once a document is bought, the feed only
re-derives what it already holds. The expansion walk was a fresh instance of the
pattern the existing MEASURED comment warns about.
scrape_profile_via_posts now passes a collect list to the lookup — the same
idiom 2.0.68 introduced for exactly this — and _recent_shortcodes mines those
documents with _profile_records_from_blocks, the extractor logged-out Posts
mode already uses. The feed is kept for the case it is good at and gated on
is_logged_in: a signed-in run resolving a business-category account whose
web_profile_info 400s, where search answers with a pk and no timeline at all.
Document-derived posts are sorted newest-first before use. The block walk is a
stack and hands them back in pop order, and for comments the difference between
the newest post and the oldest of twelve is a live thread against a dead one.
Same input on 2.0.59: 💬 @nasa: reading comments from 7 recent posts,
✅ 💬 @nasa: saved 15 comments from 1 post, zerofeed/user requests,
$0.0056. The run records profile_posts_read_from_document, which is
informational rather than truncating — nothing was lost, it is the cheaper of
the two sources.
[2.0.72] - 2026-09-03
Fixed — a follower list Instagram cut short was reported as a finished one
Run 2C6aWIYxOWbBM5ivX asked for 100 followers from each of 38 profiles and
finished ✅ Done — saved 3330 records from 37 of 38 URLs. Seven of the 37 had
returned barely half:
No throttle, no block, skipReasons: {}. Each was logged
✅ saved 46 followers
, so about 350 rows the user asked and paid for went missing
behind a tick.
Instagram serves these lists ~25 rows at a time whatever count is asked for —
3 330 rows over 128 pages on that run — and stops sending a cursor when it
decides to, which on large accounts is early. _paginate_friendships ends on
the missing cursor, which is correct; what was wrong is that ending that way was
indistinguishable from reaching the end of the list.
The profile object carries the real length of each edge, and that is what makes
the two separable: 43 of 43 is finished, 49 of 245 861 is not. A walk that ends
below both the user's limit and the account's real total now records
relationship_list_truncated — the same skipReasons channel that already
carries page ceilings and date-seek budgets into the final status message, so
the user reads "some walks ended before the target was exhausted" on the line
they cannot miss.
Deliberately quiet in three cases, each a test: an account smaller than the
limit that returned everything (crying wolf on most accounts), a walk that
reached the limit, and a target that returned nothing at all — that last one is
already reported as a URL with no data, usually a private account, and saying it
twice in different words helps nobody. A profile resolved without counts claims
nothing either: without the real length there is no evidence, and a guess is
worse than silence.
No retry is attempted. Whether a fresh exit IP gets further is a question for a
measured run, not a guess — this makes the run say what happened, which is the
part that was missing.
[2.0.71] - 2026-09-03
Fixed — saved media URLs pointed at a host only one ISP can reach
A user opened run 2C6aWIYxOWbBM5ivX's dataset in Apify Console and saw a column
of broken images: every profile picture rendered as the console's "Proxied
content" placeholder. The console does not embed remote images directly, it
fetches them through its own server-side proxy — and that proxy could not reach
the hosts we had saved.
Not expiry: the signatures in those URLs were still valid for another three
days. The hostname was the problem. *.fna.fbcdn.net is a Facebook Network
Appliance — a cache box sitting inside one ISP's network, chosen for whoever
made the request. Ours go through a US residential proxy, so the host baked into
every saved URL suited an exit IP the user has nothing to do with. 3 323 of
that run's 3 330 rows (99.8%) carried one.
MEASURED across the 22 distinct FNA hosts in that dataset: from one machine 6 of
22 would not connect at all, while the same paths on
scontent.cdninstagram.com answered 200 with byte-identical responses, 22 of
22. The oh signature is computed over the path and query rather than the
host, which is why the swap validates.
The rewrite lives in sanitize.clean_record, which every pushed row already
passes through, and applies to every string at every depth rather than to a list
of URL field names — profile pictures, HD variants, post images, video URLs,
carousel children, latest-post previews and related-profile thumbnails are all
media URLs, and a list of those names is a list to forget to add to.
One trap, pinned by a test: the query also carries _nc_ht=<the original host>,
and making it agree with the new hostname returns HTTP 403 on all 22 hosts.
Only the hostname may change; the query stays byte-identical.
The README now also says plainly that these URLs expire after a few days, which
no rewrite can change — download what you want to keep.
[2.0.70] - 2026-09-03
Fixed — Profile details billed a full price for a name and a picture
Run fBYngOPYe3cFIFhmc, Profile details over 38 accounts, all 38 reported saved
and all 38 billed. Nine of them carried 5 of 46 fields: username, id, full
name, picture, and private: false asserted without knowing. No follower count,
no post count, no bio. Nothing in the log said so — the per-URL report showed a
clean ✅ Done — saved 38 records from 38 URLs, and 24% of the run's charge
bought a name the user pasted in themselves.
web_profile_info answers HTTP 400 on business-category accounts —
refusalKinds: {http_400: 18}, both hosts, both bootstrap rounds, for the same
9 accounts on two separate runs. The lookup's cheap users/search rung caught
the fall, and the public-document rung was never reached: documentReads: 0.
The rung is not wrong; it is what makes a followers walk cheap, since that walk
needs a pk and the document costs ~700 KB. What was wrong is that one notion of
"found" served every caller. _fetch_profile_user now asks what the caller
needs it for: search_answer_is_enough=False says "I will emit this user
object as my record", so a pk-and-a-display-name does not settle the lookup and
the ladder goes on to the document. Details and Contacts pass it — the same two
modes that already pass the tight doc_cap, and for the same reason, so the
call site asks the question once (emits_the_user) and answers both.
Cost of the change, on that run's own numbers: 9 documents at the 512 KB details
cap is ~4.1 MB decompressed, ~1.05 MB on the wire at the 3.94x compression
measured on 43kEgKp3GaIgmMEaB — about $0.0084 of proxy against $0.0198 of
revenue for the nine rows it makes whole. It pays for itself, and it delivers
what the mode is sold as.
A search answer is still used when the document is refused too: a name is little,
but failing the URL outright would hand back less than Instagram was willing to
say. What stops that row costing a full profile is billing.
Fixed — an identity-only profile row is no longer charged for
record_is_chargeable had one conditional mode. Contacts promises "emails,
phones & links", so a row with none of them is delivered and not billed —
measured on run okPJnoJMV6iGzRIJd. Profile details promises the profile, and a
row carrying only identity is not the profile either, so it now answers the same
way.
The rule reads is not None rather than truthiness, deliberately: a brand-new
account really has 0 posts and an empty bio, and Instagram served every one of
those numbers. That is a delivered profile. An account Instagram would not
serialise returns None for all four of followersCount, followsCount,
postsCount and biography.
The unbilled count reaches the user as a sentence in the final status message,
and that sentence was written out at the call site in Contacts' words — "carried
no email, phone or link". The moment a second mode grew a rule that text was
wrong for it. Rule and wording now live in one _SubstanceRule entry, and a test
checks every rule has its own.
[2.0.69] - 2026-09-03
Added — the post-shaped modes read a profile, not only a post URL
On 2026-09-02 one account pasted 38 creator profiles into Comments mode three
times in thirteen minutes — first as @handle lines (wJTRcJl0HqoW6l70z), then
as full profile URLs (KvBS4nbEwfc12uvV6, ntaHwdDuuF8Mbn3gx) — and every run
finished ⚠️ Nothing to scrape: 38 lines — the wrong kind of URL for this mode.
They gave up on comments, took Profile details and Followers instead, and their
last action of the day was the same wall again in Likes (wtX9q5jf6GZn2y1QS).
Zero comments delivered, which is what they came for.
The mode read only post URLs, so the documented route was: run Posts over 38
profiles, export the post URLs, paste them back, run Comments. Nobody does that.
ModeSpec.expands_profile now marks a post-shaped mode that also reads a
profile line. Comments and Likes both carry it — the fix is the class, not
the three runs that showed it. read_line turns such a line into a profile
target, scrape_profile delegates it to scrape_profile_via_posts, and that
walks the account's newest posts, running the per-post scraper named by
_PROFILE_EXPANSION. A test pins the two registries against each other: a mode
that claims the expansion in the input layer and has no entry in the scraper
would read the line and then fail on it.
Likes stops being routed by result type and is routed by URL kind like
everything else, so the router loses a special case rather than gaining one:
post → scrape_post → scrape_post_likers, profile → scrape_profile →
scrape_profile_via_posts. scrape_post already delegated comments this way.
A profile line also dedupes across the posts it walks. That duplicate cannot
arise from a single post URL, and the run's own dedup cannot catch it —
_dedup_key keys on shortCode, which a liker row does not carry — so a fan
who likes eleven of twelve posts would have been eleven rows and eleven charges.
The resultsLimit warning for Likes now states both ceilings. Instagram's ~100
likers is per post; a profile line spreads the limit over 12 posts and reaches
about 1 200, so telling a profile user their number "can't be reached" was the
warning itself being wrong.
One pasted line stays one target with one budget. resultsLimit is documented
as "results per input", so a profile line returns at most that many comments in
total, spread newest-post-first — not that many per post, which would have
multiplied the run and the bill by a number nobody chose. Posts read per line
are capped at 12: the number web_profile_info already ships embedded in the
user object, so the common case costs no request beyond the lookup the target
needs anyway. The feed is walked only when that embedded list comes back empty —
a thin profile (web_profile_info 400s on business-category accounts and the
search fallback answers with a pk and no timeline) or a date window that
excluded every recent post. Anything else it could fetch there, it already has.
Post detail deliberately does not expand: a profile there is just Posts mode, so
it would duplicate a mode rather than unblock one. That boundary is a test, not
a comment.
The two ways a line can reach a mode that doesn't read its shape are now
contrasted in one docstring, because there are exactly two and a third would be
a mistake: recovery swaps in a different target and costs a lookup, so it
happens up front in main; expansion keeps the target and changes the work,
so it happens in the scraper, on that target's own session and budget.
Fixed — a bare @handle was filed as a skip without being read
normalize_and_dedupe only recognises full instagram.com URLs, so every
@handle landed in its dropped list — and main put that list straight into
the skip pile without calling read_line on it, even though read_line
resolves a handle to a target perfectly well and the input form advertises
@handle as an accepted shape. A paste of 38 handles reported "nothing to
scrape" while every line was readable. Both lists are now read the same way.
droppedUrls in SUPPORT_DIAGNOSTICS follows: it is filled after the lines are
read, from the ones that turned out unreadable, rather than from what URL
normalisation didn't recognise. Otherwise a scraped @handle would have been
filed under "skipped as unrecognized" — a support record contradicting the run
it describes.
[2.0.68] - 2026-09-02
Fixed — the profile page was bought twice on the path that works
_ProfileDocument exists to carry a public document "from the user-object
lookup to the record-emitting HTML fallback so the same ~707 KB document is paid
for once per target instead of twice". It was handed over by attaching it to a
raised error — so it only ever reached the fallback on the branch where the
document produced no user.
That is the wrong half. A document that failed was reusable; a document that
worked was dropped on the floor, and it is the working one the run goes on
to need: the API lookup gets blocked, the HTML rung finds the user in the
profile page, the feed walk comes back blocked too, and the fallback then needs
that same page to emit records from.
Found on the production smoke run of 2.0.20 (XqejtcgFuMyhNJKcF, @nasa, logged
out). Every API surface was blocked — feed/user 8 attempts 0 successes,
web_profile_info 5 attempts 0 successes — and the document path carried the
run. It read www.instagram.com/:profile/twice: 832 556 B to find the
user, then 741 454 B to find the posts. 1 574 010 of the run's 1 575 438 bytes
were those two documents, so the duplicate was 47% of everything downloaded —
about $0.0013 of proxy against $0.0048 of revenue for five posts.
It falls on exactly the runs that are already unprofitable: on a healthy run the
feed answers and this fallback is never reached, so the duplicate only ever
happens once Instagram has blocked the API.
_fetch_profile_user now takes an optional collect list the caller owns —
the same idiom _fetch_profile_user_from_html(collect=...) already uses one
level down — and scrape_profile passes the same list to both exits: the one
that raises, and the one that succeeds and then finds the feed blocked. The
err.documents attachment is deleted rather than extended, because a
caller-owned list has no asymmetry to fix: it fills as the documents arrive, so
it carries them whether the lookup returns or raises, and whether the failure
was its own or came from underneath it.
Changed — the early-abort measurement came back negative, and says so
identityOffsets shipped in 2.0.67 to answer one question: could the shell test
be asked about the head of the stream, turning a ~707 KB gated read into a few
KB? Measured on the same run: an 832 556 B document proved its identity only by
327 680 B, and a 741 454 B one likewise — beside the payload, not ahead of it.
So a head small enough to be worth aborting at would have read every real
profile document as a login shell, which is precisely how this could have hurt a
correct run; a head large enough to be safe is ~512 KB, already
_PROFILE_DOC_CAP_BYTES, saving 28% of a wall and nothing of the 486 KB shell.
The idea is not worth doing, and the figures and arithmetic are recorded where
the next person will look for them. Collection continues — it is free, and a
live answer beats re-deriving this.
[2.0.67] - 2026-09-02
Fixed — a gated profile document was read as data, not as a login wall
The wall latch was reachable from one place: the HTML-on-a-JSON-endpoint branch
of IgClient.request. Every profile-document read went round it, because on a
document surface HTML is the expected content type and _body_is_html is blind
there by construction.
The gated bodies are smaller than the cap that guards them — the login shell
measures 486 028 / 486 031 / 486 444 B and the wall 707 135 B, against the
900 000 B cap a Posts run reads with. So a gated document arrived whole,
took _fetch_capped_document's not truncated exit, and was handed back as if
it were a profile. Downstream that is a target which yields no rows having paid
full residential-proxy price, and — because nothing called it a wall — the next
target in the same run paid to learn the same thing again. It is the same
terminal-treatment bug LoginWallError's docstring records as costing a day of
production runs on the JSON path, still live on the document path.
looks_like_logged_out_shell settles it on the one discriminator that cannot
fire on a real document: the shell carries the token "username" exactly zero
times in 486 KB, while a profile document is keyed on it — finding it is what
_carries_profile does — so every real document carries it, private accounts
included. The bare <title>Instagram</title> is required as well, so a page
with no identity for an unrelated reason (a deleted account) is not swept up.
It is asked only about bodies that arrived complete and only when a payload
predicate actually answered "no": on a truncated slice a missing identity means
only that the identity sat past the cut.
The verdict then goes through note_document_wall, which is the JSON branch's
latch bookkeeping factored out rather than a second copy — so a document wall
rotates to a fresh exit IP first and settles only once wall_rotation_budget
distinct addresses agree.
It is classified inside get_limited, via a payload_check the caller hands
down, and that placement is the fix rather than a detail of it. get_limited
books the outcome of every response it makes. Classifying afterwards — where
this first landed — booked one response twice: bytes transferred and
record_outcome(True), then recognised as a wall. The direction is what made
it matter. A fully gated run looked healthier to the circuit breaker than a
run getting nothing, which is backwards for the stop whose job is to end exactly
that run. One response, one classification, one booking;
test_gated_document_is_a_wall pins it.
The condition in _fetch_capped_document is then left exactly as it was. It
reads not truncated or has_payload(...), which was the bug — a complete body
took the first exit without the predicate ever being asked — and is sound once
no complete-but-empty body reaches the line. Keeping that order also keeps the
cost: on a complete body has_payload is never evaluated, and it parses every
JSON block of a ~700 KB document.
Measured on tools/cost_repro.py, with a new H1_stochastic_gate scenario for
the case the fix is for — one address gated, the next healthy, which is the
common flavour and the one nothing here could previously represent:
Fixed — a gated run paid per target for a fact that is true run-wide
Three things had to hold at once, and each was hiding the next.
run_is_walled_out could not arm. It waits for two distinct walled
surfaces. That is right for the JSON rungs, where a surface may be login-only by
nature while its neighbours answer fine. It is wrong for the profile document: a
logged-out Posts run has exactly one surface that can produce an HTML wall,
because its API rungs answer 401 with an empty body, which is a block. So the
latch never armed for the commonest gated shape there is. A settled document
wall is now sufficient on its own — it is the ladder's terminal rung, reached
only after every API surface has refused, so a wall there is the last thing that
could have worked saying no across distinct addresses.
Arming it safely required the half that was missing: a document that answers
now lifts the run-wide hypothesis, exactly as a parsed JSON body always did.
Nothing did that before, which was survivable only while nothing armed the latch
from a document. Both halves are dropped through one
_clear_run_wall_hypothesis, so the lift cannot be half-implemented — a latch
that can end a run's useful life must not depend on every future caller
remembering to clear a flag.
Nothing consulted the verdict before deciding to ask. The latch armed
correctly and marked every later wall terminal, and it changed nothing, because
a settled surface was still bought again — twice per target, in fact: the
profile-lookup ladder reads the document, and when that raises, the public-HTML
fallback reads the same URL again. Five gated usernames bought nine documents.
get_limited now answers from the latch without a request.
Eight workers raced a cold latch. A verdict takes a round trip, and workers
start together, so at the default concurrency every one of them asked for the
same unproven document before the first answer came back. The first read of a
surface nobody has heard from is now exclusive, and that is the whole scope of
it: paid once per surface per run, then never again — a read that answers marks
the surface proven, a read that walls is answered by the guard.
Measured, and the shape of the result is the point — cost no longer grows with
the input:
Three, five and fifty usernames cost the same now. A1 is the one row that goes
up, and it is the rotation itself — the single gated username where recovery is
worth the most. Every healthy shape (C1, C2, D1, E1, F1, F2) is byte-identical.
Fixed — a run that stopped short reported plain success
Every branch of the final status message described targets: how many URLs
failed, how many came back empty. None described the walk. So a run that hit
a page ceiling, spent its date-seek budget, or was wound down by the throttle
circuit finished with ✅ Done — saved N records from 1 URL, and N read like the
answer rather than like where the run gave up.
Neither fact was missing from the run. skipReasons and throttled_out have
always reached SUPPORT_DIAGNOSTICS, and run_report has always glossed every
slug — tests/test_run_report has enforced that coverage for as long as the
report has existed. What was missing is that seeing either means opening the
key-value store, and the one thing that changes how a user reads their own
dataset — that there is more to fetch — was the thing kept there.
SKIP_REASON_HELP gains the column that had no home: truncates. It is on the
same record as the gloss rather than in a second set beside it, because two
hand-kept lists over the same slugs is the shape this codebase removed from the
cookie gate for exactly the reason it would fail here — nothing would say which
list a new entry belonged in. An unclassified slug counts as truncating: a build
that does not recognise a reason is the build least entitled to reassure anyone.
Two of the twenty are informational (profile_document_reused,
profile_feed_walk_not_repeated); the rest cost the user rows.
Fixed — the post page had the profile document's blind spot
The canonical post page reads with a 700 KB cap and the gated login shell is
486 KB, so it arrived whole, produced no record, and fell into the branch
commented "we saw the whole (small) page — nothing more to find here". It is
neither small nor this address's last word: another exit IP commonly serves the
real page. The target gave up silently, no wall was recorded, and nothing
downstream could learn the address was gated.
get_limited now takes the page's own _record_from_body as its
payload_check, so the same classification the profile document gets applies
here. The embed page is untouched by construction: its cap is 96 KB against the
486 KB shell, so the slice is truncated and the complete-body question is never
asked — the refetch_when_empty reasoning that already handles it stands.
Changed — the status message says it in words, not in slugs
truncating_reasons returned the raw note_skip slugs, and the status message
showed them. pagination_page_cap in a user's result line is developer-speak;
SKIP_REASON_HELP gains a label column and the message reads "a walk hit its
page ceiling". The report keeps showing the slug — it is a support document, and
the slug is what correlates a row with SUPPORT_DIAGNOSTICS. An unrecognised slug
is still reported as itself: its own name is the only honest one a build has for
a reason it does not know.
Fixed — defensive attribute access hid a fake that had fallen behind
_cut_short_sentence probed its client with getattr(ig, "skip_reasons", {}),
and main.py read getattr(res, "emitted", 0) off a TargetResult whose
emitted is a required field, one line below a direct res.failed. Both are
gone: the sentence takes the two readings it needs, typed.
Removing the first default broke 27 tests at once — FakeIgClient had no
skip_reasons, so every whole-run test had been asserting against a message the
real run could not produce. The default was not protecting anything; it was
hiding that. Two more attributes the run reads (seed_session,
note_requeue_outcome) were missing from the fake and had simply never been
reached.
tests/test_the_fake_client_keeps_up walks main.py for every ig.<name> and
asserts both the fake and the real client provide it, so the fake cannot fall
behind again — and cannot agree with a name the real client has lost either.
Fixed — dedup erased the dimension addParentData exists to record
Dedup was keyed on shortCode and scoped to the whole run. That is right
whenever two sightings are the same row. It is wrong for every mode where a post
legitimately belongs to more than one parent: one post carries several hashtags,
is tagged in several accounts, and sits in a location as well.
Scraping #cats and #kittens with addParentData on, a post in both was saved
once, under whichever tag was walked first. The second association was
dropped with no row, no counter and no skip reason — nothing anywhere to read.
The user had asked for that dimension explicitly by turning the option on, and it
was the dimension the key threw away.
The key is now the row's identity rather than the post's: the shortcode plus
whatever of the schema's five parent* fields the row carries. With
addParentData off nothing is attached, the key is the bare shortcode, and
behaviour is byte-identical — including for a run resuming against RunState
written by an older build, whose keys were bare shortcodes too.
Added — the measurement an early abort needs, taken for free
identityOffsets joins payloadOffsets in profileLookup, on the same
no-extra-request binary search. payloadOffsets says how much of a document a
reader needs (measured: 458 752 B); this says how much a classifier needs —
the point past which the shell test could be asked about the head of the
stream instead of a whole body, turning a ~707 KB gated read into a few KB.
That constant is deliberately not guessed. Too small a head and a real document
whose identity had not arrived yet reads as a gate, which is precisely the way
this could start hurting correct input. It gets measured from real documents
first, the way every other cap in this codebase was.
Fixed — the cost model could not represent recovery
_resp_healthy served the wall blob for a profile document, on the reasoning
that the healthy path resolves through the API and never reads the document.
So "healthy" and "gated" were the same bytes and no rotation could ever be shown
to pay off — a model that can only price gates as terminal. It now serves a real
document, with an identity, a payload and a real title.
The day reconstruction is repointed at 2026-09-01 (289 runs, 653 results, $1.1014
cost, $0.42228 revenue) and reads revenue per result against the developer
share rather than list price, which is what pins the mix: $0.00063/result is a
day of pure post-scraped, 31% FREE / 69% BRONZE. Productive runs are sized from
the observed 100 rows rather than E1's 12 — the old figure undercounted
zero-result runs eightfold.
[2.0.66] - 2026-08-31
Changed — the cookie gate reads the mode, off one registry
Which targets get skipped without a cookie was decided by two hand-kept lists in
main.py: five login-only result types, and six URL kinds. Between them
they covered the nine 🔑 modes exactly — test_docs_match_behaviour has pinned
that end to end since it was written, and still does — but the cover was a
coincidence of the two axes.
Four 🔑 modes cannot appear in a result-type list at all: Tagged, Hashtag and
Location all produce results_type="posts", Search produces "details", and
those are public. Naming them there would gate Posts. They were caught by the
kinds list instead, because each happens to be the only mode producing its
kind. Nothing in either list said which of the nine it was there for.
The 🔑 is a property of the mode, so it lives on the mode now: _MODE_SPEC
becomes a ModeSpec with a login_only column, and LOGIN_ONLY_MODES is
derived from it. _target_requires_user_cookies asks mode_is_login_only(mode)
and the one genuinely per-target question (forced_cookie_target_keys); both
lists are deleted. Which targets are skipped does not change —
tests/test_cookie_gate_is_one_registry checks the gate's answer is identical
for every mode x URL kind pair, and mutation-checks that clearing login_only
on any of the nine opens the gate.
Fixed — four of the nine 🔑 modes gave no warning before the run
The up-front "this mode is login-only, paste a cookie" notice read the
result-type list, so it could only ever fire for the five modes with a result
type of their own. Pick Hashtag, Location, Tagged or Search with no cookie and
there was no warning at all: every target was skipped correctly and silently,
and the run's only explanation arrived at the end as ⚠️ Done — saved 0 records.
It is keyed on the mode now and fires for all nine.
Fixed — the skip message named the wrong half of the sentence
🍪 Hashtag URLs need an Instagram cookie named the URL kind, which is the
half a user cannot act on: the same profile URL is public under Posts and
login-only under Followers. It names the mode they picked:
🍪 “Hashtag” needs an Instagram cookie — Instagram serves this data only to
a signed-in session. Skipped 🏷️ hashtag 'travel' without spending anything on it.
Removed — three URL kinds that no mode can produce
user_reels, story and highlight were in the kinds list and carried nothing:
Reels, Stories and Highlights all read their lines as usernames, so
instagram.com/nasa/reels/ and a story URL both arrive as profile. Checked
over every mode x line shape the form documents. The advice string one of them
selected — "Or paste the profile URL (no /reels/) and pick Reels" — could never
be reached.
Notes
ScrapeContext gains login_only, so the mid-run "the cookie just died" short
circuit asks the same question as the pre-scrape gate instead of re-deriving it
from results_type, which for four of the nine modes is not derivable.
1053 tests pass, offline and warning-free; ruff clean.
[2.0.65] - 2026-08-31
Both remaining run-wide stops were absolute counts. An absolute count is not a
safety limit, because the run that reaches it first is simply the largest one —
so what it actually selects for is a big legitimate job, not a sick one.
Removed — globalRateLimitBudget, which cut productive runs in half
300 throttle signals per run, after which the circuit tripped. Its own comment
in http_client already called it "a scale-blind magic number: at its default
of 300, that is 6% of a 5000-request run but three times the whole of a
100-request one" — and stopped one sentence short of the conclusion.
Measured against the real scraper, sweeping block rates over 1200 targets:
block rate requests rows what stopped the run
10%374914400 nothing — ran to completion
20%18236000 the 300-throttle budget, at request 1819
30%450 the breaker, at request 37
50%560 the breaker, at request 40
The 20% row is the whole argument. That run was returning rows steadily and was
stopped half way through its export, and the rate breaker never fired because
20% is nowhere near a rate worth stopping for. It was not a backstop, it was a
timer on run length. Removing it takes the same input from 6,000 rows to 14,400.
Nothing replaces it: the breaker judges the rate of blocks over a sliding
window, so it reaches the same verdict at any scale — hostile runs still wind
down at request 37-40 whether the input is 10 targets or 1200. Throttles are
still counted and reported (throttlesSeen in the diagnostics, in place of the
old "230/300 left", which invited the reader to treat a budget remainder as
headroom).
Fixed — maxHttpRequests was the same defect, one order of magnitude out
A flat 20,000-attempt ceiling. At the ~3.1 requests per target a healthy run
actually spends, that lands at about 6,400 targets: a legitimate large run,
aborted for being large.
It is now sized from the input — targets x requests_a_target_could_need() —
armed next to configure_concurrency and for the same reason, since neither
can be known before the input has been read. The per-target bound lives in
instagram.py beside the pagination caps it multiplies (_MAX_PAGES x
request_retries), so a target that exceeds it is not slow but stuck in a loop
that has stopped respecting its own ceiling — which is the only thing a
run-wide ceiling should be looking for.
The flat number stays as a floor. The ceiling only ever moves up, so a
small run is bounded exactly as tightly as it is today and no run that finishes
now can start failing.
Notes
1044 tests pass, offline and warning-free; ruff clean.
tests/test_request_ceiling_scales pins the new shape: the ceiling follows
the input, never drops below the floor, keeps attempts already spent when it
is lifted, and clears what a healthy run measurably spends by more than 10x.
The per-target bound is asserted to move with _MAX_PAGES and with the retry
count, so it cannot drift away from the caps it is derived from.
tests/test_retry_budgets now pins the absence: no global_rl, no
rateLimitBudget in the diagnostics, and one hot URL cannot wind down a run.
Verified by mutation: flattening the ceiling, letting the lift discard spent
attempts, and letting it lower the floor are each caught.
Both were unpublished knobs, so no published input changed;
globalRateLimitBudget is now ignored if passed.
[2.0.64] - 2026-08-30
Fixed — the cost harness did not model how a run stops
tools/cost_repro.py exists to price real run shapes by replaying them through
the real scraper, and it built its IgClient without abort_event and without
max_http_requests. Both are how a run stops. Without the abort event the
client's wind-down had nothing to signal, so a tripped circuit was announced
and then ignored: every target after it did full work.
The harness therefore priced correctly only the shapes too small to trip
anything — which was every scenario it had, all of them 1 to 5 usernames. Above
that it was wrong by a lot. 200 gated usernames read $0.1957 against a true
$0.0079, and B2_gated_block_5url drops from 135 requests to 35 now that the
run actually ends where it really ends.
Two scale scenarios were added (A5_gated_wall_50url, B3_gated_block_50url)
so this cannot quietly come back: those are the only rows in the table where a
run-wide stop can fire at all.
Fixed — the login wall was invisible to the run-wide breaker
Every gating signal fed record_outcome except the most expensive one. A
settled login wall — rotation spent, the wall now an established fact rather
than something the next exit IP might clear — fed only its own per-surface
latch. So the one signal that costs ~410 KB a read was the one signal the
instrument for noticing an unproductive run could not see.
It is now recorded, at the point the wall settles rather than on each rotation
(where it is still a hypothesis), with load_induced=False so the concurrency
governor is untouched — a wall is about what this identity may read, and
narrowing the window cannot help. A signed-in run reading one login-only
surface among productive ones stays far below the block rate the breaker
judges on, which is what makes this safe to count.
What actually bounds a run now
Removing the dollar budget in 2.0.63 took away the only stop denominated in
money, so this is what is left, measured rather than asserted:
shape 5 targets 200 targets
logged-out login wall 14 reqs 404reqs(2.8->2.0 per target)
401 require_login 27 reqs 56reqs(5.4->0.3 per target)
50% blocked 27 reqs 104 reqs
70% blocked 27 reqs 88 reqs
The bound is the block-rate breaker, and it is denominated in evidence: it
judges the rate of blocks over a sliding window, so it returns its verdict after
a fixed number of samples no matter how large the input was. Forty times the
input is not forty times the work, and the marginal target gets cheaper.
That is the distinction worth keeping straight. A stop that reads "this run is
unprofitable" withholds rows somebody asked for and was removed. A stop that
reads "this run is getting nothing" costs the user nothing, because there was
nothing to get.
One residue is deliberate: a run that has concluded it is walled out still
spends one read per target, since refusing outright cannot be revisited and
logged-out gating is stochastic — a run that latched itself shut would return
zero rows for the rest of its life, the 2.0.16 regression one level up. So the
wall shape keeps a small per-target cost on purpose. urlRetryBudget's absolute
ceiling (20,000 requests) is what bounds the pathological end of it.
Notes
1037 tests pass, offline and warning-free; ruff clean.
tests/test_hopeless_run_is_bounded pins all of it: the marginal target must
get cheaper rather than dearer, a settled run must not re-buy the full
rotation per target, the wall must reach the breaker, and — the other
direction — a run whose requests all succeed must never be wound down.
Verified by hiding the wall from the breaker again, which the suite caught.
[2.0.63] - 2026-08-30
Removed — the run ledger no longer ends a run over money
src/ledger.py kept a budget:
earned + stake x targets_not_yet_finished - spent
. When it reached zero the HTTP client wound the run down at its request
chokepoint — gracefully, keeping every row already scraped, and only after
targets were known. It was carefully built and it was the wrong thing to build.
It meant a user could be handed a short export because of our prices, with
nothing about their input at fault. Somebody who asks for a profile's followers
gets the followers. Whether that scrape earns more than it costs is a question
about the price list, and the answer belongs in the Console — it is the owner's
to fix by pricing the events, not the scraper's to fix by withholding rows
somebody asked for.
Gone with the rule: exhausted, budget_usd, allowance_usd,
DEFAULT_STAKE_USD_PER_TARGET = 0.01, enabled/_armed, and the wind-down
branch in http_client. arm(targets_total) is now note_targets(...), since
there is no longer anything to arm.
This also removes 2.0.62 (86b5870), one commit old. The drift detector, the
free-tier price ladder and ledger_is_applicable existed for one purpose: to
recognise a free-plan user, whose platform usage Apify covers, so the budget
would not cut a run that costs us nothing. With no budget there is nothing to
exempt anyone from, and a detector guarding a deleted guard is just another
table to keep current. billing.FREE_TIER went with its only reader.
PRICES_USD_PER_EVENT stays: it still documents what the Console charges, and
tests/test_single_source_of_truth still holds it as the only copy.
Kept — all of the measurement
The ledger is now a meter. It measures; it does not decide. Real wire bytes
against the proxy rate, memory-seconds against the compute rate, live charges
through ChargingManager scaled to the developer share — all still recorded,
all still in ledger in SUPPORT_DIAGNOSTICS.
That measurement is not incidental to this change, it is the reason the change
is safe to make. It is what separated the per-run cost from the per-result cost,
fitted cost = $0.01049/run + $0.000077/result over thirty days, and located
the actual defect this series fixed: a logged-out run paying the full five-address
wall rotation on every surface. Diagnosing margin is what the numbers are for.
Enforcing it against the user's export is not.
One caveat now that nothing is enforced against it: the ledger captures 75-90%
of a run's true cost (it does not model dataset, key-value or request-queue
operations, or container startup before it is constructed). Read it as a lower
bound.
Notes
1033 tests pass, offline and warning-free; ruff clean.
tests/test_money_never_stops_a_run is the new invariant: a run that has
spent five gigabytes having earned nothing is not wound down; RunLedger
exposes no verdict to read; and an AST check fails if anything outside
ledger.py reads the ledger for anything but a note_* or a snapshot.
Verified by reintroducing a cost-based wind-down, which the suite caught.
tools/cost_repro reports cost per zero-row target as a margin input rather
than as a threshold, which is all it can be now.
[2.0.62] - 2026-08-30
Fixed — a stale price table failed silently
src/billing.PRICES_USD_PER_EVENT is a copy of what the Console charges, and
billing's own docstring tells the owner to update it whenever the pricing
generation changes. One thing depends on it being current: _looks_like_free_tier
recognises a free-plan user by matching the prices the platform resolved for
them against the free-tier column.
Raise prices in the Console without updating that file and nothing raises,
nothing logs. The ladder stops matching, free-plan runs stop being recognised as
free, and the ledger begins constraining runs that cost us nothing — Apify
covers platform usage for free-tier users, so those runs are pure profit. Cut
short, with no signal anywhere.
A live price that appears at no tier of an event's ladder now means the table is
behind the Console, and the run says so once, naming the prices it saw. It does
not fail the run: this is stale documentation, not a reason to abandon a scrape.
Investigated and reverted — pricing the ledger's stake in results
DEFAULT_STAKE_USD_PER_TARGET = 0.01 is a flat dollar amount that does not
follow the Console, which looked like the same class of problem. Comparing it to
what twenty results earn made the case look strong:
mode event 20 results earn flat stake risked
followers/likes user-scraped $0.0040 $0.0100
details/contacts detail-scraped $0.0208 $0.0100
That reasoning was wrong, and the existing test caught it. The stake is not
an estimate of what a target will earn — it is the allowance for reaching the
first row, after which real earnings enter the budget and it grows on its own.
Priced at twenty results, followers and comments targets would have been cut
at $0.0040 and $0.0048 against a measured $0.0057 cost to reach that first row:
killed before they could return anything.
The twenty was itself derived by dividing the flat $0.01 by the post price — a
constant reverse-engineered from the number it was meant to replace. Reverted in
full; $0.01 is a measured reconnaissance budget and correctly independent of
price. tests/test_run_ledger already asserted exactly this, which is why it
took one run to find out.
Notes
1059 tests pass, offline and warning-free; ruff clean.
No behaviour change: the drift detector only logs.
For the record, what is price-coupled: earned_usd reads live charges
through the SDK and needs no maintenance; APIFY_REVENUE_SHARE, the proxy and
compute rates are platform constants; PRICES_USD_PER_EVENT is the one copy
that must be updated by hand, and is now the one that complains when it is not.
[2.0.61] - 2026-08-30
Fixed — the error-set collapse had missed every multi-line tuple
2.0.55 replaced the spelled-out except tuples with named sets, using a
single-line pattern. Nine handlers wrapped across two lines and were therefore
untouched — including three that named SessionBlockedError alongside
RateLimitedError, which is the same class twice, and four that listed
ScrapeError among others, which is that one class four times over.
All nine are named now, and each replacement was verified to catch exactly
the same set of classes before it was made, not after. A guard walks the AST —
not a regex — so a tuple cannot hide behind a line break again.
The three genuine survivors turned out to share a reason: each sits after a
separate HttpError handler that decides what a status means for the surface,
so catching statuses again would take that decision away. That set has a name
now, LOAD_ERRORS — congestion and blips, which clear on their own — and its
omission of HttpError is asserted rather than left looking like an oversight.
Fixed — one user-facing sentence written twice
"Instagram cookie invalid or expired: login-only data can't be scraped" appeared
at both places that produce a dead-cookie target result: the pre-scrape gate and
the mid-run CookieInvalidError handler. A user who hits both should not read
two wordings of one fact, and reworded once, they would have.
Notes
1049 tests pass, offline and warning-free; ruff clean.
No behaviour change: every collapse is provably identical, verified class by
class against the tuple it replaced.
Checked and clean: no return-type annotation is contradicted by its body (the
three apparent cases were return None inside nested functions, which
ast.walk does not separate), and the remaining repeated log strings are
adjacent except branches sharing one message, which is idiomatic.
[2.0.60] - 2026-08-30
Fixed — three parameters the body never read
_cookie_required_target_error took a forced flag it never looked at, while
its one caller computed _target_identity(target) in forced_cookie_target_keys
to supply it. A reader at that call site had every reason to believe the message
distinguished a URL the user had marked as cookie-required from one that
simply is. It does not, and never did — the concept is real and used elsewhere;
this message just does not reflect it.
_report_skips took a mode it ignored. _probe_comments_query took a
url_budget it never spent — and unlike the scrapers, it is private with one
caller, so nothing forced it to stay.
Three scrapers do legitimately ignore url_budget: scrape_target calls every
scraper with one signature, so a surface answering in a single request still has
to accept it. They say so in their docstrings now, and the guard is written to
accept exactly that — an exemption has to be recorded at the function, not in
a list inside the test.
Notes
1047 tests pass, offline and warning-free; ruff clean.
No behaviour change: all three parameters were dead on arrival, and removing
the forced one also removes the pointless set lookup its caller was doing.
Also checked and clean: no statement follows an unconditional return /
raise / continue / break anywhere in src/ or tools/, and the only
two try blocks whose body makes no call are a context manager's yield and
a genuine KeyError on dict indexing.
[2.0.59] - 2026-08-30
Measured — line coverage, and what it found
Ran the suite under the stdlib tracer (no new dependency): 77.6% of
statements in src/, 4807 of 6194. The number itself is context; two things in
it were worth acting on.
src/text_numbers.int_from_text carried two guards that cannot fire. The
pattern requires a digit after the optional sign, and stripping group separators
cannot remove it, so the stripped string is always an optional - followed by
at least one digit — which int accepts, unicode digits included. Brute-forced
over 137 560 generated inputs: the "empty digits" branch and the ValueError
handler were reached zero times.
They are gone. A guard that cannot run reads as a risk that exists, and this
repo already has a test whose docstring says why that matters. The invariant
they were pretending to protect is now asserted instead — over the same
generated inputs — and mutation-checked by weakening the pattern to not require
a digit, which fails the suite.
ig_ids.owner_of's non-dict branch and int_from_text's non-string branch were
genuinely unexercised. Both now have tests; the second matters because the
function is public and that branch is all that stands between a surprising
payload and a TypeError mid-scrape.
Notes
1046 tests pass, offline and warning-free; ruff clean.
Coverage by module, for reference: dataset_writer 90.9%, sanitize 88.7%,
modes 87.3%, parsers 86.8%, http_client 85.3%, html_extract 79.7%,
main 79.3%, instagram 78.4%. The two diagnostic probes sit at ~31% because
they only run behind the __probeCommentsHttp / __probeProfileHtml inputs.
cookies.py reports 0% and is not: the tracer starts after tests/support
has already imported it. A measurement artefact, not a gap — recorded so the
next reader does not chase it.
Checked and clean: the input schema's 11 fields are all read by the code, all
three enums match their titles, and every one of the 24 countries the dropdown
offers survives _coerce_country_code.
[2.0.58] - 2026-08-30
Fixed — dataSource documented seven values and produces thirteen
The dataset schema described the field as "profile / hashtag / location / post
/ comments / user_reels / tagged". The scraper also writes story,
highlight, highlights, likes, followers and following. A user
filtering the export on that field would see rows the documentation says cannot
exist — and, worse, would build a filter that quietly drops six kinds of record.
Nothing enforced the list, because a description is prose.
tests/ test_data_source_values
now derives the real set from the source — literals
assigned to the field, plus the two places where the value arrives through a
variable whose domain is itself a table (_FRIENDSHIP_EDGES, the section-feed
specs) — and fails if the description omits one, invents one, or if a third
variable starts feeding the field. All three are mutation-checked.
Fixed — four tests that reported success without checking anything
Three in test_sanitize called json.dumps(...) under a # must not raise
comment. That is the one thing which could not fail: a NULL byte serialises
perfectly well and then breaks the storage API on the way in, which is the
entire reason the module exists. One of the three was named
..._is_removed_too and never checked that anything was removed — it would
have passed had the scrub stopped at the depth limit, which is the exact
regression its own module docstring records having fixed.
They assert the removal now, and the assertions were verified by breaking the
scrub two ways. A guard fails the suite if any test method has no reachable
assertion; mock assertions and delegation to an asserting helper both count.
test_deterministic compared normalize_session_id(x) to itself, which agrees
under almost any breakage. It is pinned to literal values now — a session id
selects the proxy exit IP, so a changed hash silently re-shuffles which target
sticks to which address.
Fixed — the suite printed a warning on every run
A guard added in 2.0.55 contained a backslash-s in a non-raw string, so every test run
emitted SyntaxWarning: invalid escape sequence. Fixed, and the whole tree is
now confirmed free of them.
Notes
1044 tests pass, offline and warning-free; ruff clean.
No behaviour change: the only source edit is a schema description.
[2.0.57] - 2026-08-30
Added — the README is checked against the schemas it documents
A README is the one artefact nothing executes, so it drifts silently: a field
renamed, a mode dropped from the dropdown, a limit tightened, and the example
that taught someone how to use the Actor now teaches a run that fails on input
validation. Its seven JSON blocks are now validated — input examples against
input_schema.json (fields exist, enums match, bounds hold), output examples
against the fields dataset_schema.json publishes. All seven pass today; the
guard is for tomorrow. Mutation-checked with a renamed field, an out-of-range
value and an unpublished output column.
Fixed — housekeeping from this series' own edits
Ten modules had ended up with their constant block above their imports, an
artefact of adding constants mechanically after from __future__. Python does
not mind and ruff is not configured to, but a reader scanning for a module's
dependencies has to step over a page of tuning values first, and a constant that
later needs an import cannot have one without moving. Moved, and asserted —
cost_repro and report_margins are allowlisted, because they insert the repo
root into sys.path and install an apify stub before importing src.
One comment still illustrated its argument with the old global-budget default:
"50 blocks is 1% of a 5000-request run but 50% of a 100-request one". The
default has been 300 since 2.0.51, so the example read as a statement of the
current value. Updated.
Notes
1040 tests pass, offline; ruff clean.
No behaviour change: the constant moves are pure relocation (verified by
reading the diff for value changes — there are none), and the README guard
adds no runtime code.
Checked and clean: the 24 diagnostic keys are all written where a human reads
them and none is read by code under a different name; run_report reads no
key that main fails to write.
One self-inflicted near-miss: relocating constants in tools/report_margins
split its _apify_stub = ... from the sys.modules.setdefault that installs
it. Caught immediately by ruff, reverted, and the file is now allowlisted for
the reason that made it break.
[2.0.56] - 2026-08-30
Fixed — a post page was requested with a less browser-like fingerprint
Four places built the headers for "fetch an HTML document". One was a builder,
_document_headers; the other three were copies with a shorterAccept:
So a profile document went out looking like a browser and a post page did not,
on the same run, against the same host that decides whether to serve a login
wall. Nobody chose that; the copies simply never grew the rest of the header.
All four use the builder now — the full form was already in production for
profile documents, so it is the proven one.
Content-Type: application/x-www-form-urlencoded, typed out at four POST call
sites, is a named constant.
Changed — deferred imports that were not hiding a cycle
Seven imports sat inside function bodies. Five pulled src.parsers into
instagram — which already imports it at module level — and into
html_extract, where no cycle exists in either direction; two pulled
datetime. They cost a dict lookup per call in the hot parsing path, and they
tell a reader to go looking for a circular dependency that is not there.
Hoisted, and both properties are now asserted: only apify may be imported
lazily (so a module can load without the SDK), and the internal import graph is
checked to be acyclic.
Notes
1034 tests pass, offline; ruff clean.
The header change alters what goes on the wire for post and embed pages. It is
the only behaviour difference, it moves those requests toward the shape the
rest of the run already uses, and it cannot be verified offline — worth
watching in the next live run's gating rate.
Checked and left alone: the 64 dataset field names appear across up to six
modules each, which looks like duplication. Renaming one is already caught by
test_dataset_schema_matches_output (verified by mutation), so constants
would add indirection without removing a risk.
Also checked and clean: no unreachable branches, no repeated condition in any
if/elif chain, no duplicated helper across tools/, and no string shared
between the two .actor schemas.
[2.0.55] - 2026-08-30
Changed — five spellings of "whose failure is this?"
Seventeen try blocks asked whether a failure belonged to the run or to the
target it happened on, and answered by listing exception classes. Five different
tuples appeared:
ScrapeError is the base of every one of those, so the first two are spellings
of ScrapeError with three redundant names attached, and SessionBlockedError
is a RateLimitedError, so naming both says one thing twice. The redundancy is
not harmless — it hides the real difference, which is whether
CookieInvalidError and PermanentTargetError are caught. Those two are
exactly the failures a caller must not swallow: one means the credential is dead
for the whole run, the other that this target is gone for good.
RUN_TERMINAL_ERRORS, RECOVERABLE_ERRORS and TRANSPORT_ERRORS name the
three answers. Equivalence to every former tuple is asserted class by class in
tests/test_error_sets, along with the ordering the call sites depend on —
ScrapeError covers the budget errors too, so a wide handler that did not
re-raise them first would swallow an exhausted budget.
Fixed — task cleanup that logged noise and serialised cancellation
Three blocks cancelled and drained speculative tasks, disagreeing twice over.
One cancelled and awaited each in turn, which lets a slow task hold up its
siblings' cancellation. One filtered to not task.done() before awaiting, which
leaves an already-failed task's exception unretrieved — Python then prints
Task exception was never retrieved at collection time, noise in a production
log about work that was deliberately discarded.
_drain_cancelled takes the correct half of each: cancel everything first, then
await everything, finished tasks included. The ordering test is written so that
getting it wrong deadlocks rather than merely differing — asserting on the set
of cancelled tasks cannot see the difference, since the serial order cancels
them all too, just one at a time.
Changed — GraphQL connection names
edge_media_to_comment appeared 9 times across 5 modules,
edge_media_to_parent_comment 7 times across 4, and each module listed its own
subset in its own order. Instagram has used three names for the same connection
across surfaces and eras, so a parser checking only two of them found no
comments at all on the surface that used the third.
COMMENT_CONNECTION_KEYS, CAPTION_KEY and LIKES_KEY join
TIMELINE_MEDIA_KEY in graphql_shape.
Notes
1032 tests pass, offline; ruff clean.
No behaviour change. The error-set collapse is provably identical — asserted
per class, not assumed — and the task-cleanup change removes a log warning
without altering what is cancelled.
A finding that turned out not to be one, recorded because it looked
conclusive: _scrape_post_full_data_from_html takes three parameters its only
caller always passes as None, and no test passes them either. That reads as
dead flexibility, but the call is asyncio.create_task(...) fired before
oEmbed and the profile are resolved — the whole point of the overlap — so
there is nothing to pass yet. The explicit Nones document that. Left alone.
Two guards needed strengthening after their first mutation check failed: the
task-ordering test (a set assertion cannot see serialisation) and the
connection-key test (a line scan flags the docstring that documents the shape
and the regexes that match it, so it walks the AST instead).
[2.0.54] - 2026-08-30
Fixed — the same count parsed four ways, two of them wrong
Four converters turned Instagram's rendered text into an integer, and which one
answered depended on which parser tier had reached the field. They disagreed on
7 of 15 realistic inputs:
input
_to_int
_embed_int
_to_optional_int
"1,234"
1234
1234
1234
"1,234 likes"
None
1234
1234
"1.5k"
None
15
15
"2024-01-15"
None
20240115
20240115
"-5"
-5
5
5
Inconsistency is the smaller half. "1.5k" becoming 15, a date becoming an
eight-digit count, and a negative losing its sign are wrong values written to
the dataset, not missing ones — and nothing downstream can tell them from real
counts. Stripping every non-digit is what produced all three.
src/text_numbers states the rule once, and it is deliberately unwilling to
guess: group separators inside the number are dropped (so 1,234, 1 234 and
1.234 are one number in three locales), surrounding text is ignored (so
"1,234 likes" still reads), a leading minus survives, and a magnitude suffix
yields None — expanding 1.5k would be a guess about a value we were not
given, and 15 is the answer that made it a bug.
coerce_count also settles bool, which two of the four disagreed about: one
returned 1, the other None. None is right — a boolean in a count field
means the payload was not what we assumed, and inventing 1 hides that.
tests/test_text_numbers asserts all four entry points now return the same
answer for every input in that table.
Changed — three more rules that lived in two places
What Instagram accepts as a name.^@?([A-Za-z0-9._]{1,30})$ was compiled
in input_coercion and again in modes — one platform limit, two statements
of it. The hashtag patterns beside them differed by a single ?, and that
difference is real: one reads a token where the # is what marks it as a
hashtag at all, the other a field the user already labelled. As two named
functions in src/ig_names that is a decision; as two regexes in two files it
was indistinguishable from drift.
The GraphQL comment-operation name. Written out three times — twice in
comment_probe, once in tools/inspect_instagram_har — so widening it for a
new naming convention meant finding all three. It lives in ig_names, which
has no dependencies, so the HAR inspector can import it and stay runnable
without the SDK.
is_reel: four feed filters compared product_type != "clips" directly,
bypassing the module that knows a reel is a video but not every video is a
reel.
Notes
1017 tests pass, offline; ruff clean.
The count fix changes output where input was abbreviated or malformed: those
fields are now absent rather than wrong. That is the point of the change, and
it is the only intended behaviour difference in this release.
Two false leads worth recording, since both looked like findings: the re.split
calls flagged in cookies.py were str.split (a scanner bug, not a code one),
and the ~30 re.compile calls flagged in html_extract are inside a
module-level dict, compiled once at import.
[2.0.53] - 2026-08-30
Changed — Instagram's GraphQL shape, described once
Every list in a GraphQL payload arrives as
{"edges": [{"node": {...}}], "count": N, "page_info": {...}}, and forty-two
sites unpacked that by hand. They did not agree on how careful to be. Measured
against the same fourteen malformed payloads:
walk
raises on
silently yields junk
(obj.get(k) or {}).get("edges") or [] + edge.get("node") or {}
5 of 14
4 of 14
the isinstance-guarded variant
2 of 14
0
src/graphql_shape
0
0
Both forms were in the codebase for the same question, so an edges that is
null rather than absent raised in one parser and was skipped in another. The
module now describes the shape once — connection, edges, edge_nodes,
nodes_in, connection_count, next_cursor, edges_and_page_info — and the
defensiveness is uniform, so a malformed payload degrades to "no rows"
everywhere.
next_cursor also folds a rule that was worth stating: a walk stops when
Instagram says there is no next page and when it says there is but supplies
no cursor. The second is what an exhausted feed looks like on some surfaces, and
missing it re-requests page one.
Changed — three more rules that were written out rather than named
media_types.is_reel: four feed filters compared product_type != "clips"
directly, bypassing the module that knows what a reel is. A reel is a video;
not every video is a reel.
ig_ids.owner_of: Instagram calls a post's author user on the v1 API and
owner on GraphQL. Four copies read them in two different orders — the
same shape of latent bug as id/pk, though here no payload carries both.
http_client.error_means_absent:
isinstance(err, HttpError) and is_not_found(err.status)
at four lookup ladders, each deciding whether to
tell the user "no such profile" or "we could not read it".
_configure_proxy lifted out of _run_actor, which reads its input,
configures networking, walks targets and writes its report in one body.
Changed — the test suite had four copies of its own foundation
tests/support.py now installs the apify stub and holds the shared fakes. It
replaces:
68 copies of the stub itself, in four variants. Seven files added Event
because the code they exercise needs it and sixty-one did not; one used
sys.modules["apify"] = … rather than setdefault, so which stub a run got
depended on collection order. test_state.py documented the symptom in a
comment — "regardless of which test module first registered it" — which is
the note a copied fixture leaves behind instead of a fix.
8 byte-identical _client() helpers, 2 _FakeKV, 2 _FakeActor
(differing only in a docstring), 2 _FakeIg, 2 _RecordingActor, and 4
FakeResponse variants that were subsets of one another — so a test needing a
field its local copy lacked either added a fifth variant or worked around it.
Five test modules importing _FakeActor and _run from
test_zero_result_runs, which made that file a de-facto library whose own
tests could not change without breaking its importers. The driver is
support.run_actor_offline now, and no test imports another test.
Net: 917 lines removed from tests/, 744 added. A guard in
test_single_source_of_truth fails if a stub is hand-rolled again.
Notes
1004 tests pass, offline; ruff clean across src/, tools/ and tests/.
No behaviour change intended. One near-miss was caught and reverted while
folding the fake responses: the union type initially dropped headers for the
variant that had none.
_run_actor is still 970 lines. Splitting it further means threading a dozen
locals through a settings object — a structural change with real risk and no
bug behind it, so it is left as is and recorded here rather than attempted.
[2.0.52] - 2026-08-30
Changed — rules the code stated as control flow, and copies that had drifted
A second sweep, this time for duplicated logic rather than duplicated numbers.
The clone detector found four scraper bodies written twice in full; the registry
scan found five vocabularies described by more than one table. Several had
already diverged, and two of those divergences reached users.
The section feeds were one procedure written twice.scrape_hashtag /
scrape_location and _fetch_hashtag / _fetch_location — ~90 lines each pair,
sharing every decision (the two-host fallback, the wall-versus-404 reasoning, the
embedded-sections walk) and differing only in nouns and endpoint paths. They are
now _SectionFeedSpec plus one implementation, with the two callers reduced to
building a spec. The copies had already begun to diverge in wording, which is the
state in which one gets a fix and the other does not.
id and pk are not interchangeable, and the code assumed both ways.
Thirty-three sites read an entity id with a hand-built or chain, and the chains
disagreed on key order. Measured in tests/fixtures/media_info.json:
pk ="2029910590113615245"
id ="2029910590113615245_297604134"
On a media object id is the media id joined to its owner's, so
media.get("id") or media.get("pk") yields a compound value no endpoint accepts.
Both spellings were present for the same question. src/ig_ids now names one
reader per kind of entity — hiding the difference behind a single helper would
have preserved exactly the bug — and bare_media_id, which already existed to
clean up the damage downstream, moved there to sit with it.
contacts was missing from two of three display registries. An order tuple,
a name map and an emoji-label map described one vocabulary; contacts was in
only the name map. The mode was therefore absent from every "works with" list the
Actor prints, and _pretty_results_type("contacts") rendered the bare internal
slug in user-facing messages. Nothing failed — the output was just worse for one
mode, indefinitely. One registry now, ordered.
Two registries named URL kinds differently, and users saw both. A
/username/reels/ URL was "User reels" in one message and "profile reels" in
another; a tagged URL was "Tagged" against "tagged-posts". Neither word was
wro
