-
Large requests now return substantially more events. A single search on the
source stops at around 120 results however many actually match. Asking for more
than that now re-runs the search as a series of narrower ones (by country, then
industry, then event type) and merges the results. A reference request for 500
events returned 359 before and returns 500 now.
- Filters you set are never overridden. Pin a country and it splits by industry
instead.
- Only a search that came back at the limit is split, so a request for 120 or
fewer costs and takes exactly what it did before.
- Events matching more than one of the narrower searches are returned, and
charged, once.
- Expect a large run to take several times longer per event than a small one.
-
If you run with maxItems: 0 (unlimited), please read this. Such a run now
collects everything it can reach rather than the first slice, which for a broad
search can be several thousand events instead of a few hundred. Billing is per
event, so the cost of an unlimited run changes accordingly. The run log now
warns at the start when a run is unlimited. Set maxItems for a predictable
run cost; runs that already set it are capped by it exactly as before.
-
Large runs now return much more complete rows. On the same 500-event
reference request, every row now arrives with full location, organizer,
categories and pricing, where previously 43% did and 53% had no country at
all. Average populated fields per row went from ~21 to ~36.
- An event served twice by the source on the same page was returned, and
charged, twice. Now returned once. If you have been de-duplicating on
editionId downstream, you no longer need to.
- A run that ended below the requested
maxItems now says which of the three
cases applies: results could not be retrieved this time, the search was already
split as far as the filters allow, or that is everything currently listed.
The source changed how it serves event data, which stopped the scraper returning
results. This release rebuilds it on the new source. Runs work again.
- Searches returned zero results for every input. Fixed.
- Speakers — names and profile links for events that list them.
- Attendee origin — visitor counts broken down by country.
- Ticket prices — ticket types with amounts, currency and conditions, plus
price ranges with validity dates.
- Event FAQs — the event's own question/answer pairs.
- Images — event images and a banner image.
attendanceMode and types, plus organizer login activity and a
sourceFields object carrying any remaining source values.- New input
scrapeDetails (default on) controls whether each event page is
opened for the fields above. Turn it off for a faster, cheaper run.
- Dates now filter by calendar month, then trim to your exact range. A range
spanning several months is scraped one month at a time, so those runs take
longer and cost more than before.
- The
city field takes a numeric city ID only (use the dropdown, or see the
README). City IDs are unchanged.
- Complete-coverage mode now triggers on what you ask for, not on an estimated
result count: it runs when
maxItems is 0 or above 200 and no category is
set. If you run with maxItems: 0, expect roughly 2-3x the cost of a single
pass. Set a category, or maxItems of 200 or less, to keep a run to one pass.
- Past events. Only upcoming events are listed now, covering roughly the next
12 months. A date range that has already passed returns no results and logs a
warning.
sortBy and sortType. Results can no longer be ordered at the source.
Passing them is harmless, they are ignored. Every record still includes
stats.score, stats.rating and stats.estimatedVisitors, so you can rank
after export.useResidentialProxy. Connection handling is automatic. Passing it is
ignored.
The structure grew from 59 to 68 keys. All previous keys are still present in the
same order, so existing parsers keep working.
These fields are no longer populated and are now always empty:
androidAppUrl, createdDate, currentEditionCreated, eventHost, highlights, introVideoUrl, iosAppUrl, parentEvent, paymentMethod, schedule, socialMedia, verified, location.mapImage, location.venueType, organizer.address, organizer.cityName, organizer.totalFollowers, stats.connections, stats.domesticArea, stats.domesticExhibitors, stats.domesticVisitors, stats.going, stats.internationalArea, stats.internationalExhibitors, stats.internationalVisitors, stats.reviews, stats.totalArea, stats.verifiedOn.
Note in particular that schedule (session agendas) and the international vs
domestic visitor, exhibitor and exhibition-area figures are gone. Attendee
origin is now reported per country in visitorsByCountry instead, which is more
detailed than the old international/domestic split.
Everything else, including stats.score, stats.rating, stats.estimatedVisitors,
stats.estimatedExhibitors, the full location block with coordinates, and the
organizer details, is unchanged.
