Car data scraper: specs, 0-60 times and power
Pricing
from $3.00 / 1,000 car specs
Extract consolidated car specs and performance for any brand or model: power, torque, 0-60 mph, top speed and full technical data. auto-data is the backbone, three more sources enrich a variant when they match, and every value carries its source. By MrBridge
Restored: auto-data changed its HTML and the Actor read it as a block.
<table class="cardetailsout"><tr> and into <div class="cardetailsout"> with
<div class="row"><div class="par">label</div><div class="val">value</div></div>. Not
one <tr> is left on the page. The parser's selector matched nothing, no field was
extracted, and the "zero fields" guard reported TARGET_BLOCKED. The site had been
answering HTTP 200 with complete pages the whole time.
The parser now reads both shapes, the current one and the legacy one, using the
same label mappings as before. Nothing about the extracted values changes: the real
captured page yields 62 fields and a complete identity.stopReason: "targetBlocked" and a new blockedBackboneRequests count in the
OUTPUT record, so a refusal is distinguishable from a timeout and from an internal
error, which need different responses.totalErrorRows was reported as 0 while error rows were being written. It was only
counted on the invalid-input path. It now counts every error row, after the write
confirms. Error rows are still never charged.make, model, generation and modification
are never present on an error row, because the failure happens before a variant is
identified, and url, the address that failed, was missing. The view is now error,
errorMessage, source, url, scrapedAt: five columns that are always filled.lastFullyCompletedModel claimed a model on runs that produced no rows. It stays
null unless at least one record was delivered.The demo video.
Electrified powertrains, honest copy, and observability.
Electric and hybrid variants now carry their power, and one existing column changes
value on hybrids. auto-data publishes the total output of an electrified variant under
System power and System torque, rows this Actor never read, while the Power row it
did read describes the combustion engine alone. A battery-electric variant therefore
returned no power at all, and a hybrid returned the wrong one. powerHp and torqueNm
now carry the system figure whenever the page prints one.
This changes a value that was already present, on hybrids only. Measured on the
fixtures: a Chevrolet Volt II 1.5 Plug-in Hybrid published powerHp 101 and now
publishes 150, which is the figure its own variant name quotes and the figure auto-data
answers to "what is the power output". A Ford Mondeo IV Hybrid published torqueNm 173
and now publishes 300. Combustion and mild-hybrid rows are unchanged, because those
pages print no System power row: an Audi TT RS stays at 394 hp and 480 Nm, a Mercedes
E 220d Mild Hybrid at 197 hp and 440 Nm.
Nothing published is overwritten and no value is computed. The nested engine
section keeps power_hp and torque_nm exactly as the Power and Torque rows printed
them, beside the new system keys, so the combustion figure of a hybrid is still there.
The canonicalVariantId reads the nested figure and therefore does not move.
The matcher also reads the nested figure, so cross-source matching behaviour is
unchanged by this release.
The progress counter is now monotonic. A heartbeat could report collected 0 variant(s)
after Collected 8 variant(s) had already been shown, because it read a counter the
pipeline only filled at the end, and because two status writes could settle out of order.
Both are closed: one shared counter that never decreases, one serialized write path, and
each message rendered from the counter at the moment it is written.
The terminal status of a run stopped by its timeout now reads
Stopped (run timeout reached) rather than Stopped (deadline), and states that
delivered rows are kept. stopReason in OUTPUT is still deadline, so integrations
reading it are unaffected.
The Input mode field now defaults to Model names list, the same value the form is
prefilled with. An API call that omitted the field and supplied both a brand and a models
list used to expand the whole brand. A call that supplies only a brand, or only models,
behaves exactly as before, so existing tasks and integrations are unaffected.
The Max items description now states that run time follows the number of variants a model carries rather than the cap alone.
Power figures from auto-data are now compared against carperformancedata using two
anchors, the published figure and its metric-to-SAE conversion, whichever is nearer. The
published power_hp is untouched: the rule applies inside matching only, and only to that
one source pair, whose two possible conventions are documented.
dedupeConfidence can change as a result, even when the matched variant and its
confidence are identical. On the evaluation corpus one row moves, from 0.96 to 1.00,
with the same winning variant and the same exact confidence. No row changes which
variant it matched, and no power_hp value changes.
batteryCapacityKwh is the gross battery
capacity, electricRangeKm is the all-electric range as published under Electric range
or Electric range (WLTP), and powerBasis and torqueBasis name which auto-data row
filled powerHp and torqueNm, with the value system or engine. All four are null
where the source publishes nothing, and none of them is estimated or converted.engine and performance sections: system_power_hp,
system_power_rpm, system_torque_nm, system_torque_rpm, electric_motor_power_hp,
electric_motor_torque_nm, battery_capacity_kwh and electric_range_km.OUTPUT now carries sourcesMatched, the number of delivered rows carrying each
enabled source. sourcesEnabled says what was configured, sourcesMatched says what
contributed, and a run with four sources enabled and three zeros is a single-source run.
It is derived from the coverage ledger rather than counted separately, so the two can
never disagree.More of the spec sheet, and provenance you can audit.
fuelEconomy.co2_wltp_gkm and fuelEconomy.co2_nedc_gkm are new nested keys, and the new flat column co2Standard names which standard the reference figure follows.
co2GKm and fuelEconomy.co2_gkm keep their exact previous values. They remain the reference figure, WLTP when the source publishes one and NEDC otherwise. This release does not change either of them for any row.measurementKind is a new provenance property on a traced value, present only when the source states how a figure was obtained. It never defaults to measured, because a source that says nothing has not told you it measured anything. Its single flat mirror is the new column zeroToSixtyMphMethod, which is null when the source did not say.upstreamSourceName and upstreamSourceUrl are new provenance properties on the sourcesDetail entries. carperformancedata.com republishes figures measured by someone else and credits that publication on some of its records. That credit is now preserved. source stays the technical id and sourceUrl stays the page actually read, so the two describe the upstream publication and nothing else. Both are absent when no credit is published, and the other three sources publish none.compression_ratio, bore_mm, stroke_mm and trunk_l (boot volume, seats in place) were declared before this release and nothing populated them, so they never appeared in a record. They are now read from auto-data.
This changes what you see in an existing column: the flat column compression was null on every row and is now filled wherever auto-data publishes a compression ratio, as "10:1". On a whole-BMW run that is 970 of 1000 rows. No value that was already present has been altered: the change is empty to filled, never one value to another.injection_system, valvetrain, engine_layout, engine_configuration, powertrain_architecture), servicing capacities (oil_capacity_l, coolant_capacity_l), payload and towing geometry (gross_weight_kg, payload_kg, roof_load_kg, width_mirrors_mm, front_overhang_mm, rear_overhang_mm, turning_circle_m), the boot maximum trunk_max_l (kept separate from the minimum so the two can never be mixed), transmission prose (drivetrain_architecture) and chassis text (power_steering, wheel_rims).co2Standard and zeroToSixtyMphMethod. New provenance properties are measurementKind, upstreamSourceName and upstreamSourceUrl, and they hang off values and source entries rather than being fields of their own.A run that crashes now says so.
OUTPUT is attempted before the run fails, so a failed run is still explainable: it carries stopReason: "error" and the original message. If writing the summary also fails, that is logged as a secondary problem and never replaces the original error.Three missing brands, and three descriptions that did not match the product.
view=errors API calls keep working.Runs that stop on their own terms, and a billing contract that matches reality.
OUTPUT on every ending the Actor controls: normal completion, invalid input, time budget reached, spend limit reached, or an unexpected error. It reports how the run ended, how many records were delivered and billed, and the last model fully completed. It cannot be written when the platform kills the container outright.canonicalVariantId is documented as a grouping key, not a unique one. Variants that differ only in fuel, gearbox or door count share an id. For a unique key, use the auto-data source URL in sourcesDetail. The id format itself is unchanged in this release.excludeModels is documented as an exact, case-insensitive match on the model or make and model. It always behaved this way, the description said "contains".maxItems: 0 (no record cap) is now rejected on a run that also has no timeout, since nothing would stop it. Behaviour change for API callers: maxItems: null now means the default cap of 1000, where it previously meant no cap.Faster, cheaper default runs.
sources option): ultimatespecs.com is slower and nests performance variants under their base model.Audi RS3) with a low Max items, so a first run finishes in seconds for a few cents instead of walking a whole brand. Use the Brand mode and raise Max items for larger pulls.Input UX and matching improvements.
Initial release (v0.1.0).
hasConflict flag and the maximum deviation when sources disagree beyond a built-in tolerance.maxItems safety cap.