A robust, high-performance utility designed for developer automation, data integration, and AI training. Features built-in captcha bypass, headful/headless browser execution, and proxy support to scrape GSTIN data seamlessly, reliably, and at scale.
Made the CAPTCHA solver endpoint environment-only (CAPTCHA_API_URL) and HTTPS-only so ordinary run input cannot redirect an environment secret.
Reject missing solver credentials before creating a proxy, browser, or GST Portal request.
Fixed
Fail zero-result and partial-result runs instead of swallowing every per-GSTIN error and exiting successfully.
Write a bounded OUTPUT summary with sanitized per-item diagnostics on every terminal path.
Repair the Docker build by removing a reference to a nonexistent pre-copy version-check script, matching Playwright 1.60.0 to its base image, and using deterministic npm ci.
Removed the embedded CAPTCHA solver credential. Solver credentials now come only from the secret input or CAPTCHA_API_KEY environment variable and are never logged or emitted.
Changed
Rebuilt the browser workflow around session-matched fresh CAPTCHA images, bounded SWEB_9000 retries, validated GSTIN response matching, coherent fingerprints, and direct/rotating/fixed proxy metadata.
Replaced fake error dataset rows with truthful zero-row failures.
Expanded the schema with verified identity, address, jurisdiction, HSN/SAC, filing, endpoint provenance, proxy, fingerprint, status, rank and timestamp fields.
Added strict dataset validation and unit coverage for pure normalization helpers.
Added
State Jurisdiction Parsing: The stateJurisdiction field is now automatically parsed into separate fields for better data organization and filtering:
state: State name extracted from stateJurisdiction
division: Division extracted from stateJurisdiction
zone: Zone extracted from stateJurisdiction
circle: Circle/office name extracted from stateJurisdiction (without "Jurisdictional Office" suffix)
Added new fields to dataset schema with proper descriptions
Updated business_details view in dataset schema to include the new jurisdiction fields
Added state filter option in business_details view
Added missing fields to dataset schema: adhrVdt (Aadhaar validation date) and district
Changed
Enhanced transformTaxpayerDetailsKeys function in scraper to parse jurisdiction string
Updated README.md to document the new jurisdiction fields
Original stateJurisdiction field is retained for reference alongside the new parsed fields
Fixed dataset schema validation: Made array schemas flexible to allow variable-length arrays instead of fixed-length tuples
Updated schema to allow null values for optional fields
Technical Details
Parser handles jurisdiction strings in format: "State - [NAME],Division - [NAME],Zone - [NAME],Circle - [NAME] (Jurisdictional Office)"
Fields are set to null if the corresponding component is not found in the jurisdiction string
