Danawa Product Scraper

Discover public Danawa products from Korean or English keywords, product URLs, or product codes. Export product names, brands, lowest advertised prices, aggregate offer counts, images, and aggregate ratings to an Apify dataset. Both legacy HTML and Next.js search pages are supported. Search pages identify products; detail-page JSON-LD supplies product data.

Example from the committed real detail fixture, captured October 14, 2026 (Singapore time):

Field	Value
pcode	98242505
name	ESSENCORE KLEVV CRAS C910G M.2 NVMe (1TB)
brand	ESSENCORE
lowestPrice / currency	287980 / KRW
offerCount	21
rating / ratingCount	4.9 / 23
category	null (not present in JSON-LD)
sourceUrl	https://prod.danawa.com/info/?pcode=98242505

Prices and availability change. offerCount is Danawa's aggregate offer/mall count, not an independently verified list of unique shops. The Actor does not retrieve full mall-by-mall pricing from AJAX endpoints. Missing JSON-LD fields are null; a missing or mismatched Product object is counted as an empty detail result.

{
  "keywords": ["ssd"],
  "pcodes": ["98242505"],
  "productUrls": [],
  "maxItems": 50,
  "fetchDetails": true,
  "proxyMode": "none",
  "detailConcurrency": 2
}

Parameter	Default	Meaning
keywords	[]	Nonempty search terms; at least one keyword, URL, or code is required.
productUrls	[]	HTTPS prod.danawa.com/info/?pcode=... URLs; extra query parameters are removed.
pcodes	[]	Positive decimal product codes as strings.
maxItems	50	Maximum unique product codes selected, from 1 to 10000. Failed/empty details can yield fewer rows. Explicit products take priority.
fetchDetails	true	When false, output only pcode and sourceUrl, with no detail requests.
proxyMode	none	none, datacenter (Apify auto selection without an explicit group), or residential. Direct search failures from timeouts, connection errors, HTTP 403, or HTTP 429 switch search requests to auto datacenter proxy once per run, within the two-retry limit.
detailConcurrency	2	Default two concurrent detail requests (one to three configurable). Any product-host 429 lowers concurrency to one for the rest of the run. Starts remain at least one second apart.

Duplicate products are emitted once across inputs, keywords, and pages. Explicit products are processed first, followed by each search page as it is discovered. Identifier-only results are saved in batches of up to ten rows; detailed results are saved after each group of up to ten detail attempts, including a final partial batch. Each batch uses one Actor.pushData(items, 'product') call. Pagination stops on an empty page, a page with no new codes, or once maxItems unique codes have been collected. With maxItems: 10 and 40 primary cards on page 1, only page 1 is requested. Failed or empty details do not trigger replacement search pages. An empty search page may indicate no matches or a changed page layout; warnings and counters make this visible.

Install and test with Node.js 20.18.1 or later:

npm install
npm test

Run npm run check:memory to repeatedly parse search and detail fixtures padded to exactly 3,000,000 bytes, with a 128 MiB V8 heap limit. The script retains only the latest ten parsed outputs, reports heapUsed, RSS and external memory before/after garbage collection, and fails if retained heap growth exceeds 8 MB. The default is 100 iterations; use npm run check:memory -- 200 to change it. This local parser check does not measure the complete platform Actor's memory.

To run locally, place the example input in storage/key_value_stores/default/INPUT.json, then run npm start. The default dataset appears under storage/datasets/default/; the run summary is storage/key_value_stores/default/OUTPUT.json. On Apify, the SDK uses the default cloud dataset and key-value store. The Actor definition sets default memory to 256 MB. No deployment is needed for local tests.

OUTPUT records discovered products, emitted and charged items, saved batches, failed requests/products/search pages, retries, duplicate products, empty search/detail counts, emptyResults, robotsDisallowedHits, unavailable robots files, wire and decompressed response byte counts, conservative residential reservations, controlled error descriptions, and a stop reason. HTTP failures and empty/malformed results do not create billable items. Validation or batch save errors fail the run after saving its summary. Runs that emit 0 usable items also call Actor.fail after saving OUTPUT, except a clean timeout stop.

Info logs report every HTTP attempt and every completed search/detail request with host, HTTP status (or null before a response), wire bytes (bytes), decompressed bytes (decompressedBytes), elapsed milliseconds, retries, and connection failures. Request totals include redirects and retries; their elapsed time also includes robots fetching, host pacing, and backoff. Robots requests have separate logs. OUTPUT.responseBytes and each host's responseBytes count encoded response body bytes before decompression, including chunked and partial failed responses. decompressedResponseBytes records decoded body bytes separately, globally and per host. These body counts exclude HTTP/TLS overhead. OUTPUT.requestsByHost contains attempt, retry, and byte totals; connectionFailures and connectionFailuresByHost count network failures separately from HTTP errors, size limits, parsing failures, and clean stops. Robots failures count toward their host's totals. The same summary is logged at the end of the run.

OUTPUT.timings.robotsByHost, firstSearchPage, and firstDetail record elapsedMs, completedAtMs since runActor started, and whether the phase succeeded. The first detail timing follows the first scheduled detail even if a later concurrent request finishes sooner. firstSaveMs records time until the first saved batch; totalMs records time through processing and batch saves, before writing OUTPUT. Unused phases remain null or absent from robotsByHost.

Response size limits (maxBytes) apply to encoded wire body bytes: 3,000,000 for search pages, 524,288 for detail pages, and 65,536 for robots.txt. Separate decoded limits bound memory: 8,000,000 for search, 2,000,000 for details, and 65,536 for robots.txt. Decoded bytes do not count toward proxy reservations. Transport decodes UTF-8 incrementally, and parsers tokenize HTML without constructing a full DOM. Bodies are discarded after parsing and before retry waits; only detached result fields, product codes, and metadata survive. Error summaries keep at most 100 descriptions of up to 2,049 characters each; errorsDropped counts additional descriptions while failure counters continue counting every failure.

Successful items incur one product event each through the SDK's batched pushData operation, with no additional explicit charge. PRICES in src/input.js contains a proposed USD 0.004/product price for review; this does not configure or change platform pricing. Configure a matching product event in Apify Console before any monetized deployment. The Actor respects the SDK's billing budget and counts partially affordable batches. Non-PPE local runs save all rows for free. Batch storage or SDK charging failures are reported as datasetError; the SDK stores rows before charging, so an error may require operator reconciliation. There is no pay_per_event.json and no automatic publishing.

Search request starts are at least ten seconds apart, including robots, redirects, and retries. Product request starts are at least one second apart; a longer robots crawl delay wins. The product host defaults to two in-flight attempts, including robots, redirects, and retries. A 429 lowers its cap to one as soon as response headers arrive, for the remainder of the run; existing attempts finish and queued attempts wait for capacity. Connection failures retry at most twice with waits of at least ten seconds. HTTP 403/429 retries use at least 30 and 60 seconds of backoff; a longer Retry-After also applies. Redirects are followed manually through the same pacing rules and restricted to the two Danawa hosts. No browser, login, cookie jar, or CAPTCHA solving is used.

The default run timeout is 3600 seconds (1 hour); keep it at 3600 for the default input. When ACTOR_TIMEOUT_AT is set, the Actor stops fetching at least 15 seconds before that timestamp, flushes usable pending details, and saves OUTPUT with stopReason: 'timeout'. Pacing and retry waits that would reach the stop deadline are skipped, and in-flight request timeouts are capped at that deadline. A timeout stop remains successful even when it has no rows. Without a valid timeout timestamp, the normal request timeouts still apply.

Residential mode reserves a maximum response allowance plus 128 KiB overhead before every attempt, including robots, redirects, and failures, within 5,000,000 bytes/run. Reservations are never refunded, and responses are streamed with size limits on both encoded and decoded bodies. This conservative budget can stop after only a few products; search pages consume most of it. It is application-level accounting, not a measured Apify invoice. Platform proxy transfer can include protocol overhead; verify actual usage before publication. Datacenter mode requires access to that Apify proxy group. Off mode never requests an Apify proxy configuration.

Is it legal to scrape Danawa?

This independent Actor collects publicly accessible product data without logging in and outputs no individual names, email addresses, telephone numbers, or reviewer identities. Public availability alone does not establish permission to collect or reuse data. You are responsible for checking Danawa's terms and applicable laws, including intellectual property, database rights, and Korea's privacy requirements. See Apify's discussion of scraping legality.

The Actor fetches each host's search robots.txt and product robots.txt and observes crawl delays. As required by this project's specification, a Disallow match produces a warning and increments OUTPUT.robotsDisallowedHits, but the request continues. This is not a claim of permission or full robots compliance. The normal search and detail routes in the committed robots fixtures are allowed; the Actor does not need prohibited price AJAX endpoints. Review the run warnings and your authorization before using changed or prohibited routes.

This tool is not affiliated with, endorsed by, or sponsored by Danawa. Data is supplied as observed for price monitoring and analysis. Image fields are source URLs; the Actor does not download or redistribute image files.

Fixture provenance, research assumptions, and local verification are recorded in NOTES.md. npm run fixtures:fetch uses curl with the same host pacing, redirect, and retry rules to refresh the real fixtures; live prices/layouts may change the pinned test expectations.