# Danawa Product Scraper (`stone_jadex/danawa-product-scraper`) Actor

Discover public Danawa products and extract JSON-LD price comparison data.

- **URL**: https://apify.com/stone_jadex/danawa-product-scraper.md
- **Developed by:** [Stone Jade](https://apify.com/stone_jadex) (community)
- **Stats:** 2 total users, 1 monthly users, 54.5% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Danawa Product Scraper

Discover public Danawa products from Korean or English keywords, product URLs, or product codes. Export product names, brands, lowest advertised prices, aggregate offer counts, images, and aggregate ratings to an Apify dataset. Both legacy HTML and Next.js search pages are supported. Search pages identify products; detail-page JSON-LD supplies product data.

Example from the committed real detail fixture, captured October 14, 2026 (Singapore time):

| Field | Value |
| --- | --- |
| pcode | `98242505` |
| name | ESSENCORE KLEVV CRAS C910G M.2 NVMe (1TB) |
| brand | ESSENCORE |
| lowestPrice / currency | 287980 / KRW |
| offerCount | 21 |
| rating / ratingCount | 4.9 / 23 |
| category | `null` (not present in JSON-LD) |
| sourceUrl | https://prod.danawa.com/info/?pcode=98242505 |

Prices and availability change. `offerCount` is Danawa's aggregate offer/mall count, not an independently verified list of unique shops. The Actor does not retrieve full mall-by-mall pricing from AJAX endpoints. Missing JSON-LD fields are `null`; a missing or mismatched Product object is counted as an empty detail result.

```json
{
  "keywords": ["ssd"],
  "pcodes": ["98242505"],
  "productUrls": [],
  "maxItems": 50,
  "fetchDetails": true,
  "proxyMode": "none",
  "detailConcurrency": 2
}
```

| Parameter | Default | Meaning |
| --- | --- | --- |
| `keywords` | `[]` | Nonempty search terms; at least one keyword, URL, or code is required. |
| `productUrls` | `[]` | HTTPS `prod.danawa.com/info/?pcode=...` URLs; extra query parameters are removed. |
| `pcodes` | `[]` | Positive decimal product codes as strings. |
| `maxItems` | `50` | Maximum unique product codes selected, from 1 to 10000. Failed/empty details can yield fewer rows. Explicit products take priority. |
| `fetchDetails` | `true` | When false, output only `pcode` and `sourceUrl`, with no detail requests. |
| `proxyMode` | `none` | `none`, `datacenter` (Apify auto selection without an explicit group), or `residential`. Direct search failures from timeouts, connection errors, HTTP 403, or HTTP 429 switch search requests to auto datacenter proxy once per run, within the two-retry limit. |
| `detailConcurrency` | `2` | Default two concurrent detail requests (one to three configurable). Any product-host 429 lowers concurrency to one for the rest of the run. Starts remain at least one second apart. |

Duplicate products are emitted once across inputs, keywords, and pages. Explicit products are processed first, followed by each search page as it is discovered. Identifier-only results are saved in batches of up to ten rows; detailed results are saved after each group of up to ten detail attempts, including a final partial batch. Each batch uses one `Actor.pushData(items, 'product')` call. Pagination stops on an empty page, a page with no new codes, or once `maxItems` unique codes have been collected. With `maxItems: 10` and 40 primary cards on page 1, only page 1 is requested. Failed or empty details do not trigger replacement search pages. An empty search page may indicate no matches or a changed page layout; warnings and counters make this visible.

Install and test with Node.js 20.18.1 or later:

```sh
npm install
npm test
```

Run `npm run check:memory` to repeatedly parse search and detail fixtures padded to exactly 3,000,000 bytes, with a 128 MiB V8 heap limit. The script retains only the latest ten parsed outputs, reports `heapUsed`, RSS and external memory before/after garbage collection, and fails if retained heap growth exceeds 8 MB. The default is 100 iterations; use `npm run check:memory -- 200` to change it. This local parser check does not measure the complete platform Actor's memory.

To run locally, place the example input in `storage/key_value_stores/default/INPUT.json`, then run `npm start`. The default dataset appears under `storage/datasets/default/`; the run summary is `storage/key_value_stores/default/OUTPUT.json`. On Apify, the SDK uses the default cloud dataset and key-value store. The Actor definition sets default memory to 256 MB. No deployment is needed for local tests.

`OUTPUT` records discovered products, emitted and charged items, saved batches, failed requests/products/search pages, retries, duplicate products, empty search/detail counts, `emptyResults`, `robotsDisallowedHits`, unavailable robots files, wire and decompressed response byte counts, conservative residential reservations, controlled error descriptions, and a stop reason. HTTP failures and empty/malformed results do not create billable items. Validation or batch save errors fail the run after saving its summary. Runs that emit 0 usable items also call `Actor.fail` after saving `OUTPUT`, except a clean timeout stop.

Info logs report every HTTP attempt and every completed search/detail request with host, HTTP status (or `null` before a response), wire bytes (`bytes`), decompressed bytes (`decompressedBytes`), elapsed milliseconds, retries, and connection failures. Request totals include redirects and retries; their elapsed time also includes robots fetching, host pacing, and backoff. Robots requests have separate logs. `OUTPUT.responseBytes` and each host's `responseBytes` count encoded response body bytes before decompression, including chunked and partial failed responses. `decompressedResponseBytes` records decoded body bytes separately, globally and per host. These body counts exclude HTTP/TLS overhead. `OUTPUT.requestsByHost` contains attempt, retry, and byte totals; `connectionFailures` and `connectionFailuresByHost` count network failures separately from HTTP errors, size limits, parsing failures, and clean stops. Robots failures count toward their host's totals. The same summary is logged at the end of the run.

`OUTPUT.timings.robotsByHost`, `firstSearchPage`, and `firstDetail` record `elapsedMs`, `completedAtMs` since `runActor` started, and whether the phase succeeded. The first detail timing follows the first scheduled detail even if a later concurrent request finishes sooner. `firstSaveMs` records time until the first saved batch; `totalMs` records time through processing and batch saves, before writing `OUTPUT`. Unused phases remain `null` or absent from `robotsByHost`.

Response size limits (`maxBytes`) apply to encoded wire body bytes: 3,000,000 for search pages, 524,288 for detail pages, and 65,536 for robots.txt. Separate decoded limits bound memory: 8,000,000 for search, 2,000,000 for details, and 65,536 for robots.txt. Decoded bytes do not count toward proxy reservations. Transport decodes UTF-8 incrementally, and parsers tokenize HTML without constructing a full DOM. Bodies are discarded after parsing and before retry waits; only detached result fields, product codes, and metadata survive. Error summaries keep at most 100 descriptions of up to 2,049 characters each; `errorsDropped` counts additional descriptions while failure counters continue counting every failure.

Successful items incur one `product` event each through the SDK's batched `pushData` operation, with no additional explicit charge. `PRICES` in `src/input.js` contains a **proposed** USD 0.004/product price for review; this does not configure or change platform pricing. Configure a matching `product` event in Apify Console before any monetized deployment. The Actor respects the SDK's billing budget and counts partially affordable batches. Non-PPE local runs save all rows for free. Batch storage or SDK charging failures are reported as `datasetError`; the SDK stores rows before charging, so an error may require operator reconciliation. There is no `pay_per_event.json` and no automatic publishing.

Search request starts are at least ten seconds apart, including robots, redirects, and retries. Product request starts are at least one second apart; a longer robots crawl delay wins. The product host defaults to two in-flight attempts, including robots, redirects, and retries. A 429 lowers its cap to one as soon as response headers arrive, for the remainder of the run; existing attempts finish and queued attempts wait for capacity. Connection failures retry at most twice with waits of at least ten seconds. HTTP 403/429 retries use at least 30 and 60 seconds of backoff; a longer `Retry-After` also applies. Redirects are followed manually through the same pacing rules and restricted to the two Danawa hosts. No browser, login, cookie jar, or CAPTCHA solving is used.

The default run timeout is 3600 seconds (1 hour); keep it at 3600 for the default input. When `ACTOR_TIMEOUT_AT` is set, the Actor stops fetching at least 15 seconds before that timestamp, flushes usable pending details, and saves `OUTPUT` with `stopReason: 'timeout'`. Pacing and retry waits that would reach the stop deadline are skipped, and in-flight request timeouts are capped at that deadline. A timeout stop remains successful even when it has no rows. Without a valid timeout timestamp, the normal request timeouts still apply.

Residential mode reserves a maximum response allowance plus 128 KiB overhead **before every attempt**, including robots, redirects, and failures, within 5,000,000 bytes/run. Reservations are never refunded, and responses are streamed with size limits on both encoded and decoded bodies. This conservative budget can stop after only a few products; search pages consume most of it. It is application-level accounting, not a measured Apify invoice. Platform proxy transfer can include protocol overhead; verify actual usage before publication. Datacenter mode requires access to that Apify proxy group. Off mode never requests an Apify proxy configuration.

#### Is it legal to scrape Danawa?

This independent Actor collects publicly accessible product data without logging in and outputs no individual names, email addresses, telephone numbers, or reviewer identities. Public availability alone does not establish permission to collect or reuse data. You are responsible for checking Danawa's terms and applicable laws, including intellectual property, database rights, and Korea's privacy requirements. See [Apify's discussion of scraping legality](https://blog.apify.com/is-web-scraping-legal/).

The Actor fetches each host's [search robots.txt](https://search.danawa.com/robots.txt) and [product robots.txt](https://prod.danawa.com/robots.txt) and observes crawl delays. **As required by this project's specification, a Disallow match produces a warning and increments `OUTPUT.robotsDisallowedHits`, but the request continues.** This is not a claim of permission or full robots compliance. The normal search and detail routes in the committed robots fixtures are allowed; the Actor does not need prohibited price AJAX endpoints. Review the run warnings and your authorization before using changed or prohibited routes.

This tool is not affiliated with, endorsed by, or sponsored by Danawa. Data is supplied as observed for price monitoring and analysis. Image fields are source URLs; the Actor does not download or redistribute image files.

Fixture provenance, research assumptions, and local verification are recorded in [NOTES.md](NOTES.md). `npm run fixtures:fetch` uses curl with the same host pacing, redirect, and retry rules to refresh the real fixtures; live prices/layouts may change the pinned test expectations.

# Changelog

This Actor's version history is a separate document: https://apify.com/stone_jadex/danawa-product-scraper/changelog.md

# Actor input Schema

## `keywords` (type: `array`):

Korean or English search terms.

## `productUrls` (type: `array`):

HTTPS prod.danawa.com/info/?pcode=... product URLs.

## `pcodes` (type: `array`):

Positive decimal product IDs as strings.

## `maxItems` (type: `integer`):

Maximum unique product codes selected. Failed products may result in fewer output rows.

## `fetchDetails` (type: `boolean`):

Disable to output only product code and canonical source URL.

## `proxyMode` (type: `string`):

Direct by default, with one automatic search fallback to Apify auto datacenter proxy on timeout, connection error, 403 or 429. Datacenter uses auto selection. Residential mode reserves a conservative maximum of 5 MB per run.

## `detailConcurrency` (type: `integer`):

Default two detail requests in flight (one to three configurable); any product-host 429 lowers concurrency to one for the rest of the run. Starts remain at least one second apart.

## Actor input object example

```json
{
  "keywords": [
    "노트북"
  ],
  "productUrls": [],
  "pcodes": [],
  "maxItems": 50,
  "fetchDetails": true,
  "proxyMode": "none",
  "detailConcurrency": 2
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keywords": [
        "노트북"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("stone_jadex/danawa-product-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "keywords": ["노트북"] }

# Run the Actor and wait for it to finish
run = client.actor("stone_jadex/danawa-product-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keywords": [
    "노트북"
  ]
}' |
apify call stone_jadex/danawa-product-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,stone_jadex/danawa-product-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1SJWEysf5KSeXsQWK/builds/JBgb96h1av2PfVRgj/openapi.json