# FreshDirect Grocery Scraper (`crawlerbros/freshdirect-scraper`) Actor

Scrape FreshDirect - US Northeast online grocery. Search the catalog by keyword, browse departments/categories, fetch products by ID. Get name, brand, size, price, sale price, unit price, allergens, ingredients, images. No auth required.

- **URL**: https://apify.com/crawlerbros/freshdirect-scraper.md
- **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community)
- **Categories:** E-commerce, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## FreshDirect Grocery Scraper

Scrape **FreshDirect** — the online grocery delivery store serving the US Northeast. Search the catalog by keyword, browse departments and categories, fetch full product-detail pages by ID, or pull today's deals — and get product name, brand, size, price, sale price, discount, unit price, EBT eligibility, allergens, ingredients, claims and product photos. No auth, no proxy required.

> **Data source note:** This actor was originally requested for *Kroger (kroger.com)*. Kroger.com is hard-blocked from Apify cloud egress — Playwright `page.goto` times out (60s × 3) even with the Apify proxy, and TLS-impersonated requests get 403 from the platform's bot defense. FreshDirect (freshdirect.com) is the replacement: a major US online grocery delivery service in the same grocery-retail category, verified reachable from Apify cloud with plain egress (HTTP 200 on every probe).

### What this actor does

- **Six modes:** `search` (keyword), `byCategory`, `byDepartment`, `byProductIds` (full detail pages), `byUrls`, `deals`
- **Full catalog axis:** every department page embeds the complete category taxonomy — any category id can be browsed
- **Rich product records:** price, was-price, discount amount, deal percent, unit price, EBT eligibility, availability
- **Full detail pages:** allergens, ingredients, claims, nutrition highlights, origin, category breadcrumbs (mode `byProductIds`)
- **Filters:** min/max price, EBT-eligible only, in-stock only
- **Empty fields are omitted**

### Output per product

- `productId`, `skuCode`, `categoryId`
- `productName`, `productDescription`, `brandName`, `akaName`
- `unitSize`, `unitPrice`, `scaleUnit`, `servingSize`, `formattedCurrentPrice`
- `price` — current price in USD
- `wasPrice` — original price before discount
- `discountAmount`, `dealPercent`, `savingString` — savings when on sale
- `ebtEligible`, `available`, `alcoholic`, `discontinuedSoon`
- `minQuantity`, `maxQuantity`, `quantityIncrement`
- `soldOut`, `sponsored`, `yourFave`, `backOnline`, `new` — marketing tags
- `topPick`, `freeSample`, `expressEligible` — feature tags
- `productUrl`, `imageUrl` (zoom), `jumboImageUrl`, `alternateImageUrl`, `productImageUrl`, `detailImageUrl`
- `categoryPath[]`, `allergens[]`, `claims[]`, `organicClaims[]`, `ingredients`, `extraDescription`, `origin`, `seasonText`, `kosherSymbol`, `halalSymbol`, `freshnessGuarantee` — on product-detail records (`recordType: "productDetail"`)
- `sourceUrl`, `recordType`, `scrapedAt`

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `mode` | string | `search` | `search` / `byCategory` / `byDepartment` / `byProductIds` / `byUrls` / `deals` |
| `searchQuery` | string | `milk` | Free-text query (mode=search) |
| `categoryId` | string | – | Category id to browse (mode=byCategory), e.g. `dai_milk_cream` |
| `departmentId` | string | – | Department to browse (mode=byDepartment), e.g. `dai` or `supergro` |
| `productIds` | array | – | Product IDs for full detail fetch (mode=byProductIds) |
| `urls` | array | – | FreshDirect URLs to scrape (mode=byUrls) |
| `minPrice` | number | – | Min price in USD |
| `maxPrice` | number | – | Max price in USD |
| `ebtEligibleOnly` | boolean | `false` | Only EBT-eligible products |
| `inStockOnly` | boolean | `false` | Only available products |
| `maxItems` | int | `50` | Hard cap (1–500) |
| `useApifyProxy` | boolean | `false` | Optional AUTO-proxy fallback for blocked requests |

#### Example: search for a product

```json
{
  "mode": "search",
  "searchQuery": "organic eggs",
  "maxItems": 100
}
```

#### Example: browse an entire department

```json
{
  "mode": "byDepartment",
  "departmentId": "dai",
  "maxItems": 200
}
```

#### Example: fetch full product detail pages

```json
{
  "mode": "byProductIds",
  "productIds": ["dai_pid_2005607", "dai_orgval_whlmilk_01"],
  "maxItems": 5
}
```

#### Example: scrape today's deals under $10

```json
{
  "mode": "deals",
  "maxPrice": 10,
  "maxItems": 20
}
```

### Use cases

- **Price monitoring** — track grocery prices and discounts over time
- **Competitor research** — compare product assortments and price points across departments
- **Meal planning apps** — pull real product names, sizes and unit prices
- **Nutrition intelligence** — allergens, ingredients and claims from product-detail pages
- **SNAP/EBT analysis** — identify the EBT-eligible product set by department
- **Promotion tracking** — daily deal-page scrapes for sale/discount analytics

### FAQ

**What is the data source?** FreshDirect (freshdirect.com), the online grocery delivery service. This is a third-party actor using the public website — not affiliated with FreshDirect.

**Why did this actor replace Kroger?** Kroger.com hard-blocks datacenter egress (Playwright page loads time out with the Apify proxy; TLS-impersonated HTTP gets 403). FreshDirect is the same grocery-retail category and serves its full catalog over plain HTTPS without a WAF.

**Are the prices real-time?** Yes — every run reflects the current prices shown on FreshDirect at scrape time.

**How do I find a category id?** Category ids appear in FreshDirect URLs (`/dai/sc/dai_milk_cream` → `dai_milk_cream`) and in the `categoryId` dropdown, which lists popular categories. Any category id from the site works.

**What is a superdepartment?** `deli_prepared`, `fresh_produce`, `meat_seafood` and `supergro` group several departments. The actor automatically expands them into their sub-departments.

**Why are some fields missing on some records?** Empty fields are omitted. Fields like `wasPrice`, `dealPercent` or `allergens` only exist when a product is on sale or when a detail page is fetched.

**Are there rate limits?** FreshDirect does not publish limits. The actor uses polite delays and retries with backoff on 429/5xx.

**Can I scrape without a proxy?** Yes — this is the default. The optional Apify AUTO proxy is only a fallback if you ever observe blocks.

**How fresh is the data?** Live — each run scrapes the current catalog state.

# Actor input Schema

## `mode` (type: `string`):

What to fetch.

## `searchQuery` (type: `string`):

Free-text query (mode=search), e.g. `milk`, `organic chicken`, `chocolate`.

## `categoryId` (type: `string`):

FreshDirect category id to browse, e.g. `dai_milk_cream` (Cream & Creamers). Popular categories are listed below; any category id works.

## `departmentId` (type: `string`):

FreshDirect department to browse. Superdepartments (Prepared & Deli, Produce, Meat & Poultry, Grocery) are expanded into their sub-departments automatically.

## `productIds` (type: `array`):

FreshDirect product IDs (e.g. `dai_pid_2005607`, `dai_orgval_whlmilk_01`). Full product-detail pages are fetched for each.

## `urls` (type: `array`):

Any FreshDirect URL: search pages (`/search.jsp?searchParams=...`), category pages (`/sc/<categoryId>`), department pages (`/d/<deptId>`, `/sd/<superId>`), product pages (`/product/<productId>`), or the deals page.

## `minPrice` (type: `number`):

Only emit products priced at or above this amount.

## `maxPrice` (type: `number`):

Only emit products priced at or below this amount.

## `ebtEligibleOnly` (type: `boolean`):

Only emit products eligible for SNAP/EBT payment.

## `inStockOnly` (type: `boolean`):

Only emit products marked as available.

## `maxItems` (type: `integer`):

Hard cap on emitted records.

## `useApifyProxy` (type: `boolean`):

Enable to fall back to the Apify AUTO proxy if a request is blocked (403/429). FreshDirect works without a proxy; enable only if you see blocks.

## `proxyGroups` (type: `array`):

Apify proxy groups used when `useApifyProxy` is enabled.

## Actor input object example

```json
{
  "mode": "search",
  "searchQuery": "milk",
  "productIds": [
    "dai_pid_2005607"
  ],
  "urls": [
    "https://www.freshdirect.com/search.jsp?searchParams=milk"
  ],
  "ebtEligibleOnly": false,
  "inStockOnly": false,
  "maxItems": 10,
  "useApifyProxy": false,
  "proxyGroups": [
    "AUTO"
  ]
}
```

# Actor output Schema

## `products` (type: `string`):

Dataset containing all scraped FreshDirect products and product-detail records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "search",
    "searchQuery": "milk",
    "productIds": [
        "dai_pid_2005607"
    ],
    "urls": [
        "https://www.freshdirect.com/search.jsp?searchParams=milk"
    ],
    "ebtEligibleOnly": false,
    "inStockOnly": false,
    "maxItems": 10,
    "useApifyProxy": false,
    "proxyGroups": [
        "AUTO"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("crawlerbros/freshdirect-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "search",
    "searchQuery": "milk",
    "productIds": ["dai_pid_2005607"],
    "urls": ["https://www.freshdirect.com/search.jsp?searchParams=milk"],
    "ebtEligibleOnly": False,
    "inStockOnly": False,
    "maxItems": 10,
    "useApifyProxy": False,
    "proxyGroups": ["AUTO"],
}

# Run the Actor and wait for it to finish
run = client.actor("crawlerbros/freshdirect-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "search",
  "searchQuery": "milk",
  "productIds": [
    "dai_pid_2005607"
  ],
  "urls": [
    "https://www.freshdirect.com/search.jsp?searchParams=milk"
  ],
  "ebtEligibleOnly": false,
  "inStockOnly": false,
  "maxItems": 10,
  "useApifyProxy": false,
  "proxyGroups": [
    "AUTO"
  ]
}' |
apify call crawlerbros/freshdirect-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=crawlerbros/freshdirect-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MSN0EUAFjGqdLW2Xs/builds/qr1X1G7iGDmSVXLDV/openapi.json