FreshDirect Grocery Scraper

Scrape FreshDirect — the online grocery delivery store serving the US Northeast. Search the catalog by keyword, browse departments and categories, fetch full product-detail pages by ID, or pull today's deals — and get product name, brand, size, price, sale price, discount, unit price, EBT eligibility, allergens, ingredients, claims and product photos. No auth, no proxy required.

Data source note: This actor was originally requested for Kroger (kroger.com). Kroger.com is hard-blocked from Apify cloud egress — Playwright page.goto times out (60s × 3) even with the Apify proxy, and TLS-impersonated requests get 403 from the platform's bot defense. FreshDirect (freshdirect.com) is the replacement: a major US online grocery delivery service in the same grocery-retail category, verified reachable from Apify cloud with plain egress (HTTP 200 on every probe).

What this actor does

• Six modes: search (keyword), byCategory, byDepartment, byProductIds (full detail pages), byUrls, deals
• Full catalog axis: every department page embeds the complete category taxonomy — any category id can be browsed
• Rich product records: price, was-price, discount amount, deal percent, unit price, EBT eligibility, availability
• Full detail pages: allergens, ingredients, claims, nutrition highlights, origin, category breadcrumbs (mode byProductIds)
• Filters: min/max price, EBT-eligible only, in-stock only
• Empty fields are omitted

Output per product

• productId, skuCode, categoryId
• productName, productDescription, brandName, akaName
• unitSize, unitPrice, scaleUnit, servingSize, formattedCurrentPrice
• price — current price in USD
• wasPrice — original price before discount
• discountAmount, dealPercent, savingString — savings when on sale
• ebtEligible, available, alcoholic, discontinuedSoon
• minQuantity, maxQuantity, quantityIncrement
• soldOut, sponsored, yourFave, backOnline, new — marketing tags
• topPick, freeSample, expressEligible — feature tags
• productUrl, imageUrl (zoom), jumboImageUrl, alternateImageUrl, productImageUrl, detailImageUrl
• categoryPath[], allergens[], claims[], organicClaims[], ingredients, extraDescription, origin, seasonText, kosherSymbol, halalSymbol, freshnessGuarantee — on product-detail records (recordType: "productDetail")
• sourceUrl, recordType, scrapedAt

Input

Field	Type	Default	Description
mode	string	search	search / byCategory / byDepartment / byProductIds / byUrls / deals
searchQuery	string	milk	Free-text query (mode=search)
categoryId	string	–	Category id to browse (mode=byCategory), e.g. dai_milk_cream
departmentId	string	–	Department to browse (mode=byDepartment), e.g. dai or supergro
productIds	array	–	Product IDs for full detail fetch (mode=byProductIds)
urls	array	–	FreshDirect URLs to scrape (mode=byUrls)
minPrice	number	–	Min price in USD
maxPrice	number	–	Max price in USD
ebtEligibleOnly	boolean	false	Only EBT-eligible products
inStockOnly	boolean	false	Only available products
maxItems	int	50	Hard cap (1–500)
useApifyProxy	boolean	false	Optional AUTO-proxy fallback for blocked requests

Example: search for a product

{
  "mode": "search",
  "searchQuery": "organic eggs",
  "maxItems": 100
}

Example: browse an entire department

{
  "mode": "byDepartment",
  "departmentId": "dai",
  "maxItems": 200
}

Example: fetch full product detail pages

{
  "mode": "byProductIds",
  "productIds": ["dai_pid_2005607", "dai_orgval_whlmilk_01"],
  "maxItems": 5
}

Example: scrape today's deals under $10

{
  "mode": "deals",
  "maxPrice": 10,
  "maxItems": 20
}

Use cases

• Price monitoring — track grocery prices and discounts over time
• Competitor research — compare product assortments and price points across departments
• Meal planning apps — pull real product names, sizes and unit prices
• Nutrition intelligence — allergens, ingredients and claims from product-detail pages
• SNAP/EBT analysis — identify the EBT-eligible product set by department
• Promotion tracking — daily deal-page scrapes for sale/discount analytics

FAQ

What is the data source? FreshDirect (freshdirect.com), the online grocery delivery service. This is a third-party actor using the public website — not affiliated with FreshDirect.

Why did this actor replace Kroger? Kroger.com hard-blocks datacenter egress (Playwright page loads time out with the Apify proxy; TLS-impersonated HTTP gets 403). FreshDirect is the same grocery-retail category and serves its full catalog over plain HTTPS without a WAF.

Are the prices real-time? Yes — every run reflects the current prices shown on FreshDirect at scrape time.

How do I find a category id? Category ids appear in FreshDirect URLs (/dai/sc/dai_milk_cream → dai_milk_cream) and in the categoryId dropdown, which lists popular categories. Any category id from the site works.

What is a superdepartment? deli_prepared, fresh_produce, meat_seafood and supergro group several departments. The actor automatically expands them into their sub-departments.

Why are some fields missing on some records? Empty fields are omitted. Fields like wasPrice, dealPercent or allergens only exist when a product is on sale or when a detail page is fetched.

Are there rate limits? FreshDirect does not publish limits. The actor uses polite delays and retries with backoff on 429/5xx.

Can I scrape without a proxy? Yes — this is the default. The optional Apify AUTO proxy is only a fallback if you ever observe blocks.

How fresh is the data? Live — each run scrapes the current catalog state.