Pay $5.00 for 1,000 products
Nordstrom Scraper
Pay $5.00 for 1,000 products
Nordstrom web scraper to crawl product information including price and sale price, color, and images. Extract all data in a dataset in multiple formats.
Request failed with status code 400 with correct body request
Closed
I am using the Nordstrom Scraper and I am getting a response "Request failed with status code 400" which seems to be APIFY related.
This issue is very common and we were forced to add a stricter retry mechanism in order to avoid failing but we want to make sure there is no issue on our end. The request is populated with this body:
"data": "{\"startUrls\":[\"https://shop.nordstrom.com/s/olaplex-no-5-bond-maintenance-conditioner/5056281\",\"https://shop.nordstrom.com/s/shiseido-urban-environment-sun-dual-care-oil-free-mineral-broad-spectrum-sunscreen-spf-42/6739785\",\"https://shop.nordstrom.com/s/silkn-infinity-hair-removal-device/5376009\",\"https://shop.nordstrom.com/s/vapour-soft-focus-foundation/5850728\",\"https://shop.nordstrom.com/s/triple-peptide-cactus-oasis-serum/7166106\",\"https://shop.nordstrom.com/s/biggie-chain-necklace/5787614\",\"https://shop.nordstrom.com/s/gucci-the-alchemists-garden-the-last-day-of-summer-eau-de-parfum/5519933\",\"https://shop.nordstrom.com/s/charlotte-tilbury-airbrush-flawless-foundation/5368721?\",\"https://shop.nordstrom.com/s/acqua-di-parma-blu-mediterraneo-bergamotto-di-calabria-eau-de-toilette-spray/3269001\",\"https://shop.nordstrom.com/s/polished-pebble-leather-crossbody-bag-nordstrom-exclusive/7411875?\",\"ttps://www.nordstrom.com/s/mz-wallace-metro-crossbody-bag/5447687\",\"https://shop.nordstrom.com/s/gh-bass-larson-leather-penny-loafer-men/6541600\",\"https://shop.nordstrom.com/s/mini-weekender-bag/7574158\",\"https://shop.nordstrom.com/s/gypsy-water-roll-on-perfumed-oil/4771717\",\"https://shop.nordstrom.com/s/weekender-water-repellent-tote-bag/7160706\",\"https://shop.nordstrom.com/s/ugg-scuffette-ii-slipper-women/2976610\",\"https://shop.nordstrom.com/s/estee-lauder-advanced-night-repair-synchronized-multi-recovery-complex-face-serum/5700327\",\"https://shop.nordstrom.com/s/schutz-arissa-square-toe-ballet-flat-women/5421998?\",\"https://shop.nordstrom.com/s/too-faced-lip-injection-maximum-plump-extra-strength-lip-plumper/5805920\",\"https://shop.nordstrom.com/s/holiday-advent-calendar-gift-set/7577829\",\"https://shop.nordstrom.com/s/monica-vinader-siren-muse-wide-ring/5909233?origin=category-personalizedsort&breadcrumb=home%2fbrands%2fmonica%20vinader&color=710\",\"https://shop.nordstrom.com/s/sigma-beauty-essential-kit-213-value/4171846\",\"https://shop.nordstrom.com/s/hourglass-unlocked-instant-extensions-mascara/5674521\",\"https://shop.nordstrom.com/s/yves-saint-laurent-libre-eau-de-parfum-spray-fragrance/5349306\",\"https://shop.nordstrom.com/s/jenny-bird-stevie-chain-necklace/5706793\",\"https://shop.nordstrom.com/s/nuface-trinity-smart-advanced-facial-toning-system-with-lip-eye-treatment-usd-619-value/5915367\",\"https://shop.nordstrom.com/s/nike-daybreak-sneaker-women/5282135\",\"https://shop.nordstrom.com/s/transporter-20-laptop-tote/7094505\",\"https://shop.nordstrom.com/s/birdies-the-songbird-mule-women/5955534\",\"https://shop.nordstrom.com/s/bobbi-brown-long-wear-cream-eyeshadow-stick/3442219\",\"https://shop.nordstrom.com/s/charlotte-tilbury-brow-fix-clear-brow-gel/5899495\",\"https://shop.nordstrom.com/s/crown-tonic-pre-shampoo-scalp-balancing-cleanser/7249829\",\"https://shop.nordstrom.com/s/westman-atelier-vital-skin-foundation-stick/6421115\",\"https://shop.nordstrom.com/s/luminous-silk-face-undereye-concealer/5536888\",\"https://shop.nordstrom.com/s/kevyn-aucoin-beauty-contour-book-the-art-of-sculpting-defining-volume-iii/5410772\",\"https://shop.nordstrom.com/s/pom-flexible-vibrator/7162310\",\"https://shop.nordstrom.com/s/teen-period-leak-proof-super-heavy-absorbency-hipster-panties/6831860?origin=keywordsearch-personalizedsort&breadcrumb=home%2fall%20results&color=001\",\"https://shop.nordstrom.com/s/purifying-blue-clay-mask/7547312\",\"https://shop.nordstrom.com/s/the-porefessional-tight-n-toned-pore-refining-ahapha-toner/7296624\",\"https://shop.nordstrom.com/s/ugg-neumel-boot-women/5241059\",\"https://shop.nordstrom.com/s/palladium-pallabrousse-legion-boot-unisex/5883988?color=black\",\"https://shop.nordstrom.com/s/la-mer-the-hand-treatment/3057062\",\"https://shop.nordstrom.com/s/tom-ford-private-blend-oud-wood-eau-de-parfum/3623561\",\"https://shop.nordstrom.com/s/dr-martens-1461-smooth-leather-oxford-men/6628332?origin=category-personalizedsort&breadcrumb=home%2fsale%2fmen%2fnew%20markdowns&color=292\",\"https://shop.nordstrom.com/s/tierra-bath-mat/7133342?color=lilac+multi\",\"https://shop.nordstrom.com/s/mac-187s-synthetic-duo-fibre-face-brush/4856658\",\"https://shop.nordstrom.com/s/dr-dennis-gross-skincare-derminfusions-fill-repair-serum/7324210\",\"https://shop.nordstrom.com/s/tata-harper-skincare-softening-cleanser/5872238?\",\"https://shop.nordstrom.com/s/lake-skye-11-11-fragrance-oil/5950372\",\"https://shop.nordstrom.com/s/madagascar-vanilla-perfume-oil/7153691\",\"https://shop.nordstrom.com/s/sun-fruit-body-oil/7370358\",\"https://shop.nordstrom.com/s/the-body-retinol/7381602\",\"https://shop.nordstrom.com/s/olaplex-no-4c-bond-maintenance-clarifying-shampoo/7130649?\",\"https://shop.nordstrom.com/s/dansko-bessie-genuine-shearling-lined-boot-women/6580640\",\"https://shop.nordstrom.com/s/aesop-shine-hair-oil/4364580/\",\"https://shop.nordstrom.com/s/dphue-root-touch-up-kit/5639421?\",\"https://shop.nordstrom.com/s/charlotte-tilbury-airbrush-flawless-makeup-setting-spray/5674721\",\"https://shop.nordstrom.com/s/dr-dennis-gross-skincare-advanced-retinol-ferulic-triple-correction-eye-serum/6426861\",\"https://shop.nordstrom.com/s/fresh-rose-hyaluronic-acid-deep-hydration-toner/4593821\",\"https://shop.nordstrom.com/s/radiant-creamy-concealer/3479528\",\"https://shop.nordstrom.com/s/delia-mary-jane-flat-women/7531974\",\"https://shop.nordstrom.com/s/hourglass-no-28-lip-treatment-oil/3930502\",\"https://shop.nordstrom.com/s/chantecaille-blue-light-protection-hyaluronic-serum/5248766\",\"https://shop.nordstrom.com/s/chanel-rouge-allure-ink-matte-liquid-lip-colour/4477887\",\"https://shop.nordstrom.com/s/dior-addict-lip-glow-balm/5929434\",\"https://shop.nordstrom.com/s/herms-les-mains-herms-nail-enamel/6520547?color=96+bleu+encre\",\"https://shop.nordstrom.com/s/mesh-body-exfoliator/5696625\",\"https://shop.nordstrom.com/s/beis-the-backpack/5305453\",\"https://shop.nordstrom.com/s/vionic-willa-loafer-women/5716737\",\"https://shop.nordstrom.com/s/whitney-leather-loafer-women/5969019\",\"https://shop.nordstrom.com/s/marine-layer-boyfriend-v-neck-t-shirt/5860637\",\"https://shop.nordstrom.com/s/leather-ballet-flat-women/7380729\",\"https://shop.nordstrom.com/s/vintners-daughter-active-botanical-serum/4657198\",\"https://shop.nordstrom.com/s/le-pliage-coin-purse/7184260?\",\"https://shop.nordstrom.com/s/le-creuset-lamour-mini-cocotte/6643572\",\"https://shop.nordstrom.com/s/curb-chain-necklace/7375580\",\"https://shop.nordstrom.com/s/nike-3-pack-dri-fit-ultra-stretch-micro-boxer-briefs/5739700\",\"https://shop.nordstrom.com/s/shiseido-urban-environment-sun-dual-care-fresh-moisture-broad-spectrum-sunscreen-spf-42/6739787\",\"https://shop.nordstrom.com/s/reigning-champ-lightweight-terry-hoodie/5379948\",\"https://shop.nordstrom.com/s/ocean-cleansing-milk/5812399\",\"https://shop.nordstrom.com/s/freeform-sports-bra/7379527\",\"https://shop.nordstrom.com/s/hadium-floral-print-pleated-wide-leg-sweatpants/7469449\",\"https://shop.nordstrom.com/s/2750-cotu-classic-sneaker-women/7332672\",\"https://shop.nordstrom.com/s/ashton-addie-waterproof-boot-women/6715374?origin=category-personalizedsort&breadcrumb=home%2fbrands%2fugg%2fboots%2ffor%20women&color=001\",\"https://shop.nordstrom.com/s/sunday-shampoo/4660242?\",\"https://shop.nordstrom.com/s/tulip-lace-balconette-bra/4277579?\",\"https://shop.nordstrom.com/s/wacoal-red-carpet-strapless-shaping-bodysuit/5236538?\",\"https://shop.nordstrom.com/s/skarlett-blue-goddess-multi-way-strapless-bra/4869924\",\"https://shop.nordstrom.com/s/skims-fits-everybody-bandeau-bralette-regular-plus-size/5548531\",\"https://shop.nordstrom.com/s/revitabrow-advanced-eyebrow-conditioner/3222999\",\"https://shop.nordstrom.com/s/calm-the-f-ck-down-deodorant-spray/7131451\",\"https://shop.nordstrom.com/s/cole-haan-grand-ambition-slip-on-loafer-men/6824649\",\"https://shop.nordstrom.com/s/dior-diorshow-waterproof-mascara/2865160\",\"https://shop.nordstrom.com/s/clarins-lip-comfort-oil/6671648\",\"https://shop.nordstrom.com/s/locked-kiss-ink-lipstick/7233003\",\"https://shop.nordstrom.com/s/lancome-dfinicils-defining-lengthening-mascara/2786551\",\"https://shop.nordstrom.com/s/byredo-bibliotheque-eau-de-parfum/4893896\",\"https://shop.nordstrom.com/s/le-labo-another-13-eau-de-parfum/4835770\",\"https://shop.nordstrom.com/s/slip-pure-silk-pillowcase/4186794\",\"https://shop.nordstrom.com/s/levis-90s-denim-trucker-jacket/7364733\",\"https://shop.nordstrom.com/s/ultra-facial-advanced-repair-barrier-cream/7261065\",\"https://shop.nordstrom.com/s/favorite-daughter-the-gemini-sleeveless-dress/7160250\",\"https://shop.nordstrom.com/s/reset-regimen-skin-care-set-143-value/7377709\",\"https://shop.nordstrom.com/s/ha-replenishing-serum/7323284\",\"https://shop.nordstrom.com/s/hyaluronic-acid-2-b5/5489304\",\"https://shop.nordstrom.com/s/magic-body-cream-nordstrom-exclusive/7460202\",\"https://shop.nordstrom.com/s/armani-beauty-luminous-silk-perfect-glow-flawless-oil-free-foundation/2853110\",\"https://shop.nordstrom.com/s/giorgio-armani-luminous-silk-perfect-glow-flawless-oil-free-foundation/2853110\",\"https://shop.nordstrom.com/s/supersonic-hair-dryer/6411114\",\"https://shop.nordstrom.com/s/anastasia-beverly-hills-satin-velvet-lipstick/6656234\",\"https://shop.nordstrom.com/s/cotton-corduroy-zip-up-shirt-jacket/7335517?origin=category-personalizedsort&breadcrumb=home%2fmen%2fclothing%2fcoats%20%26%20jackets%2fshirt%20jackets&color=005\",\"https://shop.nordstrom.com/s/sunday-riley-c-e-o-15-vitamin-c-brightening-serum/4827526\",\"https://shop.nordstrom.com/s/magic-cream-face-moisturizer-with-hyaluronic-acid/6296851\",\"https://shop.nordstrom.com/s/sharp-double-breasted-poplin-blazer/7368401?origin=keywordsearch-personalizedsort&breadcrumb=home%2fall%20results&color=500\",\"https://shop.nordstrom.com/s/kate-spade-new-york-knott-small-leather-crossbody-bag/6700317\"],\"proxy\":{\"useApifyProxy\":true,\"apifyProxyCountry\":\"US\"}}"
The request has been made at 29-Sept on 17:16:44.048 EEST.Request failed with status code 400
Seems like you are missing the country property used to get shipping information. Since it is a separate field, it is not related to the proxy settings and needs to be provided separately. I think in your case you can just use country: "United States".
bsq_tzam
Gustavo thank you for your answer. The version I am using is 1.2.24 which as far as I can see has the country property as optional.
Also, I would like to know if there are any change logs for older versions since I am planning to upgrade to 2.X version of this actor. I could only find change logs to 2.X here
Thats is the chagelog, I just upgrated the Apify SDK and made some improvements. I beleive you should be able tonuse it with no problems. I recomend you to always use the latest version since that is the version I am always maintaining.
bsq_tzam
With the new version the availability property is always populated as null in contrast with older version where it is in the ApiProduct. Is there an issue with the stock status?
Checking
I was able to add the apiProduct in the results again but I don't see availability on the products I have tried. Can you check it out with the latest version if the data you want is in the extra.apiProduct property?
bsq_tzam
Propery apiProduct is not consistently in the responses, depending on the links. I could not find a pattern to help you out debugging this. Do you have any insights for this?
Do you have a run id that resulted in products without apiProduct data?
bsq_tzam
Yes, you can check out run ID: QyEKZjqo6THJzKB3d Container URL: https://urhfhdqn75wd.runs.apify.net/
I have updated the actor, removed the old availability and added a new property called "isAvailable". Please try it out and let me know if that works for you.
bsq_tzam
Thanks, I will check it and get back to you. Another thing I noticed is that the new version takes considerably more time. Check this dataset's log of the latest version ( Dataset ID: SjiLZerVokyfrlUG0 ) compared to the one we are using currently ( Dataset ID: 4aIFdbGrD0ahjRAMf ). The new crawler version takes about 10 minutes for 60 links and the old version lasts about a minute. Can you please check this out too?
I have released a new version that should have improved speed when scraping product URLs.
bsq_tzam
I have attached a request body.
I have the same issue where I get 400 HTTP error Bad request.
I made 6 requests with the same body with a 1-minute difference, and all failed.
The requests took place at 5-11-2023 at 16:17 to 16:20 EET timezone.
As you can see the country is present in the body request and the actor version I have been using is 2.7.2.
Can you share the run ID with me so I can take a look at the logs algo?
bsq_tzam
There is no ID since the task was never started.
The list of "startUrls" is too big and Apify doesn't allow it. You can try to split into multiple runs with fewer startUrls on each or try to use a file as the source of startUrls.
bsq_tzam
It would be helpful to know what is the limit of this list since the way you define big is too abstract.
Agreed. Unfortunately, I wasn't able to find that information on Apify. That information seems to be missing from this page: https://docs.apify.com/platform/limits. I would recommend you use a txt file where every line is a URL and upload it as the startUrls file.
bsq_tzam
The txt approach you provided is not documented on how to run an actor. Am I missing something?
On Apify you have this option on the input section:
Gustavo Rudiger- 8 monthly users
- 3 stars
- 100.0% runs succeeded
- 55 days response time
- Created in Dec 2019
- Modified about 11 hours ago
