Hey, thank you for the feedback, it helps.

First a bit of explanation: I'm scraping public data only, otherwise it will be quite hard for me to maintain LinkedIn accounts to scrape from. So to test a certain scenario, do the search from your LinkedIn account, then copy-paste the same url to an incognito browser. You will see a different number of results (sometimes less, sometimes more).

Regarding LinkedIn search logic: It is a mystery to me how this works. In your particular scenario, if I do the search while logged in I get 8 results. Copy-paste url to incognito and I only see 3 results. So the scraper is working correctly (3 public results). (Private results may be different for you, as I live in a different region.)

A few ideas for you:

• When searching for remote jobs, if you set location to country instead of the city/state, you actually get more public results (21 private, 160 public)
• If you would search for on-site jobs instead of remote, you would actually get more public results (10 private, 20 public)
• Sometimes quoting the keywords helps to narrow down results (eg: "Java" instead if Java). But not in your case (not for your keywords).

I realize I'm not giving you a good solution, but I'm limited to LinkedIn public data. Maybe consider doing the search nation wide, then filter results in another software.

Thank you

Thanks a lot for the quick and detailed reply — really appreciate it!

That makes sense regarding public vs private data. One thing I’m still wondering: when I open the same search URL in an incognito window, I immediately get the login prompt and can’t see any results at all. Do others typically bypass this somehow, or disable the login popup to access the public results?

Anyway, totally understand the limitations and I’m grateful you shared this. Thanks again!

The login prompt is just a css overlay. I think sometimes it shows, sometimes not.

When you do web scraping is not relevant, you just parse the html underneath to get the links to the jobs detail pages, then you visit each job detail page.

When you surf, like you do, you can hide it in different ways:

• When you do a quick test (like I did), you right click in your browser, select "Inspect", then delete the correct html tags from the dom. But changes go away after first page refresh.

• If you surf regularly, then you need a browser plugin to hide it. I'm not familiar with one in particular, but you can probably find a configurable ad blocker that allows you to select what to hide. Another option probably is a browser plugin that allows you to load custom css, and you hide that dialog from css (but this requires coding skills).