3 days trial then $25.00/month - No credit card required now

Dun & Bradstreet Scraper

epctex/dnb-scraper

3 days trial then $25.00/month - No credit card required now

Effortlessly extract valuable company information, financial projections, industry insights, and more from the extensive Dun & Bradstreet commercial database. Dive deep into the D&B Data Cloud, Business Directory, articles, companies, and industries with customized search terms.

All issues Create new issue

Actor doesn't return all requested companies and also returns companies which were not requested

Closed

scrape_crawl opened this issue

These companies were requested:

These were found:

These were returned but not requested:

And these were not found:

scrape_crawl

How do I tell the scraper to only crawl the submitted initial urls and not follow other urls found in the source?

tugkan

Hey Martin,

Thank you very much for reaching out. If you only want to retrieve the first page of the listing URLs, you can add the endPage property to 1. This will apply to all the sub-pages and let you crawl over only the initial URLs. About the problem above, it would be better if we could investigate how the actor behaved while you were trying it out. If possible, can you please share your Run ID? Which can be found on the top right of the screen called "Share" while you are on the Run Detail page.

Best

scrape_crawl

Thanks for explaining - the endPage property isn't documented here https://apify.com/epctex/dnb-scraper are there any other undocumented properties? As for your question this is a similar run ID where not all requested pages were in the response but the response also including pages which were not requested: 49fFAwGLzfVCfTHKt

tugkan

Hey there,

Thank you very much for sharing the Run ID and the missing property. We will update the documentation immediately. About the Run, we investigated what might go wrong there.

We found out that there are a couple of listing pages such as; https://www.dnb.com/business-directory/company-information.household_appliances_and_electrical_and_electronic_goods_merchant_wholesalers.in.tamil_nadu.kanchipuram.html
The maxItems property is set with 50 items.

In such cases, the actor enqueues all the URLs in place and starts scraping. Whenever it reaches 50 rows in output, it won't fetch the remaining start URLs. , In your case, the actor filled up this 50 limit with the ones that have been found over the listing URLs. I am not quite sure if that is the intended approach here. What exactly would you like to retrieve from the actor? Maybe we can build up an Input for achieving your goal.

Best

scrape_crawl-owner

Hi Tuğkan Cengiz, thanks for responding. In this case we only want to receive the profiles provided in the start URLs. As we were requesting 50 Profile Urls we set the maxItems property to 50 - any lower number would not follow all start urls. The Crawler should (in this case) not follow further links found on the profile pages but only scrape the provided profile.

tugkan_epctex

Hey Martin,

Thank you very much for explaining.

As mentioned, the main problem is the links that have been included in the Start URLs. Meaning that, if you provide the company profiles only, then the actor will retrieve that amount of rows in place and not touch any internal links. So in this case, you don't even need to use Max Items.

However, when a start Url includes a link with https://www.dnb.com/business-directory/company-information https://www.dnb.com/business-directory/company-information.household_appliances_and_electrical_and_electronic_goods_merchant_wholesalers.in.tamil_nadu.kanchipuram.html, the actor counts it as a listing page and starts scraping it over. Therefore, if a listing page is included, the actor will proceed forward with it until it hits the Max Items parameter. That is the main logic of the actor and unfortunately, it cannot be changed.

My suggestion for you to achieve your goal; would only provide the links which start with https://www.dnb.com/business-directory/company-profiles https://www.dnb.com/business-directory/company-profiles.mobis_india_limited.5ed64f1f15ee79861b7e876f46215921.html. These links are being counted as DnB Company Profiles as they should be. The couple of links you mentioned in your Input are the ones that contain many companies. That's where your goal is getting distracted.

Please let me know if I can help you with anything else,

Add comment

Developer

epctex

Actor metrics

34 monthly users
94.1% runs succeeded
0.047 days response time
Created in Mar 2021
Modified about 10 hours ago

Categories

Business

News

Marketing

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

63k

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Apify

43.6k

Instagram Hashtag Scraper

apify/instagram-hashtag-scraper

Scrape Instagram hashtags data. Just add one or more hashtags and extract posts, images, URLs, comments, likes, users, locations, timestamps, and more. Export scraped datasets, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

14k

Website Content Crawler

apify/website-content-crawler

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

Apify

13.1k

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract data from hundreds of Facebook posts from one or multiple Facebook pages and profiles. Get post URL, post text, page or profile URL, timestamp, number of likes, shares, comments, and more. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

7.9k

Contact Details Scraper

vdrmota/contact-info-scraper

Free email extractor to extract and download emails, phone numbers, Facebook, Twitter, LinkedIn, and Instagram profiles from any website. Extract contact information at scale from lists of URLs and download the data as Excel, CSV, JSON, HTML, and XML.

Vojta Drmota

17.3k

Twitter Scraper

quacker/twitter-scraper

Scrape tweets from any Twitter user profile. Top Twitter API alternative to scrape Twitter hashtags, threads, replies, followers, images, videos, statistics, and Twitter history. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Quacker

24.4k

GPT Scraper

drobnikj/gpt-scraper

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

Jakub Drobník

4.3k

AI Product Matcher

equidem/ai-product-matcher

Match products across multiple e-commerce websites. Use this AI product matching Actor whenever you need to find matching pairs of products from different online shops for dynamic pricing, competitor analysis or market research.

Matěj Sochor

309

Instagram Scraper

apify/instagram-scraper

Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

37.9k