Pricing

Pay per usage

Go to Store

Sitemap To Request Queue

Try for free

Developed by

Paulo Cesar

Download sitemap XMLs and put them in a RequestQueue

0.0 (0)

Pricing

Pay per usage

Total users

104

Monthly users

Runs succeeded

>99%

Last modified

3 years ago

Developer tools

Open source

Sitemap to RequestQueue

Downloads a sitemap.xml files and append them to a RequestQueue of your choice.

Example

// this is your actor
Apify.main(async () => {
  const { proxyConfig } = await Apify.getInput();
  const requestQueue = await Apify.openRequestQueue();

  // this is needed so it doesn't execute everytime there's a migration
  const run = (await Apify.getValue('SITEMAP-CALL', run)) || { runId: '', actorId: '' };

  if (!run || !run.runId) {
    // this might take a while!
    const runCall = await Apify.call('pocesar/sitemap-to-request-queue', {
      // required proxy configuration, like { useApifyProxy: true, apifyProxyGroups: ['SHADER'] }
      proxyConfig,
      // use this for this run's RequestQueue, but can be a named one, or if you
      // leave it empty, it will be placed on the remote run RQ
      targetRQ: requestQueue.queueId,
      // required sitemaps
      startUrls: [{
        url: "http://example.com/sitemap1.xml",
        userData: {
          label: "DETAILS" // userData will passthrough
        }
      }, {
        url: "http://example.com/sitemap2.xml",
      }],
      // Provide your own transform callback to filter or alter the request before adding it to the queue
      transform: ((request) => {
        if (!request.url.includes('detail')) {
          return null;
        }

        request.userData.label = request.url.includes('/item/') ? 'DETAILS' : 'CATEGORY';

        return request;
      }).toString()
    }, { waitSecs: 0 });

    run.runId = runCall.id;
    run.actorId = runCall.actId;

    await Apify.setValue('SITEMAP-CALL', run);
  }

  await Apify.utils.waitForRunToFinish(run);

  const crawler = new Apify.PuppeteerCrawler({
    requestQueue, // ready to use!
    //...
  });

  await crawler.run();
});

License

Apache 2.0

Sitemap Scraper

pvillalva/sitemap-scraper

The Sitemap Scraper extracts and outputs all URLs from a given sitemap.

Percival Villalva

Find Sitemap from url

eesti/find-sitemap-from-url

A powerful [Apify Actor] that finds sitemap URLs for any website. This Actor helps you discover XML sitemaps by checking common locations, robots.txt files, and analyzing HTML content for sitemap links.

ando

Example Sitemap Cheerio

jancurn/example-sitemap-cheerio

An example actor that first downloads a sitemap in XML format and the crawls each page from the sitemap using the fast CheerioCrawler from Apify SDK.

Jan Čurn

Sitemap Sniffer

vaclavrut/sitemap-sniffer

Sitemap sniffer will check the most used variants of sitemaps and you can use that for crawling. This will just save you time so you don't have to check manually.

Vaclav Rut

575

Internal Links Scraper

mysteriousshadow/internal-links-scraper

When given a sitemap of a website, this scraper will go through every page listed on the sitemap and find all the internal links. Useful for SEO, finding orphaned pages, and visualizing internal linking structure.

Mysterious Shadow

Get URLs from link

boring_code/get-urls-from-link

Extracts URLs from a sitemap or webpage with intuitive path matching. Use comma-separated patterns to include or exclude URL paths with smart matching: '/tags/' for exact paths, '/product' for paths starting with, or simple text for substring matches.

Audrius L.

Website Metadata Extractor (meta tags, sitemap, robots)

powerful_bachelor/website-metadata-extractor

🔍 Website Metadata Extractor 🌐 Extract essential website data: meta tags, robots.txt, and sitemap.xml in one scan. 📊 Analyze SEO elements, crawler directives, and site structure. ✅ Perfect for SEO audits, 🔎 competitor research, and 🚀 understanding how search engines view your website.

Powerful Bachelor

Spell Checker

canadesk/spell-checker

A spell checker (grammar, typos, unknown words...) that supports many languages. It's fast and costs little.

Canadesk Support

Broken Link Checker - Ensure Your Website's Integrity

dainty_screw/find-broken-links-of-your-website

Maintain your website's health and user experience with our Broken Link Checker. Easily identify and fix broken links to enhance your site's navigation, improve SEO, and keep visitors engaged.

codemaster devops

Website Spell Checker with AI

oscardz/website-spell-checker

The Website Spell Checker with AI scans your site for spelling and grammar errors, providing detailed reports with corrections. Export results in various formats or integrate into your workflow for automated checks, ensuring your content is always polished.