Sitemap-Based Web Scraper

This tool crawls every page listed on a website's sitemap and retrieves all internal links from each page. It’s ideal for SEO analysis, identifying orphaned pages, and visualizing the internal linking structure of a site.

Features

• Crawl Entire Website: Starts with a sitemap and navigates through each page listed for thorough coverage.
• Internal Link Extraction: Finds and catalogs all internal links on each page.
• SEO Insights: Helps identify orphaned pages and offers data for improving internal link structure.
• Visualization Support: Output can be used to visualize how different pages are connected within the site.

Usage

1. Provide Sitemap URL: Start by giving the scraper the URL of the sitemap (XML format).
2. Run the Scraper: The scraper will visit each URL in the sitemap and collect internal links.
3. Data Analysis: Use the output to identify orphaned pages and assess the site's internal linking structure.

Note: For larger websites, more RAM, CPU, and time may be needed to handle the extensive data collection.

Output

The scraper outputs a structured list or graph of internal links on the website, which can be used for:

• Identifying Orphaned Pages: Pages listed in the sitemap but not linked from other pages.
• Improving SEO: Helps identify pages with weak internal linking, which can be improved to enhance SEO.
• Link Structure Visualization: Upon further custom processing, data can be visualized to show how different pages connect within the site.

Sample Output

1[
2  {
3    "https://pliwriters.com": [
4      "https://pliwriters.com/blog",
5      "https://pliwriters.com/about",
6      "https://pliwriters.com/contact",
7      "https://pliwriters.com/contact",
8      "https://pliwriters.com/contact",
9      "https://pliwriters.com/about",
10      "https://pliwriters.com/blog",
11      "https://pliwriters.com/contact",
12      "https://pliwriters.com/privacy-policy",
13      "https://pliwriters.com/terms-and-conditions"
14    ],
15    "https://pliwriters.com/blog": [
16      "https://pliwriters.com",
17      "https://pliwriters.com",
18      "https://pliwriters.com/about",
19      "https://pliwriters.com/contact",
20      "https://pliwriters.com/blog/how-to-find-a-sitemap-on-any-website",
21      "https://pliwriters.com/blog/how-to-find-a-sitemap-on-any-website",
22      "https://pliwriters.com/blog/3-key-components-of-seo",
23      "https://pliwriters.com/blog/3-key-components-of-seo",
24      "https://pliwriters.com/about",
25      "https://pliwriters.com/contact",
26      "https://pliwriters.com/privacy-policy",
27      "https://pliwriters.com/terms-and-conditions"
28    ],
29    "https://pliwriters.com/internal-link-visualization-beta": [
30      "https://pliwriters.com",
31      "https://pliwriters.com/blog",
32      "https://pliwriters.com/about",
33      "https://pliwriters.com/contact",
34      "https://pliwriters.com/blog/how-to-find-a-sitemap-on-any-website",
35      "https://pliwriters.com/wp-content/uploads/2024/02/public_index.html",
36      "https://pliwriters.com/contact",
37      "https://pliwriters.com/about",
38      "https://pliwriters.com/blog",
39      "https://pliwriters.com/contact",
40      "https://pliwriters.com/privacy-policy",
41      "https://pliwriters.com/terms-and-conditions"
42    ],
43    "https://pliwriters.com/about": [
44      "https://pliwriters.com",
45      "https://pliwriters.com/blog",
46      "https://pliwriters.com/contact",
47      "https://pliwriters.com/privacy-policy",
48      "https://pliwriters.com/terms-and-conditions"
49    ],
50    "https://pliwriters.com/contact": [
51      "https://pliwriters.com",
52      "https://pliwriters.com/blog",
53      "https://pliwriters.com/about",
54      "https://pliwriters.com/privacy-policy",
55      "https://pliwriters.com/terms-and-conditions"
56    ],
57    "https://pliwriters.com/blog/how-to-find-a-sitemap-on-any-website": [
58      "https://pliwriters.com",
59      "https://pliwriters.com/blog",
60      "https://pliwriters.com/about",
61      "https://seomator.com/sitemap-finder",
62      "https://pliwriters.com/blog/how-to-find-a-sitemap-on-any-website/#respond",
63      "https://pliwriters.com/privacy-policy",
64      "https://pliwriters.com/terms-and-conditions"
65    ]
66  }
67]