Metadata Scraper

Automatically scrape metadata such as title, description, heading and article from websites. It will crawl the start URLs and then scrape the metadata from the detail pages automatically navigating through the pagination.

Features

• Scrapes metadata from specified websites
• Handles pagination and detail pages
• Extracts title, description, heading, and article content
• Configurable start URLs and maximum requests per crawl
• Ignores specified URLs so no duplicates when scraping multiple times

Input

Be sure to use JSON mode for the input and not Manual mode. Here's an overview of the input parameters:

• startUrls: An array of objects containing:
  • url: The starting URL for the scrape
  • scrapeUrlGlobs: An array of URL patterns for detail pages to scrape
  • paginationUrlGlobs: An array of URL patterns for pagination pages (optional)
• maxRequestsPerCrawl: Maximum number of requests per crawl (default: 100)
• urlsToIgnore: An array of URLs to ignore when processing (optional)

Here's an example of the input data structure:

{
  "startUrls": [
    {
      "url": "https://roger-hannah.co.uk/property-search/?search_properties=1&tenure=&property_type%5B%5D=Development&property_type%5B%5D=Industrial&size_min=0&size_max=1000000",
      "scrapeUrlGlobs": ["https://roger-hannah.co.uk/properties/*"],
      "paginationUrlGlobs": []
    }
  ],
  "maxRequestsPerCrawl": 100,
  "urlsToIgnore": [
    "https://roger-hannah.co.uk/properties/development-site-with-potential-for-10-houses-planning-permission/",
    "https://roger-hannah.co.uk/properties/lower-mill-mill-street/"
  ]
}

Using Glob Patterns

Glob patterns are used to match URLs. They are similar to regular expressions but more flexible. They are used to match the URL patterns for detail pages and pagination pages.

Here are some common glob patterns used in URL matching:

1. *: Matches any number of characters (except /) Example: https://example.com/*.html matches all HTML files in the root directory
2. **: Matches any number of characters (including /) Example: https://example.com/**/*.jpg matches all JPG files in any subdirectory
3. ?: Matches exactly one character Example: https://example.com/page?.html matches page1.html, pageA.html, etc.
4. [...]: Matches any one character in the brackets Example: https://example.com/file[123].txt matches file1.txt, file2.txt, file3.txt
5. [!...]: Matches any one character not in the brackets Example: https://example.com/img[!0-9].png matches imgA.png but not img1.png
6. {...}: Matches any of the comma-separated patterns Example: https://example.com/{blog,news}/*.html matches both blog and news HTML files

Examples in the context of web scraping:

• https://example.com/products/*.html: Matches all product detail pages
• https://example.com/category/*/page-*.html: Matches pagination pages in all categories
• https://example.com/{2021,2022,2023}/**: Matches all pages from specific years
• https://example.com/page/*: Matches all pages in the root directory
• https://example.com/page/**: Matches all pages in all subdirectories

When using glob patterns in the startGlobs configuration, make sure they accurately represent the structure of the website you're scraping to ensure all relevant pages are captured.

Output

The Actor outputs the following data for each scraped property listing:

• url: The URL of the scraped page
• title: The title of a detail page
• description: The description of a detail page
• heading: The main heading of a detail page
• article: The content of a detail page

Here's an example of the output data structure:

{
  "url": "https://roger-hannah.co.uk/properties/bolton-street/",
  "title": "Bolton Street - Roger Hannah",
  "description": "Property Information The property comprises of a detached former warehouse/showroom facility constructed by way of a steel portal frame with concrete render under a pitched tiled roof. Access to the property is via personnel entrance doors fronting Bolton Street with rear loading access off Millett Street via two electrically operated roller shutter loading doors. There is a small private yard/parking/loading area to the rear of the premises. Internally, the facility provided flexible ground fl...",
  "heading": "Bolton Street",
  "article": "Property Information The property comprises of a detached former warehouse/showroom facility constructed by way of a steel portal frame with concrete render under a pitched tiled roof. Access to the property is via personnel entrance doors fronting Bolton Street with rear loading access off Millett Street via two electrically operated roller shutter loading doors. There is a small private yard/parking/loading area to the rear of the premises. Internally, the facility provided flexible ground fl..."
}