Web Page Analyzer

Web page analyzer performs an analysis on the input website. It will search the content of the website for each keyword and output all the possible ways a keyword data can be scraped.

Intended users

This actor has been developed mainly for two categories of the users:

1. Analysts and non-deverlopers that would like to get an insight on how the web page can be scraped. The output of the analysis can give a great clue of how difficult it is to scraped the website or how many resources will be needed.
2. Developers that are developing web scrapers. Data from the analysis can be directly used for scraping purposes.

Website url

Url of a website to be analyzed.

Keywords

Keywords are strings that analyzer will search for in given website.

Input

Input can be set using JSON or the visual input UI through Apify console.

{
    // url of a  website to be analyzed
    "url": "http://example.com",
    // array of strings too look for on the website, we will refer to those string as "keywords"
    "keywords": [
        "About us",
        // numbers are also passed as strings
        "125"
        ],
}

How to use

1. Add the analyzer actor to your Apify console.
2. Enter url of a website to be analysed.
3. Add keywords to be searched for. It is recommended to directly copy the text from the website.
4. Run the actor.
5. View the analysis results by opening DASHBOARD.html file inside the key-value storage.

Output

Output of this actor is saved in Apify key-value store of the particular actor run.

Results of the analysis are saved in the OUTPUT.json file and can be viewed by opening the DASHBOARD.html file.

How to run analzyer locally on your computer

1. Download this repository.
2. Run npm install
3. Download analyzer-ui repository https://github.com/vahlunten/analyzer-ui
4. Run npm install 5 Run npm run dev
5. Run npm run start in analyzer-ts repository. It will run with ./apify_storage/key_value_stores/default/INPUT.json file as an input.
6. Open the web app started by analyzer-ui

Files stored in key-value store

Actor also saves some additional files with futher information, useful mainly for developers.

1. OUTPUT.json: Most of the analysis results are stored in this file.
2. DASHBOARD.html. Visual analysis results explorer.
3. initial.html. Initial response retrieved by the chromium browser.
4. dom-content.html. Htlm of a website rendered inside chromium browser.
5. initial-cheerio.html. Initial response retrieved by the CheerioCrawler.
6. INPUT. Actors input.
7. screenshot.jpeg. Screenshot of a loaded website.
8. xhr.json. Additional details about XHR validation.

How analyzer works

The goal of this actor is to find out the optimal way to scrape the website. It tries to find a way to scrape a website without using a browser to minimize the resources needed for scraping.

Analyzer uses a Playwright library to control the chromium browser. It navigates to the website and scans all the sources for the input keywords.

The analysis steps:

1. Analyzer opens the chromium browser and navigates to the website.
2. It searches the initial response of a website and fully rendered DOM.
3. XHR requests are intercepted and searched for the keywords.
4. These search results are then validated against the initial response retrieved by CheerioCrawler.
5. XHR request containing the keywords are then replicated using got-scraping library with different sets of headers.

Planned features

• flexible formatting of numbers & strings (4,400 <=> 4000)
• testing different proxy configurations (datacenter -> residential)
• generation of scraper/crawler code
• custom clicks