# Updates:

## v0.1 Variance-deviation
Added: Scraper now returns Standard Deviation and Variance.
## v0.2 Skewness-kurtosis
Added: Scraper now returns Skewness and Kurtosis
## v0.3 latest
- Added option scraping Lands


.actor/CHANGELOG.md

# Specify the base Docker image. You can read more about
# the available images at https://sdk.apify.com/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node:16

# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY package*.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
    && npm install --omit=dev --omit=optional \
    && echo "Installed NPM packages:" \
    && (npm list --omit=dev --all || true) \
    && echo "Node.js version:" \
    && node --version \
    && echo "NPM version:" \
    && npm --version \
    && rm -r ~/.npm

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY . ./


# Run the image.
CMD npm start --silent


.actor/Dockerfile

# Sreality statistics screaper

This actor scrape ads from [Sreality](www.sreality.cz). 
The script get all offers (according to the selected input) and calculates values such as average price, average price per m2, median price and median price per m2 for apartments. For a house, only the average price per m2 is not calculated.
Of course, it also lists the minimum and maximum price for type and the number of listings for the type and for the total group.

It's super fast a very efficient, needed only 128MB Memory.

### Updates:
v0.1 / variance-deviation - Added: Scraper now returns Standard Deviation and Variance.
v0.2 / skewness-kurtosis - Added: Scraper now returns Skewness and Kurtosis
v0.3 / latest - Added scraping Lands

------------
CZ:

Scraper pro server sreality. Script prochazi vsechny nabidky (podle zvoleneho vstupu/inputu) a vypocitava hodnoty jako prumerna cena, prumerna cena za m2, median ceny a median ceny za m2 pro byty. U domu se akorat nepocita prumerna cena za m2.
Samozrejme take vypisuje minimalni a maximalni cena a pocet inzeratu v pro dany typ a pro celkovou skupinu. 

### Aktualizace:
v0.1 / variance-deviation - Pridano: Scraper nove vraci Rozptyl a Směrodatnou odchylku
v0.2 / skewness-kurtosis - Added: Scraper nove vraci Šikmost and Špičatost
v0.3 / latest - Pridana moznost stahovani pozemku

## Cost of usage
Memory 128MB Apartments for Prague 1-10 = ~5800 offers : 4 min and $0.003

## INPUT

Choose what you want to scrape City or location(s) or both. Only input where you have to choose is if you want ads for sale or lease.

- AREA - Locality choice.
`
["Jihočeský", "Plzeňský", "Karlovarský", "Ústecký", "Liberecký", "Královéhradecký", "Pardubický", "Olomoucký", "Zlínský", "Praha", "Středočeský", "Moravskoslezský", "Vysočina", "Jihomoravský"]
`
- CITY - City of choice if you need specific locality. On web page select apartments/houses, select Locality and than you'll see all possible cities. 
`
["České Budějovice", "Plzeň", "Praha 1", "Praha 2", "Praha 8", "Praha 10", "Brno", "Ostrava"]
`
- APARTMENTS - Scrape apartments.
`
true / false
`
- HOUSES - Scrape houses.
`
true / false
`
- LANDS - Scrape lands.
`
true / false
`
- SELLING - Get statistics for Sale OR Lease.
`
true / false
`
- LANGUAGE_CZ - Get output in language cz or en. Category:" Byty/Apartments.
`
true / false
`
- DATASET_NAME - Export the entirety of the dataset to a single file in a named key-value store. 
`
Only a-z A-Z 0-9 -. Like "sreality-Houses"
`

### Example input

```
{
  "CITY": ["Praha 1", "Praha 2","Praha 3","Praha 4","Praha 5","Praha 6","Praha 7","Praha 8","Praha 9","Praha 10"],
  "APARTMENTS": true,
  "HOUSES": true,
  "LANDS": true,
  "SELLING": true,
  "LANGUAGE_CZ": true,
}
```

## OUTPUT

Actor create statistics for all types:
  - Apartments - "1+kt","1+1","2+kt","2+1","3+kt","3+1","4+kt","4+1","5+kt","5+1","6 and more","Unusual"
  - Houses - "1 room","2 rooms","3 rooms","4 rooms","5 and more  rooms","Unusual"
	- Lands - "Housings", "Meadows", "Orchard/vineyard", "Commercial", "Forests", "Garden", "Fields", "Ponds", "Others"


## Example updates:

v1.3 - Example outup for Lands - Housings
```
	"Date": "2023-05-18",
	"Area": "Jihočeský",
	"Category": "Lands",
	"Counts_All": 1632,
	"Type": "Sale",
	"Sub_Type": "Housings",
	"Count": 581,
	"min_price": 8615,
	"max_price": 49823700,
	"average_price": 4335874,
	"median_price": 2918000,
	"average_price_m2": 1463,
	"median_m2": 1300,
	"average_m2": 2964,
	"standardDeviation": 4824283,
	"variance": 23273708235068,
	"skewness": 4.299,
	"kurtosis": 29.37
```

v1.2 - Example output for House 1 room:
```
	"Date": "2023-04-20",
	"Area": "Jihočeský",
	"Counts_All": 1290,
	"Category": "Houses",
	"Type": "Sale",
	"Sub_Type": "1 room",
	"Count": 40,
	"min_price": 450000,
	"max_price": 25980000,
	"average_house_m2": 129,
	"average_land_m2": 739,
	"average_price": 4394502,
	"median_price": 2249000,
	"median_land": 373,
	"standardDeviation": 5589732,
	"variance": 31245101639048,
	"skewness": 2.657,
	"kurtosis": 9.67
```

v1.1 - Example output for Apartment 1+1:
```
	"Date": "2023-03-31",
	"Area": "Praha",
	"Category": "Apartments",
	"Counts_All": 5761,
	"Type": "Sale",
	"Sub_Type": "1+1",
	"Count": 148,
	"min_price": 2990000,
	"max_price": 9435600,
	"average_price": 5056394,
	"median_price": 4899000,
	"average_price_m2": 123327,
	"median_m2": 41,
	"average_m2": 41,
	"standardDeviation": 1147561,
	"variance": 1316895792284
```

Example output for Apartment 1kt/1+kk (cz):
```
	"Date": "2023-02-28",
	"Area": "Praha",
	"Category": "Byty",
	"Counts_All": 5837,
	"Type": "Prodej",
	"Sub_Type": "1+kk",
	"Count": 858,
	"min_price": 1600000,
	"max_price": 55000000,
	"average_price": 5087654,
	"median_price": 4978000,
	"average_price_m2": 149637,
	"median_m2": 32,
	"average_m2": 34
```

Example for Houses 1 room/jednopokojové (en):
```
	"Date": "2023-02-28",
	"Area": "Jihočeský",
	"Counts_All": 1239,
	"Category": "Houses",
	"Type": "Sale",
	"Sub_Type": "1 room",
	"Count": 32,
	"min_price": 570000,
	"max_price": 25980000,
	"average_house_m2": 135,
	"average_land_m2": 718,
	"average_price": 3469071,
	"median_price": 2374500,
	"median_land": 403
```


.actor/README.md

Sreality statistics

Sreality

.actor/actor.json

If checked then actor will scrape apartments offers.

Apartments

Enter areas what to scrape: Jihočeský, Plzeňský, Karlovarský, Ústecký, Liberecký, Královéhradecký, Pardubický, Olomoucký, Zlínský, Praha, Středočeský, Moravskoslezský, Vysočina,Jihomoravský

Scrape area(s)

Enter city name: České Budějovice, Brno, Plzeň, Praha 1, Praha 2, Praha 10

Scrape city(s)

All data will be saved to specific name. Only a-z A-Z 0-9 -

Set dataset name

If checked then actor will scrape houses offers.

Houses

If checked then actor will scrape lands offers.

Lands

Language CZ

Choose one: Lease(Pronajem) / Sale(Prodej)

Lease(Pronajem) / Sale(Prodej)

Sreality statistics scraper

.actor/input-schema.json

src/INPUT_EXAMPLE.json

export const AREA_NAMES = ["Jihočeský", "Plzeňský", "Karlovarský", "Ústecký", "Liberecký", "Královéhradecký", "Pardubický", "Olomoucký", "Zlínský", "Praha", "Středočeský", "Moravskoslezský", "Vysočina", "Jihomoravský"];
export const BASE_URL = `https://www.sreality.cz/api/cs/v2/estates`;
export const BASE_COUNT_URL = `https://www.sreality.cz/api/cs/v2/estates/count`;
export const URL_SELL_EN = 'https://www.sreality.cz/en/detail/sale/';
export const URL_SELL_CZ = 'https://www.sreality.cz/detail/prodej/';
export const URL_LEASE_EN = 'https://www.sreality.cz/en/detail/lease/';
export const URL_LEASE_CZ = 'https://www.sreality.cz/detail/pronajem/';
export const URL_ADRESAR = 'https://www.sreality.cz/adresar/';
export const CATEGORY_MAIN = ["1", "2", "3"];
export const CATEGORY_MAIN_NAME = ["Byty", "Domy", "Pozemky"];
export const CATEGORY_MAIN_NAME_EN = ["Apartments", "Houses", "Lands"];
export const CATEGORY_TYPE = ["1", "2"];
export const CATEGORY_TYPE_NAME = ["Prodej", "Pronájem"];
export const CATEGORY_TYPE_NAME_EN = ["Sale", "Lease"];
export const CATEGORY_SUB_A_ALL = ["2","3","4","5","6","7","8","9","10","11","12","16"];
export const CATEGORY_SUB_A_NAMES = ["1+kk","1+1","2+kk","2+1","3+kk","3+1","4+kk","4+1","5+kk","5+1","6 pokojů a více","atypické"];
export const CATEGORY_SUB_A_NAMES_EN = ["1+kt","1+1","2+kt","2+1","3+kt","3+1","4+kt","4+1","5+kt","5+1","6 and more","Unusual"];
export const CATEGORY_SUB_B_ALL = ["100","200","300","400","500","600"];
export const CATEGORY_SUB_B_NAMES = ["jednopokojové","dvoupokojové","třípokojové","čtyřpokojové","pětipokojové a větší","atypické"];
export const CATEGORY_SUB_B_NAMES_EN = ["1 room","2 rooms","3 rooms","4 rooms","5 and more  rooms","Unusual"];
export const CATEGORY_SUB_C_ALL = ["19","22","48","18","21","23","20","46","24"];
export const CATEGORY_SUB_C_NAMES = ["Bydlení","Louky","Sady/vinice","Komerční","Lesy","Zahrady", "Pole", "Rybníky", "Ostatní"];
export const CATEGORY_SUB_C_NAMES_EN = ["Housings", "Meadows", "Orchard/vineyard", "Commercial", "Forests", "Garden", "Fields", "Ponds", "Others"];
export const MAX_PER_PAGE = "999";
export let SORT = "0";
export let PAGE = "1";
export let PER_PAGE = "10";

export const AREA = [
  {
    "name": "Jihočeský",
    "id": 1,
    "cities": [
      {"city": "České Budějovice", "id": 1},
      {"city": "Český Krumlov", "id": 2},
      {"city": "Jindřichův Hradec", "id": 3},
      {"city": "Písek", "id": 4},
      {"city": "Prachatice", "id": 5},
      {"city": "Strakonice", "id": 6},
      {"city": "Tábor", "id": 7},
    ]
  },
  {
    "name": "Plzeňský",
    "id": 2,
    "cities": [
      {"city": "Domažlice", "id": 8},
      {"city": "Klatovy", "id": 11},
      {"city": "Plzeň-jih", "id": 13},
      {"city": "Plzeň-město", "id": 12},
      {"city": "Plzeň-sever", "id": 14},
      {"city": "Rokycany", "id": 15},
      {"city": "Tachov", "id": 17},
    ]
  },
  {
    "name": "Karlovarský",
    "id": 3,
    "cities": [
      {"city": "Cheb", "id": 9},
      {"city": "Karlovy Vary", "id": 10},
      {"city": "Sokolov", "id": 16},
    ]
  },
  {
    "name": "Ústecký",
    "id": 4,
    "cities": [
      {"city": "Chomutov", "id": 20},
      {"city": "Děčín", "id": 19},
      {"city": "Litoměřice", "id": 23},
      {"city": "Louny", "id": 24},
      {"city": "Most", "id": 25},
      {"city": "Teplice", "id": 26},
      {"city": "Ústí nad Labem", "id": 27},
    ]
  },
  {
    "name": "Liberecký",
    "id": 5,
    "cities": [
      {"city": "Česká Lípa", "id": 18},
      {"city": "Jablonec nad Nisou", "id": 21},
      {"city": "Liberec", "id": 22},
      {"city": "Semily", "id": 34},
    ]
  },
  {
    "name": "Královéhradecký",
    "id": 6,
    "cities": [
      {"city": "Hradec Králové", "id": 28},
      {"city": "Jičín", "id": 30},
      {"city": "Náchod", "id": 31},
      {"city": "Rychnov nad Kněžnou", "id": 33},
      {"city": "Trutnov", "id": 36},
    ]
  },
  {
    "name": "Pardubický",
    "id": 7,
    "cities": [
      {"city": "Chrudim", "id": 29},
      {"city": "Pardubice", "id": 32},
      {"city": "Svitavy", "id": 35},
      {"city": "Ústí nad Orlicí", "id": 37},

    ]
  },
  {
    "name": "Olomoucký",
    "id": 8,
    "cities": [
      {"city": "Jeseník", "id": 46},
      {"city": "Olomouc", "id": 42},
      {"city": "Přerov", "id": 43},
      {"city": "Prostějov", "id": 40},
      {"city": "Šumperk", "id": 44},
    ]
  },
  {
    "name": "Zlínský",
    "id": 9,
    "cities": [
      {"city": "Kroměříž", "id": 39},
      {"city": "Uherské Hradiště", "id": 41},
      {"city": "Vsetín", "id": 45},
      {"city": "Zlín", "id": 38},
    ]
  },
  {
    "name": "Praha",
    "id": 10,
    "cities": [
      {"city": "Praha 1", "id": 5001},
      {"city": "Praha 2", "id": 5002},
      {"city": "Praha 3", "id": 5003},
      {"city": "Praha 4", "id": 5004},
      {"city": "Praha 5", "id": 5005},
      {"city": "Praha 6", "id": 5006},
      {"city": "Praha 7", "id": 5007},
      {"city": "Praha 8", "id": 5008},
      {"city": "Praha 9", "id": 5009},
      {"city": "Praha 10", "id": 5010},
    ]
  },
  {
    "name": "Středočeský",
    "id": 11,
    "cities": [
      {"city": "Benešov", "id": 48},
      {"city": "Beroun", "id": 49},
      {"city": "Kladno", "id": 50},
      {"city": "Kolín", "id": 51},
      {"city": "Kutná Hora", "id": 52},
      {"city": "Mělník", "id": 54},
      {"city": "Mladá Boleslav", "id": 53},
      {"city": "Nymburk", "id": 55},
      {"city": "Praha-východ", "id": 56},
      {"city": "Praha-západ", "id": 57},
      {"city": "Příbram", "id": 58},
      {"city": "Rakovník", "id": 59},
    ]
  },
  {
    "name": "Moravskoslezský",
    "id": 12,
    "cities": [
      {"city": "Bruntál", "id": 60},
      {"city": "Frýdek-Místek", "id": 61},
      {"city": "Karviná", "id": 62},
      {"city": "Nový Jičín", "id": 63},
      {"city": "Opava", "id": 64},
      {"city": "Ostrava-město", "id": 65},
    ]
  },
  {
    "name": "Vysočina",
    "id": 13,
    "cities": [
      {"city": "Havlíčkův Brod", "id": 66},
      {"city": "Jihlava", "id": 67},
      {"city": "Pelhřimov", "id": 68},
      {"city": "Třebíč", "id": 69},
      {"city": "Žďár nad Sázavou", "id": 70},

    ]
  },
  {
    "name": "Jihomoravský",
    "id": 14,
    "cities": [
      {"city": "Blansko", "id": 71},
      {"city": "Břeclav", "id": 74},
      {"city": "Brno-město", "id": 72},
      {"city": "Brno-venkov", "id": 73},
      {"city": "Hodonín", "id": 75},
      {"city": "Vyškov", "id": 76},
      {"city": "Znojmo", "id": 77},
    ]
  }
];


src/constants.js

import { Actor, Dataset, log } from 'apify';
import * as fnc from "./routes.js";

await Actor.init();
const { AREA, CITY, APARTMENTS, HOUSES, LANDS, SELLING, LANGUAGE_CZ, DATASET_NAME, DEBUG } = await Actor.getInput();

(AREA) ? console.log('AREA: ', AREA) : '';
(CITY) ? console.log('CITY: ', CITY) : '';

if (DEBUG) {
    log.setLevel(log.LEVELS.DEBUG);
    log.debug('DEBUGGING');
    log.debug('APARTMENTS', {APARTMENTS});
    log.debug('HOUSES', {HOUSES});
    log.debug('SELLING', {SELLING});
    log.debug('Dataset name', {DATASET_NAME});
}

if (AREA && AREA.length > 0) {
    await fnc.handleRequestArea(AREA, APARTMENTS, HOUSES, LANDS, SELLING, LANGUAGE_CZ);
}

if (CITY && CITY.length > 0) {
    await fnc.handleRequestCity(CITY, APARTMENTS, HOUSES, LANDS, SELLING, LANGUAGE_CZ);
}

await Actor.exit();


src/main.js

import { Actor, Dataset, log } from 'apify';
import axios from 'axios';
import { expect } from 'chai';
import { SORT, PAGE, PER_PAGE, BASE_URL, AREA_NAMES, MAX_PER_PAGE, BASE_COUNT_URL } from './constants.js';

await Actor.init();

export async function returnLocalityID(AREA) {
  let ID;
  for (let i = 0; i < AREA_NAMES.length; i++) {
    if (AREA === AREA_NAMES[i]) {
      ID = i+1;
    }
  }
  return ID;
};

export async function returnAllCounts(CATEGORY, SELL, AREA_ID, CITY_ID) {
  let SELLING = 0;
  (SELL == 0) ? SELLING = 1 : SELLING = 2;
  let url_part = BASE_COUNT_URL;
  url_part += `?category_main_cb=${CATEGORY}`;
  url_part += `&category_type_cb=${SELLING}&locality_country_id=112`;
  url_part += `&locality_region_id=${AREA_ID}`;
  if (CITY_ID) {
    url_part += `&locality_district_id=${CITY_ID}`;
  }
  if (SELLING == 1) {
    url_part += `&no_auction=1&no_shares=1`;
  }
  let countCity = await axios.get(url_part);
  expect(countCity.status).equal(200, 'Status code');
  let counts_all = countCity.data.result_size;

  return counts_all;
};

export async function estateCountStatistics(estateURL, category, counts, AREA_NAME, CATEGORY_NAME) {
  let price = [], m2 = [], exceeded = 1, house = [], land = [], maxPrice = 0, minPrice = 999999999;
  log.info(`=> ${category}`);
  if (counts > 999) {
    exceeded = Math.ceil(counts / 999);
    log.debug(`Exceeded count value ${counts} #${exceeded}`);
  }
  for (let x = 0; x < exceeded; x++) {
    let RESPONSE = await axios.get(`${estateURL}&page=${x+1}`);
    expect(RESPONSE.status).equal(200, 'Status code');
    for (let i = 0; i < RESPONSE.data._embedded.estates.length; i++) {
      if (RESPONSE.data._embedded.estates[i].region_tip === 0 && RESPONSE.data._embedded.estates[i].price > 1000) {
        if (estateURL.includes('category_sub_cb') === true) {
          let metry = RESPONSE.data._embedded.estates[i].name.split(`${category}`);
          log.debug('',{'Title': RESPONSE.data._embedded.estates[i].name});
          log.debug('',{'Price': RESPONSE.data._embedded.estates[i].price});
          log.debug('',{'m2': Number(metry[1].replace(/\D/g, ""))});
          price.push(RESPONSE.data._embedded.estates[i].price);
          m2.push(Number(metry[1].replace(/\D/g, "")));

          if (RESPONSE.data._embedded.estates[i].price > maxPrice) {
            maxPrice = RESPONSE.data._embedded.estates[i].price;
          }
          if (RESPONSE.data._embedded.estates[i].price < minPrice) {
            minPrice = RESPONSE.data._embedded.estates[i].price;
          }
        } else if (estateURL.includes('category_sub_cb') === false) {
          log.debug('',{'Title': RESPONSE.data._embedded.estates[i].name});
          let parser = RESPONSE.data._embedded.estates[i].name.split(`m²,`);
          log.debug('',{'Dum': Number(parser[0].replace(/\D/g, ""))});
          log.debug('',{'Pozemek': Number(parser[1].replace(/\D/g, ""))});
          log.debug('',{'Price': RESPONSE.data._embedded.estates[i].price});
          house.push(Number(parser[0].replace(/\D/g, "")));
          land.push(Number(parser[1].replace(/\D/g, "")));
          price.push(RESPONSE.data._embedded.estates[i].price);

          if (RESPONSE.data._embedded.estates[i].price > maxPrice) {
            maxPrice = RESPONSE.data._embedded.estates[i].price;
          }
          if (RESPONSE.data._embedded.estates[i].price < minPrice) {
            minPrice = RESPONSE.data._embedded.estates[i].price;
          }
        }
      }
    }
  }

  if (estateURL.includes('category_sub_cb') === true) {
    const average_price = Math.round(price.reduce((a, b) => a + b, 0) / price.length);
    const average_m2 = Math.round(m2.reduce((a, b) => a + b, 0) / m2.length);
    const average_price_m2 = Math.round(average_price / average_m2);
    const median_price = await median(price);
    const median_m2 = await median(m2);
    const median_price_m2 = Math.round(median_price / median_m2);
    const variance = await calculateVariance(average_price, price);
    const standardDeviation = await calculateSD(variance);
    const skewness = await skewnessCalculate(price);
    const kurtosis = await kurtosisCalculate(price);

    log.debug('----------------------------------');
    log.debug('',{'Average price': average_price});
    log.debug('',{'Average m2': average_m2});
    log.debug('',{'Average price/m2': average_price_m2});
    log.debug('',{'Median price': median_price});
    log.debug('',{'Median m2': median_m2});
    log.debug('',{'Median price/m2': median_price_m2});
    log.debug('',{'Min price': minPrice});
    log.debug('',{'Max price': maxPrice});
    log.debug('',{'Standard Deviation': standardDeviation});
    log.debug('',{'Variance': variance});
    log.debug('',{'Prices': price});
    log.debug('',{'Skewness': skewness});
    log.debug('',{'Kurtosis': kurtosis});
    log.debug('----------------------------------');

    return [average_price, average_m2, average_price_m2, median_price_m2, median_price, median_m2, minPrice, maxPrice, standardDeviation, variance, skewness, kurtosis];

  } else if (estateURL.includes('category_sub_cb') === false) {
    const average_house = Math.round(house.reduce((a, b) => a + b, 0) / house.length);
    const average_land = Math.round(land.reduce((a, b) => a + b, 0) / land.length);
    const average_price = Math.round(price.reduce((a, b) => a + b, 0) / price.length);
    const median_price = await median(price);
    const median_land = await median(land);
    const variance = await calculateVariance(average_price, price);
    const standardDeviation = await calculateSD(variance);
    const skewness = await skewnessCalculate(price);
    const kurtosis = await kurtosisCalculate(price);

    log.debug('----------------------------------');
    log.debug('',{'Average house': average_house});
    log.debug('',{'Average land': average_land});
    log.debug('',{'Average price': average_price});
    log.debug('',{'Median price': median_price});
    log.debug('',{'Median land': median_land});
    log.debug('',{'Min price': minPrice});
    log.debug('',{'Max price': maxPrice});
    log.debug('',{'Standard Deviation': standardDeviation});
    log.debug('',{'Variance': variance});
    log.debug('',{'Prices': price});
    log.debug('',{'Skewness': skewness});
    log.debug('',{'Kurtosis': kurtosis});
    log.debug('----------------------------------');

    return [average_house, average_land, average_price, median_price, median_land, minPrice, maxPrice, standardDeviation, variance, skewness, kurtosis];
  }
};

export async function landsCountStatistics(estateURL, category, counts, AREA_NAME, CATEGORY_NAME) {
  let price = [], m2 = [], exceeded = 1, house = [], land = [], metry, maxPrice = 0, minPrice = 999999999;
  log.info(`=> ${category}`);
  if (counts > 999) {
    exceeded = Math.ceil(counts / 999);
    log.debug(`Exceeded count value ${counts} #${exceeded}`);
  }
  for (let x = 0; x < exceeded; x++) {
    let RESPONSE = await axios.get(`${estateURL}&page=${x+1}`);
    expect(RESPONSE.status).equal(200, 'Status code');
    for (let i = 0; i < RESPONSE.data._embedded.estates.length; i++) {
      if (RESPONSE.data._embedded.estates[i].region_tip === 0 && RESPONSE.data._embedded.estates[i].price > 1000) {
        if (estateURL.includes('category_sub_cb') === true) {
          metry = RESPONSE.data._embedded.estates[i].name;
          log.debug('',{'Title': RESPONSE.data._embedded.estates[i].name});
          log.debug('',{'Price': RESPONSE.data._embedded.estates[i].price});
          log.debug('',{'m2': Number(metry.replace(/\D/g, ""))});
          price.push(RESPONSE.data._embedded.estates[i].price);
          m2.push(Number(metry.replace(/\D/g, "")));
        }

        if (RESPONSE.data._embedded.estates[i].price > maxPrice) {
          maxPrice = RESPONSE.data._embedded.estates[i].price;
        }
        if (RESPONSE.data._embedded.estates[i].price < minPrice) {
          minPrice = RESPONSE.data._embedded.estates[i].price;
        }
      }
    }
  }

  if (estateURL.includes('category_sub_cb') === true) {
    const average_price = Math.round(price.reduce((a, b) => a + b, 0) / price.length);
    const average_m2 = Math.round(m2.reduce((a, b) => a + b, 0) / m2.length);
    const average_price_m2 = Math.round(average_price / average_m2);
    const median_price = await median(price);
    const median_m2 = await median(m2);
    const variance = await calculateVariance(average_price, price);
    const standardDeviation = await calculateSD(variance);
    const skewness = await skewnessCalculate(price);
    const kurtosis = await kurtosisCalculate(price);

    log.debug('----------------------------------');
    log.debug('',{'Average price': average_price});
    log.debug('',{'Average m2': average_m2});
    log.debug('',{'Average price/m2': average_price_m2});
    log.debug('',{'Median price/m2': median_price});
    log.debug('',{'Median m2': median_m2});
    log.debug('',{'Min price': minPrice});
    log.debug('',{'Max price': maxPrice});
    log.debug('',{'Standard Deviation': standardDeviation});
    log.debug('',{'Variance': variance});
    log.debug('',{'Prices': price});
    log.debug('',{'Skewness': skewness});
    log.debug('',{'Kurtosis': kurtosis});
    log.debug('----------------------------------');

    return [average_price, average_m2, average_price_m2, median_price, median_m2, minPrice, maxPrice, standardDeviation, variance, skewness, kurtosis];
  } else {
    log.info('URL not included: category_sub_cb');
  }
};

export async function generateEstateURL(Category, Type, Area, CATEGORY_SUB, district) {
  let url_part = BASE_URL;
  url_part += `?category_main_cb=${Category}`;
  url_part += `&category_type_cb=${Type}`;
  url_part += `&locality_region_id=${Area}`;
  if (district) {
    url_part += `&locality_district_id=${district}`;
  }
  if (CATEGORY_SUB.length <= 2) {
    url_part += `&category_sub_cb=${CATEGORY_SUB}`;
  } else if (CATEGORY_SUB.length === 3) {
    url_part += `&room_count_cb=${CATEGORY_SUB.slice(0,1)}`;
  }
  url_part += `&no_auction=1`;
  url_part += `&per_page=${MAX_PER_PAGE}`;
  url_part += `&tms=${await getTMS()}`;
  url_part += `&no_auction=1&no_shares=1`;
  log.debug('URL estates:', {url_part});
  return url_part;
};

export async function regerateURL(MAIN, TYPE, ID, SUB_IN) {
  let SUB, URL;
  let TMS = await getTMS();
  if (SUB_IN === undefined) {
    SUB = '';
  } else if (SUB_IN.length < 3) {
    SUB = `&category_sub_cb=${SUB_IN}`
  } else if (SUB_IN.length === 3) {
    SUB = `&room_count_cb=${SUB_IN.slice(0,1)}`;
  }
  URL = `${BASE_URL}?category_main_cb=${MAIN}${SUB}&category_type_cb=${TYPE}&locality_region_id=${ID}&page=${PAGE}&no_auction=1&per_page=${PER_PAGE}&sort=${SORT}&tms=${TMS}&no_auction=1&no_shares=1`;
  return URL;
};

async function getTMS() {
  let min = Math.ceil(1111111111111);
  let max = Math.floor(9111111111111);
  return Math.floor(Math.random() * (max - min) + min);
}

async function median(arr) {
  const mid = Math.floor(arr.length / 2),
  nums = [...arr].sort((a, b) => a - b);
  return arr.length % 2 !== 0 ? nums[mid] : (nums[mid - 1] + nums[mid]) / 2;
};

async function label_All_exist(labelsAll, label) {
  if (labelsAll.indexOf(label) > -1) {
    return true;
  } else {
    return false;
  }
};

async function calculateMean(values) {
  const mean = (values.reduce((sum, current) => sum + current)) / values.length;
  return mean;
};

async function calculateVariance(average, values) {
  if (values.length > 0) {
    const squareDiffs = values.map((value) => {
      const diff = value - average;
      return diff * diff;
    });
    const variance = await calculateMean(squareDiffs);
      return Math.round(variance);
  } else {
    return null
  }
};

async function calculateSD(variance) {
  if (variance > 0) { 
  return Math.round(Math.sqrt(variance));
  } else {
    return null
  }
};

async function skewnessCalculate(data) {
  const mean = data.reduce((acc, val) => acc + val, 0) / data.length;
  const standardDeviation = Math.sqrt(data.reduce((acc, val) => acc + Math.pow(val - mean, 2), 0) / data.length);
  const thirdMoment = data.reduce((acc, val) => acc + Math.pow(val - mean, 3), 0) / data.length;
  const skewness = thirdMoment / Math.pow(standardDeviation, 3);

  return Number(skewness.toFixed(3));
}

async function kurtosisCalculate(data) {
  let deviation = [], deviation2 = [], deviation4 = [];
  let secondMoment = 0, fourthMoment = 0;

  const n = data.length;
  const mean = data.reduce((acc, val) => acc + val, 0) / n;
  for (const value of data) {
      deviation.push(value - mean);
      deviation2.push((value - mean) ** 2);
      deviation4.push((value - mean) ** 4);
  }

  for (const value of deviation2) {
      secondMoment += value;
  }
  secondMoment = (secondMoment/n).toFixed(2);

  for (const value of deviation4) {
      fourthMoment += value;
  }
  fourthMoment = (fourthMoment / n).toFixed(2);

  const kurt = (fourthMoment / (secondMoment ** 2)).toFixed(2);

  return Number(kurt);
}


src/processing.js

src/routes.js

# configurations
.idea

# crawlee and apify storage folders
apify_storage
crawlee_storage
storage

# installed files
node_modules

# git folder
.git


.dockerignore

.gitignore

sreality-stats

package.json

Super fast and very efficient sreality.cz statistics actor.
Get all offers for city or location and it will calculates values such as average price, average price per m2, median price and median price per m2.
Only for Czech Republic.

sreality-statistics

Sreality statistics · Apify

Input · Sreality statistics · Apify

API · Sreality statistics · Apify

Changelog · Sreality statistics · Apify

Related Actors · Sreality statistics · Apify

Super fast and very efficient sreality.cz statistics actor.
Get all offers for city or location and it will calculates values such as average price, average price per m2, median price and median pr...

Where next?