Oikotie.fi scraper

Application Settings

General settings for the application behavior:

• IS_DEBUGGING: Set to True for debugging mode, which provides more verbose output helpful in development. Default is False.
• WORKER_NUM: The number of worker processes for the application. Example: 5
• IS_UPDATE_OLD_AI_RESULT: Controls whether to update old AI results. Default is True.
• FIELDS_TO_CLASSIFY_STRING: Specifies the fields in the input data that need to be classified. Example: "upcomingRenovations,doneRenovation"

Data Flow

The application processes data in the following order:

1. Empty oikotie-deals key-value store created on Apify.
2. Prepare api_headers to access the Oikotie API. Each thread has its own api_headers.
3. Prepare deal generator to scrape with set of filters. (Extracted from the Oikotie API)
4. Extract basic information from each deal.
5. Apply multithreading to extract more detailed information from each deal for each batch of deals (24 deals per page).
6. Classify the extracted data. Only the fields specified in the FIELDS_TO_CLASSIFY_STRING setting are included in the classification. Current input data format: Hardcoded in the classify_deal_by_anthropic function. ''' Past renovations in Finnish language is here: {doneRenovations} Future planned renovations in Finnish language is here: {upcomingRenovations} '''
7. Save the classified data with deal detail to the oikotie-deals key-value store. Current output data format: { "cardId": 21226538, "updatedAt": "2024-05-08T15:15:52.848291", "url": "https://asunnot.oikotie.fi/myytavat-asunnot/lahti/21226538", "roomConfiguration": "1h+k", "buildYear": 1968, "price": 29900.0, "size": 42.0, "description": "42 m2 keittiöllinen yksiö Lahden Nastolan Rakokiven palveluiden läheltä", "rooms": 1, "city": "Lahti", "country": "Suomi", "postalCode": "15550", "upcomingRenovations": "V. KPH silikonien tarkistus ja vaihto, radon mittaus, palovaroittimien uusinta, v. -25 IV-kanavien nuohous, v. -28 Mahdollinen rappukäyttävien lattiapintojen uusiminen ja seinien maalaus.", "doneRenovations": "V. -02 julkisivuremontti ja käyttövesiputket, v. -04 lämmönvaihdin ja kaapeli tv, v. -07 kylmäkellarin automatiikka, v. -08 jäteastiasto ja saunaremontti, v. -11 parkkitolppien ja kellojen uusiminen+lisätolppa sekä saunan lauteiden ja seinien korjaus, v. -16 lukituksen uusiminen, v. -19 rappukäytävien, autotallien ja pyörävarastojen ovien uusiminen, v. -23 vesikaton maalaus ja viemärisukitus", "conditionType": "Tyydyttävä", "housingType": "Omistus", "landOwnership": "Oma", "debtFreePriceText": "29 900 €", "sellingPriceText": "25 491,39 €", "pricePerSquareMeterText": "711,90 € / m2", "debtShareText": "4 408,61 €", "treatmentFeeText": "168 € / kk", "capitalConsidersationText": "63 € / kk", "totalCompanyConsidersationText": "231 € / kk", "waterFeeText": null, "otherCostsText": "Taloyhtiön hallitukselle on annettu valtuutus periä (tai olla perimättä) ylimääräisiä vastikkeita yhtiön taloudellisen aseman tai maksuvalmiuden sitä vaatiessa\n\nLisätietoja maksuista: Ovilainan rahoitusvastike 35,70 €/kk (yksi erä jäljellä), Kylmävesi 5 €/m3, lämminvesi 9 €/m3, autotallivuokra 25 €/kk. Vapaat vuorot ja paikat varmistettava isännöinnistä.", "anthropic_model": "claude-3-haiku-20240307", "anthropic_system_prompt": "Following text contains list of completed (done) renovations and planned future (upcoming) renovations of a department flat. Please evaluate the list and make conclusions if the Pipes are in original state or renovated. Output only one of the two options: \"1. Pipes are renovated\" or \"2. Pipes are in original condition\" or \"3. Pipe condition is unclear or they are partially renovated\". Texts are in Finnish language. Do not return any explanations, only the conclusion from list of three options. For example if the upcoming renovations contains text \"Linjasaneeraus alkanee alustavan arvion mukaan loppuvuonna\" then correct answer is \"2. Pipes are in original condition\"", "anthropic_classification_result": null, "anthropic_classification_at": "2024-05-08 15:15:52.848291" }