.actor

# Specify the base Docker image. You can read more about
# the available images at https://crawlee.dev/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node-puppeteer-chrome:16

# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY --chown=myuser package*.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
    && npm install --omit=dev --omit=optional \
    && echo "Installed NPM packages:" \
    && (npm list --omit=dev --all || true) \
    && echo "Node.js version:" \
    && node --version \
    && echo "NPM version:" \
    && npm --version \
    && rm -r ~/.npm

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY --chown=myuser . ./


# Run the image. If you know you won't need headful browsers,
# you can remove the XVFB start script for a micro perf gain.
CMD ./start_xvfb_and_run_cmd.sh && npm start --silent


.actor/Dockerfile

scrapper to replace webscrapper which does not work any longer with leboncoin and datadome

JavaScript function executed on every page (inside the browser). Customise it if necessary. <br><br>For details, see <a href='https://playwright.dev/docs/api/class-page' target='_blank' rel='noopener'>See Playwright page, available as 'context.page'</a> 

Function

Leboncoin may block you, so you better use proxies.<br><br>For details, see <a href='https://apify.com/apify/web-scraper#proxy-configuration' target='_blank' rel='noopener'>Proxy configuration</a>.

Avoid anti-bot detection with Proxy

Search URLs

Leboncoin scrapper

.actor/INPUT_SCHEMA.json

# Leboncoin

This actor aims to **extract information from leboncoin.fr**. Data can be exported to various formats such as JSON or CSV. You have to option to **customise the information you want to extract** by modifying the javascript function which is executed on each page.

## How to Start

### 1 - Setup
- On your laptop, simply go to leboncoin and make a search, with all the filters you want, and location you want. Dont' forget to launch the search  
[![Screenshot-2022-04-28-at-10-57-30.png](https://i.postimg.cc/02Tw1N9Q/Screenshot-2022-04-28-at-10-57-30.png)](https://postimg.cc/LnVXz2YK)  
and watch for :  
<center><img src="https://i.postimg.cc/90Y1DNC4/url1.png"  alt="url2" class="img-responsive"/></center>  
- Then copy the URL of first page of search results, and paste it in the start-URL of this Actor. Please add the root url first, so that it mimics the user behaviour, as shown in the image bellow :
<center><img src="https://i.postimg.cc/W4Xzzpc0/Screenshot-2022-01-11-at-22-50-27.png"  alt="url" class="img-responsive"/></center>  
- If you encounter issues, set up a french residential proxy. It helps because leboncoin bans non-french request from its website. Go to "Advanced Configuration" :
<center><img src="https://i.postimg.cc/7ZcqbKBY/Screenshot-2022-04-28-at-11-04-36.png"  alt="url" class="img-responsive"/></center>  

### 2 - Run & Download
- Click _Run_

Then wait for the result to appear in the "dataset" section and download them.

## Advanced Configuration

- Function : It's the function extracting the result on each leboncoin.fr pages. If the default is not enough for you, just update it using <a href="https://www.w3schools.com/jquery/jquery_ref_selectors.asp">jQuery selectors</a> to fetch other piece of information from leboncoin.fr !
- Avoid anti-bot detection with Proxy : the Proxy option sets proxies to prevent detection by leboncoin. Best to have french proxies since leboncoin bans foreign IPs. see <a href="https://apify.com/proxy">Apify Proxy</a> for more info

.actor/README.md

leboncoin-extractor

.actor/actor.json

# configurations
.idea

# crawlee and apify storage folders
apify_storage
crawlee_storage
storage

# installed files
node_modules

# git folder
.git


.dockerignore

root = true

[*]
indent_style = space
indent_size = 4
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
end_of_line = lf


.editorconfig

.eslintrc

# This file tells Git which files shouldn't be added to source control

.idea
node_modules


.gitignore

{
     "env": { "npm_config_loglevel": "silent" }
}


apify.json

import { Actor } from 'apify';
import { PuppeteerCrawler, createPuppeteerRouter, Dataset, puppeteerUtils } from 'crawlee';
import { evalFunctionOrThrow } from './tools.js';

await Actor.init();
const input = await Actor.getInput();

const proxyConfiguration = await Actor.createProxyConfiguration({
    groups: ['RESIDENTIAL'],
    countryCode: 'FR'
});

const f = evalFunctionOrThrow(input.pageFunction)

// console.log('input:',input);
const initialRequests =  input.startUrls
// const initialRequests = ['https://apify.com']; 

let router = createPuppeteerRouter();
router.addDefaultHandler(async ({ log, request, page }) => {
    log.info('handling: '+request.url)
    // await puppeteerUtils.saveSnapshot(page, { key: 'key_'+Math.random().toString(36).substring(7), saveHtml: true });
    await page.waitForSelector('h1',{visible:true,timeout:10000})
    // useful to debug in Live screenshost
    // await puppeteerUtils.saveSnapshot(page, { key: 'key_'+Math.random().toString(36).substring(7), saveHtml: true });
    await puppeteerUtils.injectJQuery(page);
    
    const data = await f({ log, request, page })     
    
    log.info(JSON.stringify(data))

    if(Array.isArray(data)){
        log.info('search page, with number of results : ', data.length)
        let promises = [];
        for (const d of data) {
            log.info(d)
            promises.push(Dataset.pushData(d))
        }
        await Promise.all(promises)
    }else{
        log.info('product page')
        await Dataset.pushData(data)
    }
    
});

// for more routing
// router.addHandler('detail', async ({ request, page, log }) => {
//     const title = await page.title();
//     log.info(`${title}`, { url: request.loadedUrl });
//     await Dataset.pushData({
//         url: request.loadedUrl,
//         title,
//     });
// });

// config for the crawler
const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    browserPoolOptions: {
        useFingerprints: true,
        fingerprintOptions: {
            fingerprintGeneratorOptions: {
                browsers: [
                    'chrome',
                    'firefox',
                ],
                devices: [
                    'desktop',
                ],
                locales: [
                    'fr-FR',
                ],
            },
        },
    },
});

// Run the crawler and wait for it to finish.
await crawler.run(initialRequests);
console.log('Crawler finished.');
await Actor.exit();


main.js

linkedin-url-people-crawler

package.json

import vm from 'vm'

/**
 * Transforms a page function string into a Function object.
 * @param {string} funcString
 * @return {Function}
 */
export const evalFunctionOrThrow = (funcString) => {
    let func;

    try {
        func = vm.runInThisContext(`(${funcString})`);
    } catch (err) {
        throw new Error(`Compilation of pageFunction failed.\n${err.message}\n${err.stack.substr(err.stack.indexOf('\n'))}`);
    }

    if (typeof func !== 'function') throw new Error('Input parameter "pageFunction" is not a function!');

    return func;
};

tools.js

Extract information from leboncoin.fr : simply provide your search URL (+ your filters and location). You will get the results easy, and fast

leboncoin

Extraire des informations de leboncoin facilement, simplement et rapidement de façon professionnelle ! Le meilleur outil pour scrapper leboncoin sans se faire détecter en tant que bot. Prix, date...

Leboncoin extractor

Leboncoin

How to Start

1 - Setup

2 - Run & Download

Advanced Configuration

Where next?