Github Trendings
DeprecatedView all Actors
This Actor is deprecated
This Actor is unavailable because the developer has decided to deprecate it. Would you like to try a similar Actor instead?
See alternative ActorsGithub Trendings
drobnikj/github-trendings
Extracts trending repositories from GitHub.
Dockerfile
1# This is a template for a Dockerfile used to run acts in Actor system.
2# The base image name below is set during the act build, based on user settings.
3# IMPORTANT: The base image must set a correct working directory, such as /usr/src/app or /home/user
4FROM apify/actor-node-chrome:v0.21.10
5
6# Second, copy just package.json and package-lock.json since it should be
7# the only file that affects "npm install" in the next step, to speed up the build
8COPY package*.json ./
9
10# Install NPM packages, skip optional and development dependencies to
11# keep the image small. Avoid logging too much and print the dependency
12# tree for debugging
13RUN npm --quiet set progress=false \
14 && npm install --only=prod --no-optional \
15 && echo "Installed NPM packages:" \
16 && (npm list --all || true) \
17 && echo "Node.js version:" \
18 && node --version \
19 && echo "NPM version:" \
20 && npm --version
21
22# Copy source code to container
23# Do this in the last step, to have fast build if only the source code changed
24COPY . ./
25
26# NOTE: The CMD is already defined by the base image.
27# Uncomment this for local node inspector debugging:
28# CMD [ "node", "--inspect=0.0.0.0:9229", "main.js" ]package.json
1{
2 "name": "apify-project",
3 "version": "0.0.1",
4 "description": "",
5 "author": "It's not you it's me",
6 "license": "ISC",
7 "dependencies": {
8 "apify": "0.21.10",
9 "mongodb": "latest"
10 },
11 "scripts": {
12 "start": "node main.js"
13 }
14}main.js
1const Apify = require('apify');
2const MongoClient = require('mongodb').MongoClient;
3
4Apify.main(async () => {
5 const input = await Apify.getValue('INPUT');
6
7 const requestList = new Apify.RequestList({
8 sources: input
9 });
10
11 // This call loads and parses the URLs from the remote file.
12 await requestList.initialize();
13
14 const crawler = new Apify.PuppeteerCrawler({
15 requestList,
16 handlePageFunction: async ({ page, request }) => {
17 await Apify.utils.puppeteer.injectJQuery(page);
18 const reposList = await page.evaluate(() => {
19 const reposList = []
20 $('.Box-row').each(function() {
21 reposList.push({
22 name: $(this).find('h1').text().trim(),
23 description: $(this).find('p').text().trim(),
24 todayStars: parseInt($(this).find('.float-sm-right').text()),
25 url: $(this).find('a').get(0).href,
26 })
27 });
28 return reposList;
29 });
30 await Apify.pushData({ language: request.userData.language, reposList });
31 },
32 handleFailedRequestFunction: async ({ request }) => {
33 // This function is called when the crawling of a request failed too many times
34 await Apify.pushData({
35 url: request.url,
36 succeeded: false,
37 errors: request.errorMessages,
38 })
39 },
40 });
41
42 await crawler.run();
43
44 const dataset = await Apify.openDataset();
45 const data = await dataset.getData({ clean: true });
46 const results = data.items;
47
48 const client = await MongoClient.connect(process.env.MONGO_URL);
49 const db = client.db()
50 const todayCollection = db.collection('trendingToday');
51 for (let result of results) {
52 const language = result.language;
53 for (let repo of result.reposList) {
54 repo.createdAt = new Date();
55 repo.language = language;
56 const repoInCollection = await todayCollection.findOne({ name: repo.name });
57 if (!repoInCollection) await todayCollection.insert(repo);
58 }
59 }
60 await client.close();
61
62
63 await Apify.setValue('OUTPUT', results);
64 console.log('done');
65});Developer
Jakub DrobníkMaintained by Community
Categories
