Opensea Collection Scraper

Features

This OpenSea collection Scraper will enable you scrape any collection from OpenSea.io.

OpenSea is also providing APIs but they come with unexplained throttling & community is screaming about it! More about it on this github issue

So, OpenSea API doesn't help here reliably if you want to get complete list of all tokens in particular collection. Moreover, if you visit any collection on website, you will find same token listed more than once on listing page.

So, even with website users there is no official way to get complete detail of collection tokens. Whether you are doing this for your next analysis project, investment analysis or even if you are owner of the collection, this scraper will help you to get complete list of all the tokens in particular collection.

Input

For simple usecase, you just need to provide browser url of opensea collection & that's all. You can leave other fields as they are to be sensible defaults.

Input example

1{
2  "startUrls": [
3    {
4      "url": "https://opensea.io/collection/parallelalpha"
5    }
6  ],
7  "maxItemsPerCollection" : 20,
8  "scrapeProperties": false,
9  "generateAssetFile": false,
10  "downloadImages": false,
11  "extendOutputFunction": "(asset) => {
12        const result = {};
13        // Uncomment to add a asset id to the output
14        // result.id = asset.id;
15        return result;
16    }",
17}

If you want some more customization :

• scrapeProperties : This option will enable scraper to go deeper into scraping to retrive more information about each asset. Please consider that turning this on will take more scraping time

• generateAssetFile : This option will generate separate JSON file for each asset. You can find zipped file under Apify 'assets' key value store output once process is complete. This zip will contain all json files under single folder.

• downloadImages : This option will download image for each asset and generate zip folder of the same. You can find zipped file under Apify 'images' key value store output once process is complete. This zip will contain all image files under single folder. This option will take considerable amount of memory. Please provide enough memory to the actor if you are enabling it. (4GB memory required)

• maxItemsPerCollection : You can limit number of items to be scraped on per collection basis with the help of this input.

Output

The scraped data is stored in the dataset of each run. The data can be viewed or downloaded in many popular formats, such as JSON, CSV, Excel, XML, RSS, and HTML.

Output example

The result for scraping a single token like this:

1{
2  "tokenId": "6",
3  "name": null,
4  "collection": "Bored Ape Yacht Club",
5  "collectionSlug": "boredapeyachtclub",
6  "imageUrl": "https://img.seadn.io/files/8d34186c8c41631bff187d1b8810bc36.png?fit=max&h=1200&w=1200&auto=format",
7  "contractAddress": "0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d",
8  "chain": "ETHEREUM",
9  "favoritesCount": 114,
10  "ownersCount": 3,
11  "bestAsk": 199.69,
12  "bestAskUsd": 257560.16,
13  "url": "https://opensea.io/assets/ethereum/0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d/6",
14  "imageName": "boredapeyachtclub-6.avif",
15  "owner": "820425",
16  "ownerLink": "https://opensea.io/0x820425d330F777370aCF68E45F884547e98dB2C0",
17  "creator": "BoredApeYachtClub",
18  "creatorLink": "https://opensea.io/BoredApeYachtClub?tab=created",
19  "views": "9.0K",
20  "attributes": [
21    {
22      "trait_type": "Background",
23      "value": "Yellow",
24      "rarity": "13%"
25    },
26    {
27      "trait_type": "Clothes",
28      "value": "Tweed Suit",
29      "rarity": "1%"
30    },
31    {
32      "trait_type": "Eyes",
33      "value": "Crazy",
34      "rarity": "4%"
35    },
36    {
37      "trait_type": "Fur",
38      "value": "Cream",
39      "rarity": "6%"
40    },
41    {
42      "trait_type": "Hat",
43      "value": "S&m Hat",
44      "rarity": "2%"
45    },
46    {
47      "trait_type": "Mouth",
48      "value": "Bored",
49      "rarity": "23%"
50    }
51  ]
52}

Modify the output

There is an additional input extendOutputFunction provided using which you can customize the output JSON. If you are good with Javascript, you can simply edit this default function to your use-case and override the output.

Limitations

Since opensea supports browsing only upto 10K items in collection, it's advisible to use this actor for collection upto the size of 10K & max upto 20K.

For custom solutions

In case you need some custom solution, you can contact me : dhrumil@techvasu.com

Or learn more about me on github : https://github.com/dhrumil4u360