Bike Components Scraper

What does Bike Components Scraper do?

This is a scraper that collects almost all the items(name, prices, colors, sizes, details, etc.) on the product pages on the bike-components.de site.

Bike Components Scraper can scrape (All field names are in snakecase):

• name
• brand
• categories
• prices
• colors
• sizes
• prouct_description
• image_urls
• Product Details
  • specifications
  • components
  • additional_components_needed
  • characteristics
  • technical_information
  • adjustment_options
  • approved_for
  • compatibility
  • features
  • functions
  • models
  • technology
  • colour_info
  • manufacturer
  • manufacturer_notice
  • manufacturer_number
  • manufacturer_numbers
  • note_to_type_designation
  • note_from_the_manufacturer
  • contents
  • order_together
  • strengths_weaknesses
  • weight
  • comments
  • notes
  • dimensions
  • hazardous_substance
  • hazard_warnings
  • safety_information
  • valve_information

Why scrape [Bike Components]?

https://www.bike-components.de is a significant and influential player in the cycling industry, renowned for its extensive offerings and exceptional service. Here’s why bike-components.de stands out as a major online retailer.

1. Extensive Product Range

   Bike-components.de boasts an impressive catalog of thousands of cycling products. From a wide variety of bikes to components, accessories, and apparel, the site provides everything a cycling enthusiast or professional might need. This comprehensive selection ensures that customers can find exactly what they’re looking for in one convenient location.

2. High-Quality Products

   The site offers products that undergo rigorous quality checks, ensuring that customers receive only the best. With a focus on trusted and reputable brands like Shimano, SRAM, and Continental, bike-components.de guarantees high standards and reliability in all their offerings.

3. Competitive Pricing

   Bike-components.de is known for its competitive pricing, making top-quality cycling gear more accessible. Their pricing strategy, combined with frequent promotions and discounts, provides excellent value for customers, making it a preferred choice for both hobbyists and professional cyclists.

4. Expert Customer Service

   Customer service at bike-components.de is top-notch, with a team of knowledgeable professionals ready to assist with any inquiries or issues. Their dedication to customer satisfaction is reflected in their prompt and helpful support, which adds significant value to the overall shopping experience.

5. Educational Content and Resources

   In addition to their extensive product range, bike-components.de offers a wealth of educational content. This includes detailed product descriptions, buyer’s guides, and informative blog posts that help customers make informed decisions and stay updated on the latest trends and technologies in the cycling world.

6. Strong Community Presence

   Bike-components.de has cultivated a strong presence within the cycling community. Through sponsorships, event participation, and an active online community, they engage with and support cyclists at all levels, further cementing their reputation as a leader in the industry.

7. Efficient Shipping and Logistics

   With a robust logistics network, bike-components.de ensures fast and reliable shipping. Their efficient delivery system, coupled with transparent tracking, allows customers to receive their orders promptly and in perfect condition.

In summary, bike-components.de is a cornerstone in the cycling industry, offering a vast array of high-quality products, exceptional customer service, and valuable resources that cater to the needs of cyclists around the world.

How to scrape [Bike Components]

It's easy to scrape https://www.bike-components.de with Bike Components Scraper. Just follow these few steps and you'll get your data in a few minutes.

1. Click on Try for free.
2. Enter start urls(it should be product detail pages) you want to scrape.
3. Click on Run.
4. When Bike Components Scraper has finished, preview or download your data from the Dataset tab.

Results

The data collected must exported in JSON format.

1[
2    {
3        "image_urls": [
4            "https://www.bike-components.de/assets/p/i/640x480/537027.jpg",
5            "https://www.bike-components.de/assets/p/i/640x480/537028.jpg",
6            "https://www.bike-components.de/assets/p/i/640x480/537029.jpg",
7            "https://www.bike-components.de/assets/p/i/640x480/537030.jpg",
8            "https://www.bike-components.de/assets/p/i/640x480/537031.jpg",
9            "https://www.bike-components.de/assets/p/i/640x480/537032.jpg",
10            "https://www.bike-components.de/assets/p/i/640x480/537033.jpg",
11            "https://www.bike-components.de/assets/p/i/640x480/537034.jpg",
12            "https://www.bike-components.de/assets/p/i/640x480/537035.jpg",
13            "https://www.bike-components.de/assets/p/i/640x480/537036.jpg",
14            "https://www.bike-components.de/assets/p/i/640x480/537037.jpg",
15            "https://www.bike-components.de/assets/p/i/640x480/537038.jpg",
16            "https://www.bike-components.de/assets/p/i/640x480/537039.jpg",
17            "https://www.bike-components.de/assets/p/i/640x480/537040.jpg",
18            "https://www.bike-components.de/assets/p/i/640x480/537041.jpg",
19            "https://www.bike-components.de/assets/p/i/640x480/537050.jpg",
20            "https://www.bike-components.de/assets/p/i/640x480/537051.jpg",
21            "https://www.bike-components.de/assets/p/i/640x480/537052.jpg",
22            "https://www.bike-components.de/assets/p/i/640x480/537053.jpg",
23            "https://www.bike-components.de/assets/p/i/640x480/537054.jpg",
24            "https://www.bike-components.de/assets/p/i/640x480/537055.jpg",
25            "https://www.bike-components.de/assets/p/i/640x480/537056.jpg",
26            "https://www.bike-components.de/assets/p/i/640x480/537057.jpg",
27            "https://www.bike-components.de/assets/p/i/640x480/537058.jpg"
28        ],
29        "name": "Factor OSTRO V.A.M. Disc T47a Shimano Carbon Frameset - 2023 Model",
30        "brand": "Factor",
31        "item_number": "93128",
32        "ctgr_lg": "Home",
33        "ctgr_md": "Components",
34        "ctgr_sm": "Frames",
35        "colors": [
36            "factor racing green",
37            "factor racing green",
38            "factor racing green",
39            "factor racing green",
40            "factor racing green",
41            "factor racing green",
42            "factor racing green",
43            "factor racing green",
44            "factor racing green",
45            "chrome",
46            "chrome",
47            "chrome",
48            "chrome",
49            "chrome",
50            "chrome",
51            "chrome",
52            "chrome",
53            "chrome",
54            "chrome",
55            "chrome",
56            "chrome"
57        ],
58        "sizes": [
59            "56 cm, 100 mm x 38 cm, SB 25 mm, in stock  5",
60            "56 cm, 110 mm x 40 cm, SB 25 mm, in stock  5",
61            "56 cm, 120 mm x 42 cm, SB 25 mm, in stock  5",
62            "58 cm, 110 mm x 40 cm, SB 0 mm, sold out  5",
63            "58 cm, 110 mm x 40 cm, SB 25 mm, in stock  3",
64            "58 cm, 120 mm x 40 cm, SB 0 mm, sold out  5",
65            "58 cm, 120 mm x 40 cm, SB 25 mm, in stock  3",
66            "58 cm, 130 mm x 42 cm, SB 0 mm, sold out  5",
67            "58 cm, 130 mm x 42 cm, SB 25 mm, in stock  3",
68            "49 cm, 80 mm x 36 cm, SB 0 mm, sold out  5",
69            "49 cm, 80 mm x 36 cm, SB 25 mm, in stock  3",
70            "49 cm, 90 mm x 38 cm, SB 0 mm, sold out  5",
71            "49 cm, 90 mm x 38 cm, SB 25 mm, in stock  3",
72            "49 cm, 100 mm x 38 cm, SB 0 mm, sold out  5",
73            "49 cm, 100 mm x 38 cm, SB 25 mm, in stock  3",
74            "61 cm, 110 mm x 40 cm, SB 0 mm, sold out  5",
75            "61 cm, 110 mm x 40 cm, SB 25 mm, in stock  3",
76            "61 cm, 120 mm x 42 cm, SB 0 mm, sold out  5",
77            "61 cm, 120 mm x 42 cm, SB 25 mm, in stock  3",
78            "61 cm, 130 mm x 42 cm, SB 0 mm, sold out  5",
79            "61 cm, 130 mm x 42 cm, SB 25 mm, in stock  3"
80        ],
81        "prices": [
82            "5,452.94€",
83            "5,452.94€",
84            "5,452.94€",
85            "5,452.94€",
86            "3,453.53€",
87            "5,452.94€",
88            "3,453.53€",
89            "5,452.94€",
90            "3,453.53€",
91            "5,452.94€",
92            "3,453.53€",
93            "5,452.94€",
94            "3,453.53€",
95            "5,452.94€",
96            "3,453.53€",
97            "5,452.94€",
98            "3,453.53€",
99            "5,452.94€",
100            "3,453.53€",
101            "5,452.94€",
102            "3,453.53€"
103        ],
104        "explain": "The OSTRO V.A.M. carbon frameset from Factor, here in the version with Shimano bottom bracket, allows you to build a super-light, aerodynamic disc brake road bike. The fork, frame and seatpost have all been optimised with regard to aerodynamics. At the same time, the lightweight frameset impresses with its outstanding stiffness thanks to TeXtreme carbon fibres. TeXtreme consists of very flat fibres that can be laid wide, which results in very little material being used. These fibres are supported by pitch-based carbon fibres. The tyre clearance measures 32 mm, and the specially-designed fork can accommodate tyres up to 32 mm wide, both of which make mounting wider, more comfortable tyres easy and stress-free. A high-end headset and bottom bracket from CeramicSpeed deliver lower friction loss while riding, Thanks to the Aero handlebar stem unit from Black Inc, the frameset lives up to its name - OSTRO is one of the main Mediterranean winds. Do you want a bike for classic races like Paris-Roubaix? With the OSTRO V.A.M. frameset, you get the right basis for a comfortable, fast road bike.",
105        "id": "7417aed4e92b07c3b20cef686eaa83e8d3d10507",
106        "loaded_url": "https://www.bike-components.de/en/Factor/OSTRO-V-A-M-Disc-T47a-Shimano-Carbon-Frameset-2023-Model-p93128/",
107        "specifications": [
108            {
109                "application": "Road",
110                "wheel_size": "28\"",
111                "size": "49 cm, 52 cm, 54 cm, 56 cm, 58 cm, 61 cm",
112                "frame_material": "carbon",
113                "fork_material": "carbon"
114            }
115        ],
116        "technical_information": [
117            {
118                "target_parts": "frame",
119                "wheel_standard": "12 x 142 mm (standard)",
120                "axle_type": "12 mm thru-axle",
121                "headset": "IS41/28.6 - IS49/37(36) (top to bottom)",
122                "bottom_bracket_standard": "T47a",
123                "shell_width": "77 mm",
124                "chainline": "46.5 mm",
125                "brake_mount": "flat mount",
126                "front_derailleur_mount": "braze-on",
127                "cable_routing": "internal, electronic",
128                "bottle_cage_mounts": "2"
129            },
130            {
131                "target_parts": "fork",
132                "steerer_tube": "1 1/8\" - 1 3/8\" tapered",
133                "wheel_standard": "12 x 100 mm (standard)",
134                "axle_type": "12 mm thru-axle",
135                "brake_mount": "flat mount",
136                "offset": "43 mm, 48 mm, 53 mm",
137                "trail": "57.1 mm, 57.3 mm, 57.5 mm",
138                "axle_to_crown": "363 mm",
139                "over_locknut_dimension": "100 mm",
140                "steerer_tube_material": "carbon",
141                "crown_material": "carbon",
142                "blade_material": "carbon",
143                "dropout_material": "carbon"
144            }
145        ],
146        "approved_for": [
147            {
148                "weight_limit": "110 kg (system weight)",
149                "chainrings": "1x, 2x",
150                "rotor_size_(max.)": "160 mm",
151                "tyre_size_(max.)": "32-622 (700x32c)",
152                "chainring_size_(max.)": "55-tooth"
153            }
154        ],
155        "features": [
156            "lightweight, aerodynamic frame-fork kit for road bikes",
157            "high stiffness with low material usage due to special carbon fibre set-up",
158            "generous tyre clearance for more comfort",
159            "aerodynamic handlebar stem unit (120 mm drop, 80 mm reach, -6° angle)",
160            "friction-reducing headset and smooth-running bottom bracket from CeramicSpeed"
161        ],
162        "contents": [
163            "1 x Factor OSTRO V.A.M. frame",
164            "1 x Factor OSTRO fork",
165            "1 x Factor OSTRO seatpost",
166            "1 x CeramicSpeed headset",
167            "1 x CeramicSpeed T47a Shimano bottom bracket",
168            "1 x Black Inc Aero Integrated handlebar stem unit",
169            "1 x bike computer mount",
170            "1 x pair handlebar tape",
171            "1 x headset cover 5 mm",
172            "1 x headset cover 20 mm",
173            "1 x Factor spacer 2.5 mm",
174            "1 x Factor spacer 5 mm",
175            "1 x Factor spacer 10 mm"
176        ]
177    },
178    {
179        "image_urls": [
180            "https://www.bike-components.de/assets/p/i/640x480/403107.jpg",
181            "https://www.bike-components.de/assets/p/i/640x480/403108.jpg",
182            "https://www.bike-components.de/assets/p/i/640x480/403109.jpg",
183            "https://www.bike-components.de/assets/p/i/640x480/403110.jpg",
184            "https://www.bike-components.de/assets/p/i/640x480/403111.jpg",
185            "https://www.bike-components.de/assets/p/i/640x480/403112.jpg",
186            "https://www.bike-components.de/assets/p/i/640x480/403113.jpg",
187            "https://www.bike-components.de/assets/p/i/640x480/403114.jpg",
188            "https://www.bike-components.de/assets/p/i/640x480/403115.jpg",
189            "https://www.bike-components.de/assets/p/i/640x480/403116.jpg",
190            "https://www.bike-components.de/assets/p/i/640x480/403117.jpg",
191            "https://www.bike-components.de/assets/p/i/640x480/403118.jpg"
192        ],
193        "name": "Continental Grand Prix 5000 28\" Folding Tyre",
194        "brand": "Continental",
195        "item_number": "68409",
196        "ctgr_lg": "Home",
197        "ctgr_md": "Components",
198        "ctgr_sm": "Tyres",
199        "colors": [
200            "black",
201            "black",
202            "black",
203            "black",
204            "black",
205            "black-creme",
206            "black-creme",
207            "black-transparent",
208            "black-transparent"
209        ],
210        "sizes": [
211            "23-622 (700x23c)",
212            "25-622 (700x25c)",
213            "28-622 (700x28c)",
214            "30-622 (700x30c)",
215            "32-622 (700x32c)",
216            "25-622 (700x25c)",
217            "28-622 (700x28c)",
218            "25-622 (700x25c)",
219            "28-622 (700x28c)"
220        ],
221        "prices": [
222            "30.29€",
223            "34.61€",
224            "34.61€",
225            "34.61€",
226            "34.61€",
227            "34.61€",
228            "34.61€",
229            "34.61€",
230            "34.61€"
231        ],
232        "explain": "The Grand Prix 5000 folding tyre in a 28-inch size from Continental is the logical tyre choice if you ride your road bike with a tube-tyre combination. This premium tyre impresses with low rolling resistance, high mileage and puncture protection at top speeds. The BlackChili natural rubber compound has a good balance of grip, rolling resistance and mileage. The Lazer Grip structure gives you good grip, especially in curves, and prevents the tyre from slipping. This keeps you safe and fast in all conditions, rain or shine, warmth or cold. The integrated Vectran™ Breaker high-tech fibres and the three-layer carcass ensure high puncture protection at low weight. Thanks to Active Comfort Technology, you ride comfortably at all times because vibrations are absorbed. Whether as a training or competition tyre, the Grand Prix 5000 offers you full performance at all times.",
233        "id": "5f6b849a1d974a84568ac3d2729f491a0e93ffd5",
234        "loaded_url": "https://www.bike-components.de/en/Continental/Grand-Prix-5000-28-Folding-Tyre-p68409/?v=53662-black",
235        "specifications": [
236            {
237                "application": "Road",
238                "product_line": "Premium",
239                "ground_conditions": "street",
240                "type": "folding tyre",
241                "tubeless_system": "none"
242            }
243        ],
244        "technical_information": [
245            {
246                "rubber_compound": "BlackChili",
247                "carcass": "330 tpi, 3 layers",
248                "sidewall": "skin",
249                "puncture_protection": "Vectran Breaker"
250            }
251        ],
252        "models": [
253            {
254                "target_model": "23-622 (700x23c)",
255                "dimensions": "700 x 23c",
256                "etrto": "23-622",
257                "air_pressure": "7.5 - 8.5 bars",
258                "tube_recommendation": "Race 28",
259                "manufacturer_number": "0101623"
260            },
261            {
262                "target_model": "25-622 (700x25c)",
263                "dimensions": "700 x 25c",
264                "etrto": "25-622",
265                "air_pressure": "6.5 - 8.5 bars",
266                "tube_recommendation": "Race 28",
267                "manufacturer_numbers": "0101624 (black), 010947 (black-cream), 0101895 (black-transparent)"
268            },
269            {
270                "target_model": "28-622 (700x28C)",
271                "dimensions": "700 x 28c",
272                "etrto": "28-622",
273                "air_pressure": "6.5 - 8.0 bars",
274                "tube_recommendation": "Race 28 Wide",
275                "manufacturer_numbers": "0101625 (black), 0101948 (black-cream), 0101896 (black-transparent)"
276            },
277            {
278                "target_model": "30-622 (700x30c)",
279                "dimensions": "700 x 30c",
280                "etrto": "30-622",
281                "air_pressure": "6.0 - 7.0 bars",
282                "tube_recommendation": "Race 28 Wide",
283                "manufacturer_number": "0101813"
284            },
285            {
286                "target_model": "32-622 (700x32c)",
287                "dimensions": "700 x 32c",
288                "etrto": "32-622",
289                "air_pressure": "6.0 - 7.0 bars",
290                "tube_recommendation": "Race 28 Wide",
291                "manufacturer_number": "0101626"
292            }
293        ],
294        "features": [
295            "12 % lower rolling resistance, 20 % better puncture protection, 10 g lighter (25-622) than predecessor GP 4000 S II",
296            "Legendary BlackChili rubber compound on a new optimized level for excellent grip, higher mileage and even more efficient rolling",
297            "Vectran Breaker made of synthetically produced high-tech fibre, even more tear-resistant at lowest weight, even stronger and more puncture-proof",
298            "increased cornering grip thanks to Lazer Grip",
299            "increased comfort thanks to Active Comfort Technology"
300        ],
301        "technology": [
302            {
303                "title": "Premium",
304                "content": "The Continental Premium tyres have been designed to deliver top performance. They are equipped with the latest technologies, which have been developed together with the most successful athletes in the world. They are handmade by the experienced specialists in Korbach (Germany). Added to this are selected materials for the highest quality and outstanding properties."
305            },
306            {
307                "title": "BlackChili Compound",
308                "content": "The rolling characteristics of the rubber compound of a tyre are determined by three interacting factors: Static friction (grip), rolling resistance and mileage. The dilemma: If one of these factors is improved, at least one other factor deteriorates in return. BlackChili Compound is the name of Continental's revolutionary tread compound based on the latest findings in polymer and raw material research. For BlackChili, Continental refines newly developed synthetic rubbers with proven natural rubber into particularly high-performance tread compounds. Conti adds nanoscale carbon black particles to these rubbers, which are optimised in shape and surface properties in order to match the compound to the respective application of the tyre. Tyres with BlackChili Compound achieve an unprecedented level of performance.The complex compound technology is so sensitive that tyres with BlackChili Compound are produced exclusively at Continental in Korbach, Germany."
309            },
310            {
311                "title": "Vectran™ Breaker",
312                "content": "The reference in puncture protection. A synthetically produced high-tech fibre with spun silk as a natural model. Vectran is spun from the molten liquid polymer Vectra and processed into a multi-fibre yarn. Enormously tear-resistant at the lowest possible weight - just right to be used as puncture protection in a premium tyre by Continental. And this without affecting the rolling resistance."
313            },
314            {
315                "title": "Lazer Grip",
316                "content": "Lazer Grip makes you one with the road. The laser cut micro-profile structure focuses on the shoulder area of the tyre for enhanced cornering grip."
317            },
318            {
319                "title": "Active Comfort",
320                "content": "Active Comfort is a revolutionary approach to road cycling. Integrated into the tyre construction, Active Comfort Technology absorbs shocks and ensures outstanding driving experience."
321            }
322        ],
323        "contents": ["1 x Continental Grand Prix 5000 folding tyres"],
324        "weight": [
325            "black/23-622 (700x23c): 205 g",
326            "black/25-622 (700x25c): 220 g",
327            "black/28-622 (700x28c): 250 g",
328            "black/30-622 (700x30c): 260 g",
329            "black/32-622 (700x32c): 295 g",
330            "black-creme/25-622 (700x25c): 260 g",
331            "black-creme/28-622 (700x28c): 290 g",
332            "black-transparent/25-622 (700x25c): 230 g",
333            "black-transparent/28-622 (700x28c): 245 g"
334        ]
335    },
336    {
337        "image_urls": [
338            "https://www.bike-components.de/assets/p/i/640x480/338448.jpg",
339            "https://www.bike-components.de/assets/p/i/640x480/338449.jpg",
340            "https://www.bike-components.de/assets/p/i/640x480/338450.jpg",
341            "https://www.bike-components.de/assets/p/i/640x480/338451.jpg"
342        ],
343        "name": "SRAM XG-1275 12-speed Cassette for GX Eagle",
344        "brand": "SRAM",
345        "item_number": "57614",
346        "ctgr_lg": "Home",
347        "ctgr_md": "Components",
348        "ctgr_sm": "Drivetrain",
349        "colors": ["black", "black"],
350        "sizes": ["10-52", "10-50"],
351        "prices": ["129.82€", "86.55€"],
352        "explain": "The SRAM XG-1275 cassette for GX Eagle has twelve shifting levels for riders who value reliability and durability. Mud and dirt do not stand a chance to get stuck between the sprockets in its open design. The FULL-PIN™ technology, which combines precision machined steel sprockets with high-resistance stainless steel pins results in a lighter, robust cassette with good self-cleaning abilities, smooth shifting and high durability. With 10-52 teeth, you get a huge range that will give you the right gear for even the toughest climbs. Or would you like to switch to a larger chainring and reach higher speeds without compromising the climbing ability? You have all the options!",
353        "id": "dfa3a35ab45f2536d74be33541f625c8b9662399",
354        "loaded_url": "https://www.bike-components.de/en/SRAM/XG-1275-12-speed-Cassette-for-GX-Eagle-p57614/",
355        "specifications": [
356            {
357                "application": "Cross Country, All Mountain, Enduro",
358                "gearing": "12-speed",
359                "gradation": "10-50, 10-52",
360                "sprocket_material": "steel (sprockets 1 - 11), aluminium (sprocket 12)",
361                "freehub_compatibility": "SRAM XD"
362            }
363        ],
364        "technical_information": [
365            {
366                "series": "GX Eagle",
367                "recommended_chain": "PC GX Eagle"
368            }
369        ],
370        "compatibility": [
371            {
372                "target_model": "10-50",
373                "values": {
374                    "contents": [
375                        "electronic XX1/X01 Eagle AXS 12-speed rear derailleurs",
376                        "mechanical XX1/X01/GX/SX/NX Eagle 12-speed rear derailleurs, all models"
377                    ]
378                }
379            },
380            {
381                "target_model": "10-52",
382                "values": {
383                    "caution": " The 10-52 gradation is only compatible with mechanical 12-speed rear derailleurs of the GX Eagle, X01 Eagle and XX1 Eagle series as of model 2020 and cannot be combined with older rear derailleurs.",
384                    "contents": [
385                        "electronic XX1/X01 Eagle AXS 12-speed rear derailleurs"
386                    ]
387                }
388            }
389        ],
390        "features": [
391            "large range of 520 % with 10-52 teeth",
392            "optimised switching stages over the entire range",
393            "open,mud resistant design",
394            "colour-coordinated with Eagle components"
395        ],
396        "technology": [
397            {
398                "title": "FULL PIN™:",
399                "content": "The FULL-PIN technology uses eleven lightweight sprockets made of pressed steel, which are securely held together by 123 pins made of high-strength steel. The result is an light and durable cassette that is perfect for any ride with a wide gear ratio. Similar to the X-DOME™ cassette, the open design keeps out mud and dirt while ensuring clean shifts and long component life."
400            },
401            {
402                "title": "X-Glide 2™:",
403                "content": "The X-Glide 2 is the next generation of X-Glide technology. It has been specially optimised for the Eagle 12-speed cassette - for fast and precise shifting performance in all gears. X-Glide technology increases chain tension on the larger sprockets."
404            }
405        ],
406        "contents": ["1 x SRAM XG-1275 cassette"],
407        "weight": ["black/10-50: 450 g"]
408    }
409]

Is it legal to scrape [Bike Components]?

Note that personal data is protected by GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. We also recommend that you read our blog post: is web scraping legal?