# This file tells Git which files shouldn't be added to source control


.gitignore

# First, specify the base Docker image.
# You can see the Docker images from Apify at https://hub.docker.com/r/apify/.
# You can also use any other image from Docker Hub.
FROM apify/actor-python:3.9

# Second, copy just requirements.txt into the actor image,
# since it should be the only file that affects "pip install" in the next step,
# in order to speed up the build
COPY requirements.txt ./

# Install the packages specified in requirements.txt,
# Print the installed Python version, pip version
# and all installed packages with their versions for debugging
RUN echo "Python version:" \
 && python --version \
 && echo "Pip version:" \
 && pip --version \
 && echo "Installing dependencies from requirements.txt:" \
 && pip install -r requirements.txt \
 && echo "All installed Python packages:" \
 && pip freeze

# Next, copy the remaining files and directories with the source code.
# Since we do this after installing the dependencies, quick build will be really fast
# for most source file changes.
COPY . ./

# Specify how to launch the source code of your actor.
# By default, the main.py file is run
CMD python3 main.py


Dockerfile

An Awesome scrapper for data in Wikipedia

Search Term or Url

Language to Search

Return Only the description of the article

Description only

Output Type array or key value

Combine paragraphs into one when output_type is key_value

Combine paragraphs

Wikipedia Scrapper

INPUT_SCHEMA.json

# Wikipedia Scraper

## What does Wikipedia Scraper do?

Wikipedia Scraper will enable you to get more data from Wikipedia, one of the biggest Human repository research, An extremely updated and Trustable way to get your Data.

Wikipedia Scraper can scrape:

- Article Description
- Article full content

## Why scrape Wikipedia Scraper?

Wikipedia has 43 millions users and is a great source of data for all fields and cases.

Here are just some of the ways you could use that data:

- Quick and fast answers inside your website
- Resource data to training your machine training algorithm
- Tool for your application

If you would like more inspiration on how scraping Wikipedia Scraper could help your business or organization, check out our [industry pages](https://apify.com/industries).

## How to scrape Wikipedia

It's easy to scrape [Wikipedia](https://www.wikipedia.org/) with Wikipedia Scraper. Just follow these few steps and you'll get your data in a few minutes.

1. Click on Try for free.
2. Enter the keywords or search terms you want to scrape. Or enter a URL to start scraping.
3. Click on Run.
4. When Wikipedia Scraper has finished, preview or download your data from the Dataset tab.

## Results

```json
[{
  "version": "0.1.0",
  "input": {
    "lang": "en",
    "input_value": "python development",
    "output_type": "key_value",
    "paragraph_together": false,
    "only_description": false
  },
  "output": [
    {
      "input_value": "python development",
      "output_value": {
        "Description": [
          "The programming language Python was conceived in the late 1980s, and its implementation was started in December 1989 by Guido van Rossum at CWI in the Netherlands as a successor to ABC capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL). (However, van Rossum stepped down as leader on July 12, 2018.). Python was named after the BBC TV show Monty Python's Flying Circus.",
          "Python 2.0 was released on October 16, 2000, with many major new features, including a cycle-detecting garbage collector (in addition to reference counting) for memory management and support for Unicode. However, the most important change was to the development process itself, with a shift to a more transparent and community-backed process.",
          "Python 3.0, a major, backwards-incompatible release, was released on December 3, 2008 after a long period of testing. Many of its major features have also been backported to the backwards-compatible, though now-unsupported, Python 2.6 and 2.7."
        ],
        "Early history": [
          "In February 1991, Van Rossum published the code (labeled version 0.9.0) to alt.sources. Already present at this stage in development were classes with inheritance, exception handling, functions, and the core datatypes of `list`, `dict`, `str` and so on. Also in this initial release was a module system borrowed from Modula-3; Van Rossum describes the module as \"one of Python's major programming units\". Python's exception model also resembles Modula-3's, with the addition of an `else` clause. In 1994 comp.lang.python, the primary discussion forum for Python, was formed, marking a milestone in the growth of Python's userbase."
        ],
        "Version 1": [
          "Python reached version 1.0 in January 1994. The major new features included in this release were the functional programming tools `lambda`, `map`, `filter` and `reduce`. Van Rossum stated that \"Python acquired lambda, reduce(), filter() and map(), courtesy of a Lisp hacker who missed them and submitted working patches\".",
          "The last version released while Van Rossum was at CWI was Python 1.2. In 1995, Van Rossum continued his work on Python at the Corporation for National Research Initiatives (CNRI) in Reston, Virginia from where he released several versions.",
          "By version 1.4, Python had acquired several new features. Notable among these are the Modula-3 inspired keyword arguments (which are also similar to Common Lisp's keyword arguments) and built-in support for complex numbers. Also included is a basic form of data hiding by name mangling, though this is easily bypassed.",
          "During Van Rossum's stay at CNRI, he launched the Computer Programming for Everybody (CP4E) initiative, intending to make programming more accessible to more people, with a basic \"literacy\" in programming languages, similar to the basic English literacy and mathematics skills required by most employers. Python served a central role in this: because of its focus on clean syntax, it was already suitable, and CP4E's goals bore similarities to its predecessor, ABC. The project was funded by DARPA. As of 2007, the CP4E project is inactive, and while Python attempts to be easily learnable and not too arcane in its syntax and semantics, outreach to non-programmers is not an active concern.",
          "In 2000, the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team, under the direction of early Google alum Domenic Merenda. CNRI requested that a version 1.6 be released, summarizing Python's development up to the point at which the development team left CNRI. Consequently, the release schedules for 1.6 and 2.0 had a significant amount of overlap. Python 2.0 was the only release from BeOpen.com. After Python 2.0 was released by BeOpen.com, Guido van Rossum and the other PythonLabs developers joined Digital Creations.",
          "The Python 1.6 release included a new CNRI license that was substantially longer than the CWI license that had been used for earlier releases. The new license included a clause stating that the license was governed by the laws of the State of Virginia. The Free Software Foundation argued that the choice-of- law clause was incompatible with the GNU General Public License. BeOpen, CNRI and the FSF negotiated a change to Python's free software license that would make it GPL-compatible. Python 1.6.1 is essentially the same as Python 1.6, with a few minor bug fixes, and with the new GPL-compatible license."
        ],
        "Version 2": [
          "Python 2.0, released October 2000, introduced list comprehensions, a feature borrowed from the functional programming languages SETL and Haskell. Python's syntax for this construct is very similar to Haskell's, apart from Haskell's preference for punctuation characters and Python's preference for alphabetic keywords. Python 2.0 also introduced a garbage collector capable of collecting reference cycles.",
          "Python 2.1 was close to Python 1.6.1, as well as Python 2.0. Its license was renamed Python Software Foundation License. All code, documentation and specifications added, from the time of Python 2.1's alpha release on, is owned by the Python Software Foundation (PSF), a non-profit organization formed in 2001, modeled after the Apache Software Foundation. The release included a change to the language specification to support nested scopes, like other statically scoped languages. (The feature was turned off by default, and not required, until Python 2.2.)",
          "Python 2.2 was released in December 2001; a major innovation was the unification of Python's types (types written in C) and classes (types written in Python) into one hierarchy. This single unification made Python's object model purely and consistently object oriented. Also added were generators which were inspired by Icon.",
          "Python 2.5 was released in September 2006 and introduced the `with` statement, which encloses a code block within a context manager (for example, acquiring a lock before the block of code is run and releasing the lock afterwards, or opening a file and then closing it), allowing Resource Acquisition Is Initialization (RAII)-like behavior and replacing a common try/finally idiom.",
          "Python 2.6 was released to coincide with Python 3.0, and included some features from that release, as well as a \"warnings\" mode that highlighted the use of features that were removed in Python 3.0. Similarly, Python 2.7 coincided with and included features from Python 3.1, which was released on June 26, 2009. Parallel 2.x and 3.x releases then ceased, and Python 2.7 was the last release in the 2.x series. In November 2014, it was announced that Python 2.7 would be supported until 2020, but users were encouraged to move to Python 3 as soon as possible. Python 2.7 support ended on January 1, 2020, along with code freeze of 2.7 development branch. A final release, 2.7.18, occurred on April 20, 2020, and included fixes for critical bugs and release blockers. This marked the end-of-life of Python 2."
        ],
        "Version 3": [
          "Python 3.0 (also called \"Python 3000\" or \"Py3K\") was released on December 3, 2008. It was designed to rectify fundamental design flaws in the language—the changes required could not be implemented while retaining full backwards compatibility with the 2.x series, which necessitated a new major version number. The guiding principle of Python 3 was: \"reduce feature duplication by removing old ways of doing things\".",
          "Python 3.0 was developed with the same philosophy as in prior versions. However, as Python had accumulated new and redundant ways to program the same task, Python 3.0 had an emphasis on removing duplicative constructs and modules, in keeping with \"There should be one— and preferably only one —obvious way to do it\".",
          "Nonetheless, Python 3.0 remained a multi-paradigm language. Coders could still follow object-oriented, structured, and functional programming paradigms, among others, but within such broad choices, the details were intended to be more obvious in Python 3.0 than they were in Python 2.x.",
          "Python 3.0 broke backward compatibility, and much Python 2 code does not run unmodified on Python 3. Python's dynamic typing combined with the plans to change the semantics of certain methods of dictionaries, for example, made perfect mechanical translation from Python 2.x to Python 3.0 very difficult. A tool called \"2to3\" does the parts of translation that can be done automatically. At this, 2to3 appeared to be fairly successful, though an early review noted that there were aspects of translation that such a tool would never be able to handle. Prior to the roll-out of Python 3, projects requiring compatibility with both the 2.x and 3.x series were recommended to have one source (for the 2.x series), and produce releases for the Python 3.x platform using 2to3. Edits to the Python 3.x code were discouraged for so long as the code needed to run on Python 2.x. This is no longer recommended; as of 2012 the preferred approach was to create a single code base that can run under both Python 2 and 3 using compatibility modules.",
          "Some of the major changes included for Python 3.0 were:",
          "Subsequent releases in the Python 3.x series have included additional, substantial new features; all ongoing development of the language is done in the 3.x series."
        ],
        "Table of versions": [
          "Releases before numbered versions:",
          "Table notes:"
        ]
      }
    }
  ],
  "status_code": 200,
  "status": "OK"
}]
```

## Tips for scraping Wikipedia

- Verify the keywords for the searching
- If using url to search verify if follows this pattern: `https://{lang}.wikipedia.org/wiki/{page_name}`

## Is it legal to scrape Wikipedia

Note that personal data is protected by GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. We also recommend that you read our blog post: [is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/)

README.md

apify.json

__version__ = '0.1.0'
__author__ = 'github.com/kilerhg'

import os
import json
import requests
import re
import logging

from bs4 import BeautifulSoup
from parsel import Selector
import html2text

TEMPLATE_URL = 'https://{lang}.wikipedia.org'
TEMPLATE_URL_WIKI = '/wiki/{title}'

BASE_URL_SEARCH = '/w/index.php?search={search}'

REGEX_VERIFY_URL_WIKI = r'http(s)?\:\/\/([a-z\-]*)\.wikipedia.org\/wiki\/(.*)'

XPATH_CONTENT_PAGE = '//div[@class="mw-parser-output"]/p|//div[@class="mw-parser-output"]/h2|//div[@class="mw-parser-output"]/ul|//div[@class="mw-parser-output"]/ol'
XPATH_CONTENT_H = '//span[@class="mw-headline"]'
XPATH_SEARCH_PAGE = '//div[@class="mw-search-result-heading"]/a/@href'

SEARCH_CONTENT = 'mw-search-result-heading'
EMPTY_PAGE = 'noarticletext mw-content-ltr'

ALL_OUTPUT_TYPE = ['array', 'key_value']

HTTP_CODE = {
    200: 'OK',
    400: 'Bad Request',
}

ALL_LANGS_WIKI = [
    'af',
    'pl',
    'sk',
    'ar',
    'ast',
    'az',
    'bg',
    'nan',
    'bn',
    'be',
    'ca',
    'cs',
    'cy',
    'da',
    'de',
    'et',
    'el',
    'en',
    'es',
    'eo',
    'eu',
    'fa',
    'fr',
    'gl',
    'ko',
    'hy',
    'hi',
    'hr',
    'id',
    'it',
    'he',
    'ka',
    'la',
    'lv',
    'lt',
    'hu',
    'mk',
    'arz',
    'ms',
    'min',
    'nl',
    'ja',
    'no',
    'nn',
    'ce',
    'uz',
    'pt',
    'kk',
    'ro',
    'ru',
    'simple',
    'ceb',
    'sl',
    'sr',
    'sh',
    'fi',
    'sv',
    'ta',
    'tt',
    'th',
    'tg',
    'azb',
    'tr',
    'uk',
    'ur',
    'vi',
    'vo',
    'war',
    'zh-yue',
    'zh',
    'my'
]


def request_website(url):
    respose = requests.get(url).content
    website = BeautifulSoup(respose,'html.parser')
    return website

def get_text_html(html_content):
    h = html2text.HTML2Text()
    h.ignore_links = True
    h.ignore_emphasis = True
    h.ignore_images = True
    h.ignore_tables = True

    text_clear = h.handle(str(html_content))

    text_clear = text_clear.replace('\n', ' ')
    text_clear = re.sub(r'( )+', ' ', text_clear)
    text_clear = str(text_clear).strip()
    
    return text_clear

def verify_page_empty(response):
    is_page_empty = EMPTY_PAGE in str(response)
    return is_page_empty

def verify_lang_valid(lang : str) -> str:
    is_lang_valid = lang in ALL_LANGS_WIKI
    return is_lang_valid

def verify_url_wiki(url : str) -> bool:
    is_url_valid = not not re.match(REGEX_VERIFY_URL_WIKI, url) # not empty
    return is_url_valid

def search_url(lang, search) -> str:
    
    search = str(search).strip().replace(' ', '+')
    template_url_lang = TEMPLATE_URL.format(lang=lang)
    template_url_lang_search = template_url_lang + BASE_URL_SEARCH.format(search=search)

    response = request_website(template_url_lang_search)

    if SEARCH_CONTENT in str(response):
        obj_sel = Selector(str(response))
        content = obj_sel.xpath(XPATH_SEARCH_PAGE)[0].get()
        redirect_url = f'{template_url_lang}{content}'
    else:
        redirect_url = re.findall(r'<link href="(.*)?" rel="canonical"\/>', str(response))[0]
    
    return redirect_url

def content_process_page_array(response, first_section_name='Description', only_first_section=False):
    obj_sel = Selector(str(response))
    content = obj_sel.xpath(XPATH_CONTENT_PAGE)
    list_all_values = []
    section_name = first_section_name
    for tag in content:
        html_content = tag.get()
        name_tag = re.findall(r'^<([a-z\d]*)', html_content)[0]
        if name_tag[0] == 'h':
            if only_first_section:
                break
            obj_sel1 = Selector(str(html_content))
            content_h_tag = obj_sel1.xpath(XPATH_CONTENT_H)
            html_content = content_h_tag[0].get()
            section_name = get_text_html(html_content)
            
        elif name_tag[0] == 'p' or name_tag in ('ul', 'ol'):
            clear_text = re.sub(r'<sup(.*?)<\/sup>', '', html_content)
            clear_text = re.sub(r'<abbr(.*?)<\/abbr>', '', clear_text)
            clear_text = get_text_html(clear_text)
            clear_text = str(clear_text).strip()
            if clear_text:
                dict_register = {
                    'section_name': section_name,
                    'text': clear_text
                }
                list_all_values.append(dict_register)
        else:
            print(f'Invalid TAG: {name_tag}')
    return list_all_values

def content_process_page_key_value(response, first_section_name='Description', only_first_section=False, paragraph_together=False):
    obj_sel = Selector(str(response))
    content = obj_sel.xpath(XPATH_CONTENT_PAGE)
    dict_all_values = {}
    section_name = first_section_name
    for tag in content:
        html_content = tag.get()
        name_tag = re.findall(r'^<([a-z\d]*)', html_content)[0]
        if name_tag[0] == 'h':
            if only_first_section:
                break
            obj_sel1 = Selector(str(html_content))
            content_h_tag = obj_sel1.xpath(XPATH_CONTENT_H)
            html_content = content_h_tag[0].get()
            section_name = get_text_html(html_content)
            
        elif name_tag[0] == 'p' or name_tag in ('ul', 'ol'):
            clear_text = re.sub(r'<sup(.*?)<\/sup>', '', html_content)
            clear_text = re.sub(r'<abbr(.*?)<\/abbr>', '', clear_text)
            clear_text = get_text_html(clear_text)
            clear_text = str(clear_text).strip()
            if clear_text:
                if section_name not in dict_all_values:
                    
                    if paragraph_together:
                        dict_all_values[section_name] = ''
                        dict_all_values[section_name] += clear_text
                    else:
                        dict_all_values[section_name] = []
                        dict_all_values[section_name].append(clear_text)
                else:
                    if paragraph_together:
                        dict_all_values[section_name] += clear_text
                    else:
                        dict_all_values[section_name].append(clear_text)
        else:
            print(f'Invalid TAG: {name_tag}')
    return dict_all_values


def verify_fields_input_required(actor_input, list_errors):
    is_lang_valid = is_value_valid = False
    if 'lang' in actor_input and 'input_value' in actor_input:
        lang_value = str(actor_input['lang']).strip().lower()
        is_lang_valid = verify_lang_valid(lang_value)
        # if not is_lang_valid:
        #     error_dict['lang'] = f'Invalid language: {lang_value}'
        list_errors.append(f'Invalid language: {lang_value}')
        input_value = actor_input['input_value']
        if type(input_value) == list or type(input_value) == str:
            is_value_valid = len(input_value) > 0
        else:
            # error_dict['input_value'] = f'type {type(input_value)} not valid should be list or str'
            list_errors.append(f'type {type(input_value)} not valid should be list or str')
            is_value_valid = False
    elif not 'lang' in actor_input:
        list_errors.append(f'key \'lang\' not found')
    elif not 'input_value' in actor_input:
        list_errors.append(f'key \'input_value\' not found')
            
    all_required = all([is_lang_valid, is_value_valid])
    return all_required

def process_input_value(actor_input):
    if type(actor_input['input_value']) == str:
        return [actor_input['input_value']]
    else:
        return actor_input['input_value']

def verify_output_type(actor_input):
    if 'output_type' in actor_input:
        output_type = str(actor_input['output_type']).strip().lower()
        if output_type == 'array':
            return 'array'
        elif output_type == 'key_value':
            return 'key_value'
        else:
            return 'key_value' # default
    else:
        return 'key_value' # default

def verify_paragraph_together(actor_input):
    if 'paragraph_together' in actor_input:
        paragraph_together = actor_input['paragraph_together']
        if str(paragraph_together).strip().lower() == 'true':
            return True
        else:
            return False # default
    else:
        return False # default

def verify_only_description(actor_input):
    if 'only_description' in actor_input:
        only_description = actor_input['only_description']
        if str(only_description).strip().lower() == 'true':
            return True
        else:
            return False # default
    else:
        return False # default
        
def verify_and_process_fields_input(actor_input, list_errors):
    valid_actor_input = {}
    is_valid_fields = False
    if verify_fields_input_required(actor_input, list_errors):
        valid_actor_input['lang'] = str(actor_input['lang']).strip().lower()
        valid_actor_input['input_value'] = process_input_value(actor_input)
        valid_actor_input['output_type'] = verify_output_type(actor_input)
        valid_actor_input['paragraph_together'] = verify_paragraph_together(actor_input)
        valid_actor_input['only_description'] = verify_only_description(actor_input)
        is_valid_fields = True
    return is_valid_fields, valid_actor_input

def process_output(response, output_type, paragraph_together=False, first_section_name='Description', only_first_section=False):
    if output_type == 'array':
        result = content_process_page_array(response, first_section_name=first_section_name, only_first_section=only_first_section)
    elif output_type == 'key_value':
        result = content_process_page_key_value(response, first_section_name=first_section_name, only_first_section=only_first_section, paragraph_together=paragraph_together)
    else:
        result = []
        logging.warning(f'Invalid output_type: {output_type}')
    return result

def process_actor_main(actor_input):
    list_result_output = []
    list_errors = []
    list_warnings = []
    status_code = 200
    is_valid_fields, valid_actor_input = verify_and_process_fields_input(actor_input, list_errors)
    if not is_valid_fields:
        logging.warning(f'Invalid fields input')
        status_code = 400
        return list_result_output, list_errors, list_warnings, status_code
    for item in valid_actor_input['input_value']:
        if verify_url_wiki(item):
            response = request_website(item)
            if not verify_page_empty(response):
                result_url = process_output(response, output_type=valid_actor_input['output_type'], paragraph_together=valid_actor_input['paragraph_together'], only_first_section=valid_actor_input['only_description'])
                dict_item_url = {
                    'input_value': item,
                    'output_value': result_url
                }
                list_result_output.append(dict_item_url)
            else:
                list_warnings.append(f'Page invalid: {item}')
                logging.warning(f'Page invalid: {item}')
        else:
            redirect_url = search_url(valid_actor_input['lang'], item)
            response = request_website(redirect_url)
            result = process_output(response, output_type=valid_actor_input['output_type'], paragraph_together=valid_actor_input['paragraph_together'], only_first_section=valid_actor_input['only_description'])
            dict_item = {
                'input_value': item,
                'output_value': result
            }
            list_result_output.append(dict_item)
    dict_json = {
        'version': __version__,
        'input': actor_input,
        'output': list_result_output,
        #'warnings': list_warnings,
        #'errors': list_errors,
        'status_code': status_code,
        'status': HTTP_CODE[status_code],
    }
    # return list_result_output, list_errors, list_warnings, status_code
    return dict_json
    dict_json = process_actor_main(actor_input)

functions.py

import json
import os

import functions

from apify_client import ApifyClient


# Run the main function of the script, if the script is executed directly
if __name__ == '__main__':
    # Initialize the main ApifyClient instance
    client = ApifyClient(os.environ['APIFY_TOKEN'], api_url=os.environ['APIFY_API_BASE_URL'])

    # Get the resource subclient for working with the default key-value store of the actor
    default_kv_store_client = client.key_value_store(os.environ['APIFY_DEFAULT_KEY_VALUE_STORE_ID'])

    # Get the value of the actor input and print it
    actor_input = default_kv_store_client.get_record(os.environ['APIFY_INPUT_KEY'])['value']
    print('Actor input:')
    print(json.dumps(actor_input, indent=2))

    # Set the 'OUTPUT' key-value store record to the same value as the input
    default_kv_store_client.set_record('OUTPUT', actor_input)

    # Get the resource subclient for working with the default dataset of the actor
    default_dataset_client = client.dataset(os.environ['APIFY_DEFAULT_DATASET_ID'])

    dict_json = functions.process_actor_main(actor_input)

    # Push some dummy items to the default dataset
    default_dataset_client.push_items(dict_json)


main.py

# Add your dependencies here.
# See https://pip.pypa.io/en/latest/cli/pip_install/#requirements-file-format
# for how to format them

apify-client==0.5.0
beautifulsoup4==4.11.1
bs4==0.0.1
certifi==2021.10.8
charset-normalizer==2.0.12
cssselect==1.1.0
html2text==2020.1.16
idna==3.3
lxml==4.8.0
parsel==1.6.0
requests==2.26.0
six==1.16.0
soupsieve==2.3.2.post1
urllib3==1.26.9
w3lib==1.22.0

requirements.txt

Wikipedia Scraper, designed and developed to make your life easier on searching for answers.

Language to Search

Search Term or Url

Output Type array or key value

Combine paragraphs

Description only

Where next?