Medium Scraper

What does Medium Scraper do?

Medium Scraper collects articles from Medium: from tag pages, author profiles, publications and single article links. For every article you get the title, subtitle, author, publication, publish date, reading time, claps, responses, tags, member-only flag, cover image and link, and optionally the full text of free articles. No Medium account is needed.

Try it with the prefilled tag: open the Actor, click Start, and 20 Python articles arrive in a few seconds. Because it runs on the Apify platform, you also get API access, scheduled runs, integrations (Google Sheets, Make, Zapier, Slack, webhooks) and run monitoring.

Why scrape Medium?

• Content research. See what gets written and clapped for on any topic, which writers lead it, and which publications carry them.
• Trend tracking. Run a tag on a schedule with its /latest link to follow new articles as they appear.
• Influencer and writer discovery. Rank authors in a niche by claps and responses, and follow their output over time.
• Publication monitoring. Track everything a publication releases, its reading times and how readers respond.
• Datasets for analysis and AI. Full text of free articles, with clean metadata, ready for NLP, summarization or search indexes.

How to scrape Medium articles

1. Open a tag, author, publication or article on Medium and copy the link from your browser's address bar. Or just type a tag word, such as python or machine learning.
2. Paste it into Medium links or tags, one per line. Mix tags, authors, publications and articles freely.
3. Set Max articles per link and, if you want the article bodies, turn on Include full article text.
4. Click Start, then download the results when the run finishes.

Input

Everything is set on the Input tab. In JSON, the input looks like this:

{
    "start_urls": [
        "https://medium.com/tag/python",
        "https://medium.com/tag/javascript/latest",
        "machine learning",
        "https://medium.com/@karpathy",
        "https://medium.com/data-science-collective",
        "levelup.gitconnected.com",
        "https://medium.com/@piotr.kalanski/time-series-feature-engineering-why-positional-lags-silently-break-on-sparse-data-416c540b07c8"
    ],
    "max_articles_per_input": 50,
    "include_full_text": false
}

Field	What it does
start_urls	Medium links or tag words, one per line. Tags: https://medium.com/tag/python lists articles in Medium's recommended order; /tag/python/latest lists the newest first; a plain word such as python, machine learning or node.js is read as a tag. Authors: https://medium.com/@karpathy, karpathy.medium.com or @karpathy. Publications: https://medium.com/data-science-collective, or the publication's own domain such as levelup.gitconnected.com. Articles: any article link, including medium.com/p/<id> and links on a publication's own domain. Links work without https://, and API callers can also send {"url": "..."} objects.
max_articles_per_input	Stops after this many articles for each link or tag, in the order Medium lists them. Default: 50.
include_full_text	Adds the full text and word count of every free article. Member-only articles keep all their other details, with no text. Default: off.

Output

Each article becomes one row. Here is a real example from the run with full text on, with the text shortened:

{
    "id": "a64152b37c35",
    "url": "https://karpathy.medium.com/software-2-0-a64152b37c35",
    "title": "Software 2.0",
    "subtitle": "I sometimes see people refer to neural networks as just “another tool in your machine learning toolbox”. They have some pros and cons, they…",
    "author_name": "Andrej Karpathy",
    "author_username": "karpathy",
    "author_url": "https://medium.com/@karpathy",
    "publication_name": null,
    "publication_url": null,
    "published_at": "2017-11-11T22:18:53+00:00",
    "updated_at": "2021-03-13T18:18:43+00:00",
    "reading_time_minutes": 9,
    "claps": 61029,
    "responses": 194,
    "tags": ["machine-learning", "artificial-intelligence", "programming", "software-development", "future"],
    "is_member_only": false,
    "image": "https://miro.medium.com/v2/resize:fit:1200/1*CHcu2L0NmAZwCpQgmS1ByA.jpeg",
    "language": "en",
    "text": "I sometimes see people refer to neural networks as just “another tool in your machine learning toolbox”. They have some pros and cons, they work here or there, …",
    "word_count": 2127,
    "input": "https://medium.com/@karpathy",
    "position": 1
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. The Output tab has two views: Articles (the main columns) and Full details (every field, including the text).

Data fields

Field	Description
id	Medium's ID of the article.
url	Link to the article.
title	Article title.
subtitle	The subtitle as Medium shows it in article previews. When the author set none, Medium uses the article's opening line instead.
author_name, author_username, author_url	The writer's name, Medium username and profile link.
publication_name, publication_url	The publication the article appeared in; empty for articles published on the author's own profile.
published_at	When the article was first published, in UTC (ISO 8601).
updated_at	When the author last published a change to it, in UTC; the same as published_at if never edited.
reading_time_minutes	Reading time as Medium shows it ("9 min read").
claps	Number of claps.
responses	Number of responses (comments).
tags	The article's tags, as they appear in tag links (machine-learning).
is_member_only	true for member-only (paywalled) articles.
image	The article's cover image; empty when it has none.
language	Language Medium detected for the article, as a code such as en.
text	Full text of a free article, as plain paragraphs. Only with Include full article text; empty for member-only articles.
word_count	Number of words in text. Only with Include full article text.
input	The link or tag that led to this article.
position	The article's place in the list Medium showed for that input (1 = first).

How much does it cost to scrape Medium?

The Actor uses pay-per-event pricing: you pay per article saved, and the exact price is on the Pricing tab. The full text of free articles is included at no extra charge, and there is no proxy cost to add.

It is fast: a test run saved 1,227 articles from five tags, authors and publications in 25 seconds, and the prefilled run takes about 4 seconds. Links that return nothing are never charged as results, and if you set a maximum cost for a run, the Actor stops as soon as it is reached. Apify's free plan includes monthly credit you can use to try it.

Tips for scraping Medium

• Use /latest for fresh articles and for depth. A tag's own page lists about 500 recommended articles; https://medium.com/tag/<tag>/latest lists the tag's articles newest first and goes much further back.
• Schedule a tag's /latest link with a small maximum to monitor a topic every day.
• Turn on full text only when you need it. The rows are the same otherwise, and runs without text return smaller datasets.
• Paste several tags at once. An article found under two of your links is saved only once, under the first link that reached it, so overlapping tags don't cost twice.
• Publications list articles in the order they were added to the publication, which is usually, but not always, publish order.

FAQ, disclaimers and support

Does it unlock member-only articles? No. The Actor never bypasses Medium's paywall and uses no account or cookies. Member-only articles are saved with everything Medium shows to readers who are not logged in (title, subtitle, author, dates, reading time, claps, responses, tags and image), marked is_member_only: true, and without text.

What links are not supported? Search pages, reading lists, tag pages inside a publication, tag archives by date and Medium app share links (link.medium.com/...: open one and paste the article address from your browser instead). They are listed in the FAILED_INPUTS record with a suggestion of what to paste instead.

What happens to links that return nothing? Unknown tags, authors and publications, deleted articles, authors who have published nothing, and links that are not Medium links are not saved as rows, so you are not charged for them. Each one is listed with its reason in the FAILED_INPUTS record of the run's key-value store, and the status message gives the counts. A run is marked failed only when nothing was saved because Medium could not be reached for every input.

Do I need a Medium account or a proxy? No. The Actor reads only what Medium shows to readers who are not logged in, and chooses its own connection: it retries through Apify Proxy on its own if Medium refuses.

Is it legal to scrape Medium? The Actor collects only articles that anyone can read on Medium without an account. Articles carry authors' public names and usernames, which are personal data under rules such as the GDPR and CCPA: only collect what you need, store it securely, have a lawful reason to process it, and don't use it to spam or profile people. Article texts are protected by copyright, so don't republish them without permission. Medium's terms restrict automated access, so you are responsible for how you use the data. If you are unsure, ask a lawyer.

Found a problem or need a custom solution? Open an issue on the Issues tab and we will look into it. Custom scrapers and content data feeds are available on request.