Substack Scraper avatar
Substack Scraper
Try for free

1 day trial then $20.00/month - No credit card required now

View all Actors
Substack Scraper

Substack Scraper

qpayre/substack-scraper
Try for free

1 day trial then $20.00/month - No credit card required now

The Substack Author Scraper is a powerful Apify actor that makes it easy for content creators to scrape and retrieve all posts from their favorite Substack authors. With structured data presented in a user-friendly format, analyzing and processing valuable information has never been easier.

User avatar

The subscription baed substack accounts are impossible to scrape

Closed

lusavkaedu opened this issue
12 days ago

this is one of the subsctiption based accounts I tried to scrape:

2024-04-03T09:07:38.678Z ACTOR: Pulling Docker image of build 79Jhj6dXdLhgwsEkQ from repository. 2024-04-03T09:07:41.656Z ACTOR: Creating Docker container. 2024-04-03T09:07:41.741Z ACTOR: Starting Docker container. 2024-04-03T09:07:44.037Z INFO Initializing actor... 2024-04-03T09:07:44.038Z INFO System info ({"apify_sdk_version": "1.0.0", "apify_client_version": "1.0.0", "python_version": "3.11.6", "os": "linux"}) 2024-04-03T09:07:44.126Z 2024-04-03 09:07:44 [INFO] substack_scraper.main - Scraping 10 most recent posts from asiancenturystocks, (scrape article content: True) - (main.py:20) 2024-04-03T09:07:44.128Z 2024-04-03 09:07:44 [INFO] substack_scraper.posts_scraper - Scraping posts from author asiancenturystocks using url https://asiancenturystocks.substack.com (profile: https://asiancenturystocks.substack.com) - (posts_scraper.py:32) 2024-04-03T09:07:44.235Z ERROR Actor failed with an exception 2024-04-03T09:07:44.236Z Traceback (most recent call last): 2024-04-03T09:07:44.237Z File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 971, in json 2024-04-03T09:07:44.238Z return complexjson.loads(self.text, **kwargs) 2024-04-03T09:07:44.239Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:07:44.239Z File "/usr/local/lib/python3.11/json/init.py", line 346, in loads 2024-04-03T09:07:44.240Z return _default_decoder.decode(s) 2024-04-03T09:07:44.241Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:07:44.241Z File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode 2024-04-03T09:07:44.242Z obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 2024-04-03T09:07:44.243Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:07:44.244Z File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode 2024-04-03T09:07:44.244Z raise JSONDecodeError("Expecting value", s, err.value) from None 2024-04-03T09:07:44.245Z json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2024-04-03T09:07:44.246Z 2024-04-03T09:07:44.247Z During handling of the above exception, another exception occurred: 2024-04-03T09:07:44.248Z 2024-04-03T09:07:44.248Z Traceback (most recent call last): 2024-04-03T09:07:44.249Z File "/usr/src/app/src/main.py", line 43, in main 2024-04-03T09:07:44.250Z data = await scrap(actor_input) 2024-04-03T09:07:44.251Z ^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:07:44.251Z File "/usr/src/app/src/main.py", line 23, in scrap 2024-04-03T09:07:44.252Z posts: list[Post] = PostsScraper.scrap(author, limit) 2024-04-03T09:07:44.253Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-03T09:07:44.253Z File "/usr/src/app/src/substack_scraper/posts_scraper.py", line 50, in scrap 2024-04-03T09:07:44.254Z data = response.json() 2024-04-03T09:07:44.255Z ^^^^^^^^^^^^^^^ 2024-04-03T09:07:44.255Z File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 975, in json 2024-04-03T09:07:44.256Z raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) 2024-04-03T09:07:44.257Z requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2024-04-03T09:07:44.257Z INFO Exiting actor ({"exit_code": 91})

User avatar

QPS (qpayre)

9 days ago

Hello,

It's because the input require the author of the article. In your case the author name is "fritz"

Please give it a try and let us know if that resolve your issue

Regards, Yoann

User avatar

QPS (qpayre)

7 days ago

Hello, the issue has been fixed

Developer
Maintained by Community
Actor metrics
  • 7 monthly users
  • 71.7% runs succeeded
  • 1.2 days response time
  • Created in Apr 2023
  • Modified 9 days ago