Lemon8 Feeds Scraper (Python/Scrapling)

A Python implementation of the Lemon8 feeds scraper using Scrapling for advanced anti-bot bypass capabilities.

Features

• 22 Feed Categories - Scrape from For You, Beauty, Food, Fashion, and more
• 10+ Regions - Support for US, Australia, Japan, and more
• Full Post Data - Titles, content, statistics, hashtags
• Comment Extraction - All comments including replies
• Media Downloads - Save images and videos to Key-Value Store
• Anti-Bot Bypass - Uses Scrapling's StealthyFetcher with Camoufox
• Apify Compatible - Same input/output schema as Node.js version

Installation

# Install dependencies
pip install -r requirements.txt

# Install Scrapling browser dependencies
scrapling install

Usage

Run Locally

# Quick test
python local_test.py

# Run as Apify actor (requires input.json)
python main.py

Run with Docker

$docker-compose up --build

Deploy to Apify

$apify push

Input Configuration

Parameter	Type	Default	Description
category	int	required	Feed category (0-21)
region	string	"us"	Region code
limit	int	100	Max posts to extract
getDetails	bool	true	Extract full post details
detailsLimit	int	10	Max posts for detail extraction
saveImages	bool	false	Download images to KVS
saveVideos	bool	false	Download videos to KVS
proxy	object	null	Proxy configuration

Example Input

{
  "category": 2,
  "region": "us",
  "limit": 50,
  "getDetails": true,
  "detailsLimit": 10,
  "saveImages": false,
  "saveVideos": false
}

Categories

ID	Name	ID	Name
0	For You	11	Travel
1	Beauty	12	Money
2	Food	13	Career
3	Fashion	14	Art
4	Home	15	Pets
5	Wellness	16	Photography
6	Fitness	17	Gaming
7	Entertainment	18	Reading
8	Relationship	19	Cars
9	Tech	20	Outdoors
10	Parenting	21	Education

Regions

us, au, nz, jp, th, id, vn, my, sg, ca

Output Format

Same structure as the Node.js version for compatibility:

{
  "posts": [
    {
      "id": "7412987407534162437",
      "author": {
        "name": "Author Name",
        "profileUrl": "https://...",
        "profileImageUrl": "https://..."
      },
      "title": "Post Title",
      "content": "Content preview...",
      "postUrl": "https://...",
      "statistics": {
        "savedCount": "0",
        "likesCount": "6437",
        "commentsCount": "0"
      },
      "images": [...],
      "isVideo": false,
      "category": "Food",
      "categoryId": 2,
      "details": {...},
      "allComments": [...],
      "commentStats": {...}
    }
  ],
  "metadata": {
    "feedsUrl": "https://...",
    "category": "Food",
    "categoryId": 2,
    "region": "us",
    "totalScraped": 50,
    "scrollsPerformed": 15,
    "videoPostsFound": 5,
    "detailedPostsScraped": 10
  }
}

Project Structure

lemon8_feed_python/
├── .actor/
│   ├── actor.json      # Apify actor configuration
│   └── input.json      # Default input
├── src/
│   ├── __init__.py
│   ├── categories.py   # Category/region definitions
│   ├── selectors.py    # CSS selectors
│   ├── scraper.py      # Core scraping logic
│   └── media.py        # Media download utilities
├── main.py             # Apify entry point
├── local_test.py       # Local testing script
├── requirements.txt
├── Dockerfile
└── docker-compose.yml

Dependencies

• scrapling - Anti-bot web scraping library
• playwright - Browser automation
• apify - Apify SDK for Python
• aiohttp - Async HTTP client

License

MIT