Gumtree Property Scraper

An Apify-compatible scraper that extracts property rental listings from Gumtree UK, including titles, prices, descriptions, images, map coordinates, and seller phone numbers.

How It Works

The scraper uses HTTP requests with session cookies to fetch Gumtree pages, then parses the HTML with BeautifulSoup. This approach completely bypasses Gumtree's Radware bot detection (which blocks headless browsers).

Scraping Flow

1. Search: Fetches search result pages for the configured location/filters
2. Parse Tiles: Extracts listing URLs, prices, and post dates from search results
3. Listing Details: Visits each listing page to extract full data (title, attributes, description, images, coordinates)
4. Phone Numbers: If cookies are provided, fetches the phone reveal URL (?srn=true) to extract seller phone numbers
5. Storage: Pushes results to Apify Dataset and optionally to Supabase

Providing Cookies (Required for Phone Numbers)

Cookies authenticate your Gumtree session. Without cookies, the scraper still works but cannot extract phone numbers.

Step-by-Step

1. Install a cookie export extension in your browser:

   • Chrome: Cookie-Editor or EditThisCookie
   • Firefox: Cookie-Editor

2. Log into Gumtree at www.gumtree.com with your account

3. Export cookies:

   • Click the cookie extension icon
   • Click Export → Export as JSON (copies to clipboard)
   • The exported JSON is an array of cookie objects

4. Use the cookies:

   • Apify UI: Paste the JSON array into the "Gumtree Session Cookies" input field
   • Local testing: Save the JSON to cookies.json in the project root

Cookie Format

The exported JSON looks like this (array of objects):

[
  {
    "domain": ".gumtree.com",
    "name": "gt_rememberMe",
    "value": "...",
    "path": "/",
    "secure": true,
    "httpOnly": true,
    "expirationDate": 1806936367
  },
  {
    "domain": ".gumtree.com",
    "name": "gt_s",
    "value": "id:...",
    "path": "/",
    "session": true
  }
]

Important Notes

• Export ALL cookies — don't cherry-pick. The scraper needs the full set
• Session cookies expire — re-export cookies if you get bot challenges or empty phone numbers
• Key cookies: gt_rememberMe, gt_s, GTSELLERSESSIONID, conversationsToken, waap_id
• The waap_id cookie (Radware) helps avoid bot challenges

Local Testing

# Create virtual environment
python -m venv .venv
.venv\Scripts\activate  # Windows
# source .venv/bin/activate  # Linux/Mac

# Install dependencies
pip install -r requirements.txt

# Save your cookies (see above)
# Then run:
python test_local.py

Results are saved to test_output.json.

Input Parameters

Field	Type	Required	Default	Description
location	string	Yes	—	Search location (e.g. "London")
propertyCategory	string	No	property-to-rent	Category slug
minPrice	int	No	—	Minimum price filter
maxPrice	int	No	—	Maximum price filter
minBedrooms	int	No	—	Minimum bedrooms
maxBedrooms	int	No	—	Maximum bedrooms
distance	int	No	—	Search radius in miles
maxPages	int	No	6	Max search result pages
maxDaysOld	int	No	7	Skip listings older than N days
cookies	array	No	—	Session cookies (see above)
supabaseUrl	string	No	—	Supabase project URL
supabaseKey	string	No	—	Supabase service role key

Output Schema

Each listing produces a record like:

{
  "listing_id": "uuid",
  "title": "2 Bed Flat, Newton Place, E14",
  "price_per_month": 2400.0,
  "price_currency": "£",
  "period": "pm",
  "last_updated": "3 hours ago",
  "platform_name": "Gumtree",
  "listing_url": "https://www.gumtree.com/p/...",
  "address": "London",
  "property_type": "Flat",
  "bedroom": 2,
  "images": ["url1", "url2", "..."],
  "description": "...",
  "scrape_timestamp": "2026-04-05T20:06:38",
  "longitude": -0.026247,
  "latitude": 51.4924,
  "seller_type": "Private",
  "phone_number": "02037441177"
}

Architecture

src/
├── main.py          # Apify Actor entry point
├── scraper.py       # HTTP client, search pagination, listing processing
├── extractors.py    # BeautifulSoup-based HTML data extraction
├── storage.py       # Optional Supabase integration
└── utils.py         # Price parsing, date filtering, coordinate extraction

• No headless browser — uses httpx for HTTP requests, BeautifulSoup + lxml for parsing
• Completely avoids Radware/ShieldSquare bot detection
• ~2 seconds per listing (HTTP fetch + parse)