🇮🇳 Shine.com Jobs Scraper — AI-Ready Indian Jobs

LICENSE

Extract 272,833+ job listings from India's leading job portal Shine.com
Perfect for AI agents, job aggregators, market research, and recruitment automation.

---

🎯 Features

🚀 Zero Competition Market

• First-to-market Shine.com scraper on Apify
• Access to 272K+ Indian job listings (software engineers alone!)
• Untapped data source for AI training and job aggregation

🤖 AI-Native Design

• Claude MCP Integration — Use directly in Claude Desktop
• ChatGPT Actions — Integrate via OpenAI custom GPTs
• Clean JSON Output — Ready for LLM consumption
• Structured Data — Perfect for RAG systems and embeddings

⚡ Production-Ready

• Camoufox Browser — Stealth automation with real Firefox fingerprints
• Residential Proxies — Built-in Apify proxy support
• Smart Retry Logic — 3 retries with exponential backoff
• Error Handling — Never crashes, returns null for missing data
• Pagination Support — Auto-follows to next pages

📊 Rich Data Extraction

Extract 13 data points per job:

• ✅ Job ID, Title, Company
• ✅ Location, Salary, Experience
• ✅ Skills, Industry, Posted/Expiry Dates
• ✅ Full Description (HTML cleaned)
• ✅ Direct Job URL
• ✅ Scrape Timestamp

🎛️ Flexible Filters

• Search Query — Job titles, keywords, skills
• Location — Bangalore, Mumbai, Delhi, Pune, etc.
• Experience Level — Entry, Mid, Senior levels
• Result Limits — 1 to 10,000 jobs per run

---

📊 Output Schema

Sample Job Record

{
  "jobId": "123456789",
  "jobTitle": "Senior Python Developer",
  "company": "Tech Innovations Pvt Ltd",
  "location": "Bangalore, Karnataka",
  "salary": "₹15-25 Lacs P.A.",
  "experience": "5 to 10 Yrs",
  "skills": "Python, Django, REST API, PostgreSQL, Docker",
  "industry": "IT Services & Consulting",
  "postedDate": "2024-03-15",
  "expiryDate": "2024-04-15",
  "description": "We are seeking an experienced Python Developer to join our growing team. You will be responsible for developing scalable backend services, designing RESTful APIs, and mentoring junior developers. Strong experience with Django, PostgreSQL, and cloud platforms required...",
  "url": "https://www.shine.com/job/senior-python-developer-tech-innovations-123456789",
  "scrapedAt": "2024-03-20T10:30:45.123Z"
}

Output Format

Field	Type	Description	Example
jobId	string	Unique job identifier	"123456789"
jobTitle	string	Position title	"Senior Python Developer"
company	string	Hiring company name	"Tech Innovations Pvt Ltd"
location	string	Job location (city)	"Bangalore, Karnataka"
salary	string	Salary range or 'Hidden'	"₹15-25 Lacs P.A."
experience	string	Required experience	"5 to 10 Yrs"
skills	string	Required skills/keywords	"Python, Django, REST API"
industry	string	Industry sector	"IT Services & Consulting"
postedDate	string	When job was posted	"2024-03-15"
expiryDate	string	Application deadline	"2024-04-15"
description	string	Full job description (max 5000 chars)	"We are seeking..."
url	string	Direct link to job posting	"https://www.shine.com/job/..."
scrapedAt	string	ISO timestamp of scraping	"2024-03-20T10:30:45.123Z"

---

🚀 Quick Start

1. Basic Usage

{
  "searchQuery": "software engineer",
  "location": "bangalore",
  "maxResults": 50
}

2. Advanced Search

{
  "searchQuery": "data scientist",
  "location": "mumbai",
  "experience": "3 to 7 Yrs",
  "maxResults": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

3. Bulk Extraction

{
  "searchQuery": "full stack developer",
  "location": "pune",
  "maxResults": 1000,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

---

🤖 AI Integration

🎨 Claude Desktop (MCP)

Add to your Claude Desktop configuration:

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/mcp-server-apify"],
      "env": {
        "APIFY_API_TOKEN": "your_apify_token"
      }
    }
  }
}

Use in Claude:

Find me 50 Python developer jobs in Bangalore from Shine.com

Claude will automatically:

1. Run this Apify actor with your query
2. Parse the results
3. Present formatted job listings
4. Answer follow-up questions about the data

💬 ChatGPT Actions

Create a custom GPT with this OpenAPI spec:

openapi: 3.0.0
info:
  title: Shine Jobs API
  version: 1.0.0
servers:
  - url: https://api.apify.com/v2
paths:
  /acts/YOUR_USERNAME~shine-jobs-scraper/runs:
    post:
      operationId: searchShineJobs
      summary: Search jobs on Shine.com
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                searchQuery:
                  type: string
                  description: Job title or keywords
                location:
                  type: string
                  description: City (e.g., bangalore, mumbai)
                maxResults:
                  type: integer
                  description: Number of jobs to return

Use in ChatGPT:

Search for React developer jobs in Delhi and show me the top 10 with highest salaries

🔗 Make.com / Zapier

1. Trigger: Schedule (daily/weekly)
2. Action: Run Apify Actor shine-jobs-scraper
3. Input:

   {
     "searchQuery": "{{job_title}}",
     "location": "{{city}}",
     "maxResults": 100
   }
   

4. Post-Processing:
   • Send to Google Sheets
   • Email digest
   • Post to Slack
   • Store in Airtable

🐍 Python Integration

from apify_client import ApifyClient

client = ApifyClient('your_apify_token')

# Run the actor
run = client.actor('YOUR_USERNAME/shine-jobs-scraper').call(
    run_input={
        'searchQuery': 'machine learning engineer',
        'location': 'hyderabad',
        'maxResults': 200
    }
)

# Fetch results
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(f"{item['jobTitle']} at {item['company']} - {item['salary']}")

📊 Data Analysis with Pandas

import pandas as pd
from apify_client import ApifyClient

client = ApifyClient('your_apify_token')

# Get dataset
dataset_id = 'your_dataset_id'
items = list(client.dataset(dataset_id).iterate_items())

# Convert to DataFrame
df = pd.DataFrame(items)

# Analyze salary trends
df['salary_clean'] = df['salary'].str.extract(r'₹(\d+)').astype(float)
avg_salary_by_location = df.groupby('location')['salary_clean'].mean()

print(avg_salary_by_location)

---

📖 Input Parameters

Parameter	Type	Required	Default	Description
searchQuery	string	✅ Yes	-	Job title or keywords to search for
location	string	❌ No	""	City or location (e.g., Bangalore, Mumbai)
experience	string	❌ No	""	Experience range (leave empty for all levels)
maxResults	integer	✅ Yes	50	Maximum number of jobs to scrape (1-10000)
proxyConfiguration	object	❌ No	Residential	Apify proxy settings (residential recommended)

Location Examples

• bangalore / bengaluru
• mumbai
• delhi / new delhi
• pune
• hyderabad
• chennai
• kolkata
• gurgaon / gurugram
• noida

Experience Format

• "0 to 3 Yrs" — Entry level
• "3 to 7 Yrs" — Mid level
• "7 to 15 Yrs" — Senior level
• "" (empty) — All levels

---

🛠️ Technical Details

Architecture

┌─────────────────────────────────────────────────────┐
│                  Apify Platform                     │
│  ┌───────────────────────────────────────────────┐  │
│  │         shine-jobs-scraper Actor              │  │
│  │                                               │  │
│  │  ┌─────────────┐      ┌──────────────┐       │  │
│  │  │   Python    │──────│  Camoufox    │       │  │
│  │  │   3.11      │      │   Browser    │       │  │
│  │  └─────────────┘      └──────────────┘       │  │
│  │         │                     │               │  │
│  │         ▼                     ▼               │  │
│  │  ┌─────────────────────────────────┐         │  │
│  │  │    Residential Proxy Pool       │         │  │
│  │  └─────────────────────────────────┘         │  │
│  │                 │                             │  │
│  └─────────────────│─────────────────────────────┘  │
└───────────────────│───────────────────────────────┘
                    │
                    ▼
         ┌─────────────────────┐
         │   Shine.com         │
         │   (Next.js SPA)     │
         │   __NEXT_DATA__     │
         └─────────────────────┘

Technology Stack

• Language: Python 3.11
• Framework: Apify SDK 2.0+
• Browser: Camoufox 0.4+ (stealth Firefox)
• Proxy: Residential proxies (anti-bot)
• Rendering: Full SPA support (Next.js)
• Extraction: NEXT_DATA JSON parsing

Why Camoufox?

• 🎭 Real Firefox fingerprints — Undetectable automation
• 🌍 GeoIP support — Location-based browsing
• ⚡ Faster than Playwright — Lightweight async API
• 🛡️ Anti-detection — Passes bot detection tests

Data Source

Shine.com is a Next.js SPA that embeds all job data in __NEXT_DATA__ script tags:

<script id="__NEXT_DATA__" type="application/json">
{
  "props": {
    "pageProps": {
      "initialState": {
        "jsrp": {
          "searchresult": {
            "data": {
              "results": [ /* jobs here */ ]
            }
          }
        }
      }
    }
  }
}
</script>

This allows fast, reliable extraction without XPath/CSS selectors.

---

🔍 Use Cases

🤝 Recruitment Agencies

• Aggregate jobs across multiple portals
• Auto-match candidates to openings
• Track market trends and salary benchmarks

📈 Market Research

• Analyze job market trends by location
• Track skill demand over time
• Salary benchmarking by industry

🤖 AI Training

• Build job recommendation systems
• Train resume parsers
• Create job description generators

📱 Job Aggregator Apps

• Build your own job search platform
• Combine multiple job sources
• Offer advanced filtering

💼 Corporate Intelligence

• Monitor competitor hiring
• Track industry expansion
• Identify talent shortages

---

⚙️ Configuration Examples

Entry-Level Jobs

{
  "searchQuery": "fresher software developer",
  "location": "bangalore",
  "experience": "0 to 2 Yrs",
  "maxResults": 100
}

Remote Jobs

{
  "searchQuery": "remote work from home",
  "location": "",
  "maxResults": 200
}

High-Salary Positions

{
  "searchQuery": "chief technology officer",
  "location": "mumbai",
  "experience": "15 to 25 Yrs",
  "maxResults": 50
}

Specific Skills

{
  "searchQuery": "kubernetes devops",
  "location": "pune",
  "maxResults": 150
}

---

📈 Performance

Metric	Value
Average Speed	50 jobs in ~2 minutes
Success Rate	99.5% with retries
Data Quality	13 fields per job
Proxy Support	✅ Residential required
Max Results	10,000 jobs per run
Rate Limit	Auto-handled by retries

---

🐛 Error Handling

Automatic Retries

• 3 attempts per page with exponential backoff (2s, 4s, 8s)
• Logs all failures for debugging
• Never crashes on missing data

Null Safety

All fields return null if data is unavailable:

{
  "jobId": "123",
  "jobTitle": null,
  "company": null,
  "salary": null
}

Timeout Handling

• Page load: 90 seconds max
• Extra SPA wait: 3 seconds
• Network idle detection

---

📚 FAQ

Q: Do I need my own proxies?

A: No! The actor uses Apify's built-in residential proxies. Just enable useApifyProxy in the configuration.

Q: How much does it cost?

A: Apify charges based on compute units. 50 jobs = 0.01-0.02 compute units ($0.002-$0.004).

Q: Can I scrape specific companies?

A: Yes! Use searchQuery: "company:TechCorp" to filter by company name.

Q: How often is Shine.com updated?

A: Jobs are posted/updated continuously. Run the scraper daily for fresh data.

Q: Can I integrate with my ATS?

A: Yes! Use webhooks or the Apify API to push data to your Applicant Tracking System.

Q: Is this legal?

A: Yes! The data is publicly accessible. Review Shine.com's terms of service and comply with applicable laws.

---

🔗 Related Actors

• Naukri.com Jobs Scraper — India's #1 job portal
• LinkedIn Jobs Scraper — Global professional network
• Indeed Jobs Scraper — Worldwide job aggregator
• Monster India Scraper — Another major Indian job site

---

🤝 Support

• Documentation: Apify Docs
• Community: Apify Discord
• Issues: GitHub Issues

---

📄 License

Apache 2.0 License — Free for commercial and personal use.

---

🎉 Credits

Built with ❤️ using:

• Apify SDK
• Camoufox
• Python 3.11

---

🚀 Get Started Now!

1. Try it on Apify — Free tier available
2. Read the Docs — Learn more
3. Join Discord — Get help

---