Creative Scotland Opportunities Scraper

Scrape opportunities.creativescotland.com — Scotland's creative-sector hub for paid jobs, workshops, training, festivals, and exhibitions. The site sits behind Cloudflare's "Sorry, you have been blocked" WAF rule that hits ~90% of HTTP requests even from residential proxies. This actor solves that by rotating residential proxy exits until one gets through, then fetching the public /api/rss/ endpoint which returns every live opportunity in a single XML feed. JSON or CSV out, no compute charge per run, just per result.

How it works

✨ Why use this scraper?

Tracking Scotland's creative sector is hard — Creative Scotland's Opportunities tool is the central hub but Cloudflare's WAF makes direct scraping nearly impossible. This actor solves the proxy-rotation problem so you can focus on the data.

• 🛡️ Cloudflare WAF bypassed. No browser automation, no Camoufox, no cf_clearance cookie capture — just residential-proxy rotation. With 30 attempts (default), there's a ~96% chance one Evomi/Apify-residential exit gets through.
• ⚡ One successful fetch → 25+ opportunities. Once we crack CF, /api/rss/ returns the whole live list inline (title, link, contact email, description teaser, modified date).
• 🎨 Jobs + workshops + festivals + training in one feed. The actor auto-categorises each row by title (type: "job" vs type: "opportunity"), and tags categories: ["job"|"workshop"|"event"|"training"|"opportunity"] for downstream filtering.
• 📧 Contact email captured. Creative Scotland's RSS puts the organiser's contact email in <guid> for most items — extracted as applyEmail automatically.
• 🇬🇧 Scotland-wide. Edinburgh galleries, Highland community workshops, Glasgow theatres, rural craft schools, festival organisers.
• 📤 Clean exports. One row per opportunity. JSON + CSV exported automatically.

🎯 Use cases

Team	What they build
Creative-sector recruiters	Daily new-vacancy feeds across Scottish arts orgs
Festival / event aggregators	Auto-populate Scotland-wide creative events calendars
Craft schools / workshop platforms	Discover competitor workshop offerings
Funders & cultural strategists	Track who's hiring + what training is being offered in Scottish creative sector
Researchers	Scottish creative labour-market datasets

📥 Supported inputs

URL pattern	Behaviour
https://opportunities.creativescotland.com/api/rss/	RSS feed (default) — every live opportunity
https://opportunities.creativescotland.com/opportunity/index/<uuid>	Single opportunity — synthetic stub + optional detail fetch
(anything else on the domain)	Routed to the RSS feed

Leave startUrls empty for the RSS feed.

⚠️ Proxy requirement (mandatory)

Cloudflare blocks direct + datacenter requests instantly. A residential proxy pool is REQUIRED. Defaults to Apify Residential — you can also bring your own (Evomi, BrightData, Smartproxy, etc.) via the proxy input.

The actor rotates exits each retry (Actor.createProxyConfiguration().newProxyInfo() returns a fresh URL). With Apify Residential's pool size, you typically need 5–15 attempts to get through; we default cfMaxAttempts to 30 for ~96% success rate.

🔄 How it works

1. Resolve start URLs — default is the RSS endpoint.
2. For each URL, attempt up to cfMaxAttempts times via the proxy pool:
   • Instantiate a fresh impit with a new proxy URL
   • GET with Firefox-144 UA + realistic accept headers
   • If 200 OK AND body doesn't contain "Sorry, you have been blocked" → success
   • Otherwise, rotate and retry
3. Parse the RSS — extract title, link, opportunityId (UUID from link), guid (often contact email), description, modified date.
4. Auto-categorise each row by title keywords (officer/manager → job; workshop/class → workshop; festival/exhibition → event; course/school → training; else opportunity).
5. Push one normalised row per opportunity.
6. Optional: when enrichDetail: true, fetch each detail page via the same CF rotation (expensive — each detail call needs its own 5–30 attempts).

⚙️ Input parameters

Parameter	Type	Default	Description
startUrls	array	["https://opportunities.creativescotland.com/api/rss/"]	RSS URL, opportunity URLs, or anything on the domain.
enrichDetail	boolean	false	When true, fetches each detail page for fuller body + structured organiser/location/deadline/fee. EXPENSIVE — each detail call needs its own CF rotation.
cfMaxAttempts	integer	30	Max proxy attempts per URL. Higher = more cost + slower; lower = risks giving up on CF too early.
postedWithinHours	integer	(none)	Only return rows posted in the last N hours (24 = last day, 72 = last 3 days). Empty/0 = all. Ideal for daily monitoring runs that only want fresh postings.
maxItems	integer	1000	Hard cap on rows pushed (~25 live opportunities).
maxConcurrency	integer	3	Parallel detail-page fetch limit (matters only with enrichDetail: true).
maxRequestRetries	integer	5	Reserved — superseded by cfMaxAttempts for this actor.
proxy	object	Apify Residential	REQUIRED. Default Apify Residential; bring-your-own Evomi/BrightData/Smartproxy also works.

📊 Output overview

Each scraped opportunity is one single dataset row. Jobs get type: "job"; everything else gets type: "opportunity". The cpt-style discriminator is in categories[0] (job / workshop / event / training / opportunity).

📦 Output sample

{
  "type": "job",
  "source": "opportunities.creativescotland.com",
  "opportunityId": "86a158c5-89c5-4d2c-9a74-0a6fd30c9713",
  "jobId": "86a158c5-89c5-4d2c-9a74-0a6fd30c9713",
  "slug": "86a158c5-89c5-4d2c-9a74-0a6fd30c9713",
  "jobUrl": "https://opportunities.creativescotland.com/opportunity/index/86a158c5-89c5-4d2c-9a74-0a6fd30c9713",
  "opportunityUrl": "https://opportunities.creativescotland.com/opportunity/index/86a158c5-89c5-4d2c-9a74-0a6fd30c9713",
  "title": "Native Woodland Partnership Officer",
  "description": null,
  "descriptionText": "In this role you will lead the Trusts outreach for native woodland restoration across Scotland, developing several large scale projects.",
  "descriptionTeaser": "In this role you will lead the Trusts outreach for native woodland restoration across Scotland, developing several large scale projects.",
  "companyName": null,
  "companyWebsite": null,
  "companyDomain": "nts.org.uk",
  "location": null,
  "remote": false,
  "salary": null,
  "salaryRaw": null,
  "fee": null,
  "categories": ["job"],
  "employmentTypes": [],
  "contractType": null,
  "status": "publish",
  "postedDate": null,
  "closingDate": null,
  "modifiedDate": "2026-05-22T09:30:00.000Z",
  "applyType": "email",
  "applyUrl": "https://opportunities.creativescotland.com/opportunity/index/86a158c5-89c5-4d2c-9a74-0a6fd30c9713",
  "applyEmail": "workforus@nts.org.uk",
  "externalApplyUrl": null,
  "scrapedAt": "2026-05-22T20:01:00.000Z"
}

🗂 Key output fields

Group	Fields
Identifiers	type (job or opportunity), source, opportunityId (UUID), jobId (= opportunityId), slug, jobUrl, opportunityUrl, scrapedAt
Content	title, description (HTML, only when enrichDetail: true), descriptionText (plain), descriptionTeaser (from RSS)
Dates	modifiedDate (ISO from RSS <a10:updated>), closingDate (only with enrichDetail)
Employer / organiser	companyName (with enrichDetail), companyDomain (derived from contact email)
Categorisation	categories[] (auto-detected: job / workshop / event / training / opportunity)
Apply flow	applyType (email if contact email present, else internal), applyUrl (the opportunity page), applyEmail (from RSS <guid>)
Optional (with enrichDetail)	location, fee, full HTML description

❓ FAQ

Why does the actor sometimes take 30+ retries to fetch one URL? Cloudflare's WAF blocks roughly 90% of residential proxy exits. The actor rotates exits each retry. With 30 attempts the probability of zero successes is 0.9^30 ≈ 4%. If you see "CF rotation exhausted", bump cfMaxAttempts higher (up to 100) or use a larger residential pool.

Why is enrichDetail off by default? Each detail-page fetch needs its own CF rotation (another 5-30 proxy attempts). For 25 items that's potentially 750 proxy hits. The RSS already includes title, link, contact email, description teaser, and modified date — enough for most use cases. Enable detail fetch only when you need the full HTML body / location / deadline / fee.

Why does type sometimes say opportunity when it's clearly a workshop? type is a binary classifier (job vs not-job). The categories[] array carries the finer-grained label (workshop / event / training / opportunity). Filter on categories[0] for the specific shape.

Can I scrape private pages or authenticated content? No. Only the public /api/rss/ endpoint and public /opportunity/index/<uuid> pages.

How do I limit results? Set maxItems. With ~25 live opportunities, maxItems: 100 covers everything.

💬 Support

• For issues or feature requests, please use the Issues tab on the actor's Apify Console page.
• Author's website: https://muhamed-didovic.github.io/
• Email: muhamed.didovic@gmail.com

🛠 Additional services

• Custom output shape, additional fields, or one-off datasets: muhamed.didovic@gmail.com
• Cloudflare-WAF-bypass scrapers for other CF-protected sites in your portfolio (same residential-rotation pattern works on many): drop an email.
• For API access (no Apify fee, just usage): muhamed.didovic@gmail.com

🔎 Explore more scrapers

See other scrapers at memo23's Apify profile — covering job boards, real estate, social media, and more.

---

⚠️ Disclaimer

This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Creative Scotland, Alba Chruthachail, or any of their subsidiaries or affiliates. All trademarks mentioned are the property of their respective owners.

The scraper accesses only the publicly available RSS feed and public opportunity pages on opportunities.creativescotland.com — no authenticated endpoints, recruiter-only features, or content behind a login. Cloudflare WAF rotation is used solely to overcome unintentional IP-reputation false positives that block legitimate scraping; we do not bypass JS challenges, CAPTCHAs, or any security mechanism that requires interactive verification. Users are responsible for ensuring their use complies with opportunities.creativescotland.com's Terms of Service, applicable data-protection law (GDPR, CCPA, etc.), and any contractual obligations of their own organisation.

---

SEO Keywords

creative scotland scraper, scrape opportunities.creativescotland.com, creative scotland opportunities api, scottish creative sector jobs scraper, scottish arts jobs api, scotland workshops scraper, scottish festivals data, scottish creative jobs scraper, Apify creative scotland, cloudflare waf bypass scraper, residential proxy rotation scraper, scotland cultural sector jobs api, scottish gallery jobs scraper, scottish theatre jobs scraper, edinburgh creative jobs scraper, glasgow arts jobs scraper, asva alternative scraper, highlandjobs alternative scraper, s1jobs alternative scraper, scotland creative recruitment data