# Haystack News Scraper (`fanndev/haystack-tv-scraper`) Actor

Scrape Haystack News (haystack.tv): video headlines by category, channel, topic or city, deep video metadata with AI topic tags and view analytics, the 483-channel directory, the full category taxonomy, news polls with live vote counts, and local events -- 8 modes, no login or API key needed.

- **URL**: https://apify.com/fanndev/haystack-tv-scraper.md
- **Developed by:** [Faisal Ahdan naufal](https://apify.com/fanndev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Haystack News Scraper

Extract video news headlines, channel directories, news polls and local events from **Haystack News (haystack.tv)** — the free streaming service carrying 400+ local and world news channels.

### Why use this actor

- **No account, no login, no API key.** Everything it returns is public.
- **400+ real newsrooms in one feed** — ABC7 Los Angeles, KHOU Houston, Bloomberg, Al Jazeera, NBC News and hundreds of local US stations, all normalised into one schema.
- **Video metadata you cannot get from an RSS feed** — captions (WebVTT), thumbnails at three sizes, duration, publish time, topic tags and a popularity score.
- **AI content classification.** The detail mode returns Haystack's own per-story classification: category, content type, urgency, editorial stance, emotional intensity, broad appeal and headline magnetism — plus view counts and viewer drop-off.
- **Eight modes in one actor**, from a single-city local news feed to the complete 483-channel directory.
- **Stable JSON output** ready for a database, spreadsheet or analysis pipeline. Every row carries the same `_input` / `_source` / `_scrapedAt` / `recordType` envelope, so mixed runs split cleanly into tables.
- **Automatic retries** and honest error rows — a bad input produces a labelled error record instead of silently disappearing.

### How it works

1. Pick a **mode** and fill in the matching input list (some modes need no input at all).
2. The actor collects the data straight from Haystack's own service, retrying automatically if the service is briefly slow.
3. Every item is written to your dataset as a flat JSON row, tagged with which input produced it.
4. Download as JSON, CSV or Excel, or run it on a schedule to track headlines over time.

You do not need to manage scrapers, browsers, accounts or blocks.

### Modes

| Mode | Reads | Returns |
|---|---|---|
| `videos` | Feed targets | Video headlines from a category, channel, topic or city |
| `video-detail` | Video URLs | One video with full metadata, AI classification and view analytics |
| `search` | Search queries | Videos matching a keyword |
| `discover` | Search queries | Channels, cities and topics matching a keyword |
| `channels` | *(nothing)* | The complete channel directory (483 channels) |
| `taxonomy` | *(nothing)* | Categories, live channels, special collections and 299 cities |
| `polls` | *(nothing)* | Every news poll with live vote counts |
| `local-events` | Locations | Concerts, sports and community events in a city |

### Input

#### Videos — headlines from a category, channel, topic or city

```json
{
  "mode": "videos",
  "targets": ["news", "channel/abc7", "tag/trump", "location/houston-tx"],
  "maxItemsPerTarget": 50,
  "includeTagGroups": false,
  "maxConcurrency": 4
}
```

Targets accept a full `https://www.haystack.tv/...` URL or a short identifier, mixed freely:

| Target | Meaning |
|---|---|
| `news`, `sports`, `business`, `live`, `weather` | A category feed |
| `channel/abc7` | Everything from one channel |
| `tag/trump` | Everything on one topic |
| `location/houston-tx` | Local news for one city |

#### Other modes

```json
{ "mode": "video-detail", "videoUrls": ["https://www.haystack.tv/v/trump-love-unified-ireland-6daOFLevf"] }
{ "mode": "search",       "queries": ["ukraine", "federal reserve"] }
{ "mode": "discover",     "queries": ["houston"] }
{ "mode": "channels" }
{ "mode": "taxonomy" }
{ "mode": "polls" }
{ "mode": "local-events", "locations": ["houston-tx", "san-francisco-ca"], "deepLocalEvents": true }
```

| Field | Type | Description |
|---|---|---|
| `mode` | string | Which of the eight modes to run. Default `videos`. |
| `targets` | array | `videos` mode — categories, channels, topics or cities. |
| `videoUrls` | array | `video-detail` mode — video page URLs, or the bare slug or id. |
| `queries` | array | `search` and `discover` modes — one keyword or phrase per line. |
| `locations` | array | `local-events` mode — city slugs such as `houston-tx`. |
| `maxItemsPerTarget` | integer | Caps videos returned per target or query. Default 50. |
| `includeTagGroups` | boolean | `videos` mode — also return the trending-topic counts for each feed. Default `false`. |
| `deepLocalEvents` | boolean | `local-events` mode — collect far more events per city at a higher request cost. Default `true`. |
| `maxConcurrency` | integer | How many targets to process in parallel. Default 4. |
| `proxyConfiguration` | object | Optional. Not required — see **Notes and limits**. |

### Output

#### `videos` — one row per headline

```json
{
  "_input": "channel/abc7",
  "_source": "S1-playlist-channel",
  "_scrapedAt": "2026-09-14T11:43:18Z",
  "recordType": "VIDEO",
  "videoId": "MaE86JEtf",
  "webUrl": "https://www.haystack.tv/v/lapd-issues-citations-impounds-dozens-minibikes-massive-6th-street-bridge-takeover-bust-MaE86JEtf",
  "feedChannel": "abc7",
  "feedTitle": "ABC7 Los Angeles",
  "title": "LAPD issues citations, impounds dozens of minibikes in massive 6th Street Bridge takeover bust",
  "author": "abc7",
  "publishedDate": "2026-09-14T06:48:41.000Z",
  "ageLabel": "5h ago",
  "duration": 260.227,
  "lang": "en",
  "categories": ["news"],
  "topics": [{ "tag": "lapd" }],
  "popularityScore": 0.4525687061312156,
  "cc": [
    { "label": "en", "lang": "en", "type": "vtt", "url": "https://d2ufudlfb4rsg4.cloudfront.net/abc7/MaE86JEtf/adaptive/MaE86JEtf_subs0.vtt" }
  ],
  "snapshotHighUrl": "https://img.haystack.tv/v1/480x270/haystack-thumbnails/abc7/MaE86JEtf/MaE86JEtf_720.jpg",
  "mediaFiles": { "adaptive": { "url": "https://d2ufudlfb4rsg4.cloudfront.net/abc7/MaE86JEtf/adaptive/MaE86JEtf_master.m3u8", "type": "m3u8" } },
  "channel": { "name": "abc7", "title": "ABC7 Los Angeles", "tag": "abc7" }
}
```

| Field | Type | Description |
|---|---|---|
| `videoId` | string | Haystack's short video id. |
| `webUrl` | string | Permalink to watch the story. |
| `title` | string | The headline. |
| `author` | string | Publishing channel, e.g. `abc7`, `bloomberg`. |
| `publishedDate` | string | Publish time, ISO 8601 UTC. |
| `ageLabel` | string | Human-readable freshness, e.g. `5h ago`. |
| `duration` | number | Clip length in seconds. |
| `categories` | array | Every category the clip belongs to. |
| `topics` | array | Topic tags attached to the clip. |
| `popularityScore` | number | Haystack's popularity ranking for the clip. |
| `cc` | array | Caption tracks — a WebVTT file per language. |
| `snapshotUrl` / `snapshotHighUrl` / `snapshot720Url` | string | Thumbnails at 240px, 480px and 720px wide. |
| `mediaFiles` | object | Video stream manifests. |
| `feedTitle` | string | Which feed the clip was collected from. |

#### `video-detail` — 51 fields per story

Everything above, plus:

```json
{
  "recordType": "VIDEO_DETAIL",
  "videoId": "6daOFLevf",
  "title": "Trump Says He'd Love to See 'Unified' Ireland",
  "author": "bloomberg",
  "views": 1942,
  "exit_ratio": 0.0394,
  "haystackTags": ["trump", "ireland", "... 3 more"],
  "downloadUrl": { "hd1080": "https://bbgvod-azure-us-east1-zenko.akamaized.net/vod/vod/m/.../34981474-1476-4249-a1f2-42a3efb3ef1a.mp4" },
  "llm": {
    "primaryCategory": "international_relations",
    "secondaryCategory": "politics",
    "contentType": "interview",
    "scope": "international",
    "urgency": "standard",
    "editorialStance": "analysis",
    "primaryUserNeed": "give-me-perspective",
    "primaryEmotion": "curiosity",
    "isWeatherAlert": false,
    "emotionalIntensity": 0.4,
    "broadAppeal": 0.65,
    "headlineMagnetism": 0.8,
    "narrativeResolution": 0.3
  }
}
```

| Field | Type | Description |
|---|---|---|
| `llm` | object | Haystack's own AI classification of the story — category, content type, scope, urgency, editorial stance, reader need, emotion, and scores for emotional intensity, broad appeal, headline magnetism and narrative resolution. |
| `views` | number | View count. |
| `exit_ratio` | number | Share of viewers who left during the clip. |
| `haystackTags` | array | Internal editorial tags. |
| `sourceUrl` | string | The publisher's own page for the story. |
| `downloadUrl` | object | Direct video file, when the publisher provides one. |

#### `channels` — the directory

```json
{
  "recordType": "CHANNEL",
  "webUrl": "https://www.haystack.tv/channel/wpvi",
  "title": " 6abc Philadelphia",
  "short_title": "ABC6 Philadelphia",
  "subtitle": "Philly",
  "name": "wpvi",
  "tag": "wpvi",
  "slug": "wpvi",
  "hashtags": ["wpvi"],
  "thumbnail": "https://img.haystack.tv/v1/512x512/haystack-channel-thumbnails/large/wpvi_512x512.jpg"
}
```

#### `polls` — questions with live vote counts

```json
{
  "recordType": "POLL",
  "pollId": "6aa497a8066a8911b61c07f7",
  "webUrl": "https://www.haystack.tv/poll/6aa497a8066a8911b61c07f7",
  "question": "Should the Fed raise interest rates to cool inflation?",
  "choices": ["Yes", "No", "Not sure"],
  "choiceVoteCount": [638, 414, 136],
  "totalVoteCount": 1188,
  "results": [
    { "choice": "Yes", "votes": 638, "pct": 53.7 },
    { "choice": "No", "votes": 414, "pct": 34.85 },
    { "choice": "Not sure", "votes": 136, "pct": 11.45 }
  ],
  "startDate": "2026-09-12T00:06:07.900Z",
  "endDate": "2026-09-16T00:06:07.000Z"
}
```

`results` is computed for you so you do not have to zip the choices and counts yourself.

#### `local-events` — what is on in a city

```json
{
  "recordType": "LOCAL_EVENT",
  "_input": "houston-tx",
  "locationSlug": "houston-tx",
  "locationCity": "Houston",
  "locationRegion": "TX",
  "id": "Bk9EMyZCk6s7oa2ZGF",
  "title": "Artemas",
  "description": "Artemas headlines a community music night at White Oak Music Hall - Downstairs in Houston, Texas, delivering an intimate live performance...",
  "labels": [
    { "tag": "concert", "name": "General Concerts", "shortName": "Concert", "category": "music_concerts" }
  ],
  "location": {
    "lat": 29.76043, "lon": -95.3698, "venue": "",
    "address": "Bagby Street, Houston, TX 77002, United States of America",
    "city": "Houston", "state": "Texas", "country": "US", "postalCode": "77002"
  },
  "startDate": "2026-09-15T23:00:00Z",
  "endDate": "2026-09-17T02:00:00Z",
  "timezone": "America/Chicago",
  "local_rank": 55
}
```

Each city also gets one summary row telling you exactly how complete the harvest was:

```json
{
  "recordType": "LOCAL_EVENTS_SUMMARY",
  "locationSlug": "houston-tx",
  "city": "Houston",
  "eventsHarvested": 107,
  "eventsTotalUpstream": 700,
  "eventsHiddenByPaywall": 690,
  "slicesTried": 19,
  "slicesFailed": 0,
  "_warning": "premium_capped_partial_harvest"
}
```

#### Error rows

Any input that cannot be resolved produces a labelled row rather than vanishing:

```json
{
  "_input": "channel/notarealchannel12345",
  "_source": "none",
  "recordType": "ERROR",
  "_error": "empty_feed",
  "_errorDetail": "kind=channel value='notarealchannel12345' returned 0 streams."
}
```

Filter on `recordType == "ERROR"` to see which inputs failed and why.

### Notes and limits

- **Feeds are a live snapshot, not an archive.** Haystack serves roughly **15–30 videos per feed** and offers no way to page further back, so `maxItemsPerTarget` above ~30 has no extra effect. To get volume, scrape **more targets** (use `channels` or `taxonomy` mode to get the full list of 483 channels and 299 cities), and re-run on a schedule to build history over time.
- **Local events are capped for free visitors.** Haystack shows anonymous visitors 10 events per request out of several hundred. With `deepLocalEvents` on, the actor issues 19 differently-filtered requests per city and de-duplicates, which typically lifts a city from 10 to **about 100 events** — verified 107 for Houston and 99 for San Francisco. The summary row always reports `eventsTotalUpstream` and `eventsHiddenByPaywall` so you know what was left behind. Turn `deepLocalEvents` off for a fast 10-event sample.
- **Search always returns results.** Haystack's search is fuzzy and never reports "no matches" — a nonsense query still returns ~20 loosely-related clips. Each query therefore gets a `SEARCH_SUMMARY` row with `termMatchRatio`, and a `_warning` of `low_relevance_no_literal_term_match` when none of the returned clips mention your terms.
- **Proxy is optional.** The actor runs fine with no proxy at all, and that is the default. Enable a US proxy only if you want the personalised *My Headlines* feed geo-located to the United States.
- **Freshness.** Category and channel feeds update continuously through the day; polls run for several days; local events look ahead weeks. Scheduling a run every few hours is a reasonable cadence for headline tracking.

# Actor input Schema

## `mode` (type: `string`):

What to scrape. Each mode reads ONE of the input lists below. TARGET modes: videos (reads "Feed targets"), video-detail (reads "Video URLs"). QUERY modes: search, discover (both read "Search queries"). CATALOGUE modes need no input at all: channels, taxonomy, polls. LOCATION mode: local-events (reads "Locations").

## `targets` (type: `array`):

Used by the VIDEOS mode. A full haystack.tv URL or a bare identifier, mixed freely: a category (news, sports, live, weather, business), a channel (channel/abc7), a topic (tag/trump), or a city (location/houston-tx). A bare word with no prefix is read as a category.

## `videoUrls` (type: `array`):

Used by the VIDEO-DETAIL mode. A video page URL (https://www.haystack.tv/v/<slug>), a short id URL (https://www.haystack.tv/id/<id>), or the bare slug or id.

## `queries` (type: `array`):

Used by the SEARCH mode (finds videos) and the DISCOVER mode (finds channels, cities and topics). One keyword or phrase per line.

## `locations` (type: `array`):

Used by the LOCAL-EVENTS mode. A city slug (houston-tx, new-york-city-ny, san-francisco-ca) or a full https://www.haystack.tv/location/<slug> URL.

## `maxItemsPerTarget` (type: `integer`):

Caps how many videos are pushed per feed target or search query. Haystack serves a fixed snapshot of roughly 15-30 videos per feed and offers no deeper pagination, so raising this above ~30 has no effect -- scrape more targets instead.

## `includeTagGroups` (type: `boolean`):

VIDEOS mode only. Also emit one TAG\_GROUP row per trending topic attached to each feed, with how many videos currently carry that topic.

## `deepLocalEvents` (type: `boolean`):

LOCAL-EVENTS mode only. Haystack caps anonymous visitors at 10 events per query. Deep mode issues 18 differently-filtered queries per city and de-duplicates, which raises a typical city from 10 to roughly 60-70 events at 18x the request cost. Turn off for a fast 10-event sample.

## `maxConcurrency` (type: `integer`):

How many targets to process in parallel. Haystack has no anti-bot layer, but keep this modest to stay a polite client.

## `proxyConfiguration` (type: `object`):

Optional. haystack.tv has no anti-bot protection and the actor runs fine with no proxy at all. Use a US residential or datacenter proxy if you want the personalised My Headlines feed (the bare / target) to be geo-located to the US.

## Actor input object example

```json
{
  "mode": "videos",
  "targets": [
    "news",
    "channel/abc7",
    "tag/trump",
    "location/houston-tx"
  ],
  "videoUrls": [],
  "queries": [],
  "locations": [],
  "maxItemsPerTarget": 50,
  "includeTagGroups": false,
  "deepLocalEvents": true,
  "maxConcurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Every video, channel, category, topic, poll and local-event record produced by this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "videos",
    "targets": [
        "news",
        "channel/abc7",
        "tag/trump",
        "location/houston-tx"
    ],
    "videoUrls": [],
    "queries": [],
    "locations": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("fanndev/haystack-tv-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "videos",
    "targets": [
        "news",
        "channel/abc7",
        "tag/trump",
        "location/houston-tx",
    ],
    "videoUrls": [],
    "queries": [],
    "locations": [],
}

# Run the Actor and wait for it to finish
run = client.actor("fanndev/haystack-tv-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "videos",
  "targets": [
    "news",
    "channel/abc7",
    "tag/trump",
    "location/houston-tx"
  ],
  "videoUrls": [],
  "queries": [],
  "locations": []
}' |
apify call fanndev/haystack-tv-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fanndev/haystack-tv-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VvDOdPqaOmsLcX39u/builds/jDqYMiezUCiMIfHXf/openapi.json