China Public Web Intelligence Extractor

Turn public Chinese web pages and text-based PDFs into structured, analysis-ready records. The Actor returns clean text, title, publication date, issuing body, Chinese/English keywords, focus-keyword counts, hashes, source metadata, optional Markdown, and an optional English AI summary.

What it is good for

• Public government policy and regulatory pages
• Company announcements and public research pages
• Text-based PDF reports and notices
• Monitoring inputs for analysts, researchers, and AI agents
• JSON/JSONL/CSV/Excel exports through the Apify dataset API

This Actor uses raw HTTP requests rather than a browser. It is fast and inexpensive, but it does not execute JavaScript and does not perform OCR.

Safety and compliance boundaries

Only submit content you are legally allowed to access and process. The Actor deliberately rejects localhost, private networks, link-local addresses, internal hostnames, credential-bearing URLs, and non-HTTP protocols. It does not log in, bypass CAPTCHAs, evade paywalls, or defeat access controls. Respect each source's terms, robots policy, copyright, privacy rights, and applicable law.

Input

• startUrls: 1-20 public HTTP(S) pages or PDFs.
• focusKeywords: optional terms whose case-insensitive occurrence counts are returned.
• topKeywords: number of automatically extracted keywords.
• includeMarkdown: include a ready-to-save Markdown field.
• generateEnglishSummary: call an OpenAI-compatible API using the encrypted BYOK fields.
• Resource limits cap returned characters, PDF pages, download bytes, and request time.

The optional API key is marked secret in the Apify input schema. It is sent only to the configured public HTTPS endpoint. When an Actor-owner SUMMARY_API_KEY secret is configured, the endpoint and model are taken from owner-controlled environment variables, preventing an input URL from receiving the owner's key.

Output

Each requested URL creates one dataset record. Successful records include content, metadata, hashes, keywords, and optional Markdown/summary. Failed records include a stable errorCode and readable errorMessage; one bad source does not fail the whole batch.

Successful results use the document-processed pay-per-event event. Invalid or failed URLs are returned without that custom charge.

Local development

py -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt
.\.venv\Scripts\python.exe -m pytest
.\.venv\Scripts\python.exe -m public_intel.local_cli examples\input.json --output output\results.jsonl

If a local transparent proxy returns RFC 2544 198.18.0.0/15 fake DNS addresses, opt in only for that local command:

$env:PUBLIC_FETCH_ALLOW_PROXY_FAKE_IPS = "1"
.\.venv\Scripts\python.exe -m public_intel.local_cli examples\input.json --output output\results.jsonl
Remove-Item Env:PUBLIC_FETCH_ALLOW_PROXY_FAKE_IPS

This override is intentionally off by default and must not be enabled in the Apify deployment.

The Apify SDK entrypoint is:

.\.venv\Scripts\python.exe -m public_intel

When run through the SDK locally, its default storage is written under storage/.

Monetization setup

In the Actor's Apify Console Publication tab, select pay per event and add document-processed as the primary custom event. Keep Apify's synthetic apify-actor-start event enabled. Start with a low price and confirm real per-run compute/storage costs before increasing it.

Known limitations

• No JavaScript rendering
• No scanned-PDF OCR
• Metadata quality depends on source markup
• Some sites block data-center IPs or raw HTTP clients
• Automatic keyword extraction is statistical and may need domain-specific tuning