Precious Metal Price Scraper

A comprehensive web scraping actor for collecting precious metal prices from global and Indonesian sources with AI-powered trend analysis.

Description

This actor aggregates gold and metal prices from over 20 sources worldwide, including major global exchanges and Indonesian local markets. It provides real-time price data, historical trends, and AI-powered analysis for investment decision-making. The scraper covers multiple precious metals including gold, silver, platinum, and palladium, with automatic currency conversion and quality assessment.

Features

• Multi-source Data Collection: Aggregates prices from 20+ global and Indonesian sources
• AI Trend Analysis: Provides intelligent trend analysis with momentum indicators and volatility scoring
• Data Normalization: Converts all prices to standard units (oz, gram, kg) and currencies (USD, IDR)
• Quality Assessment: Evaluates data quality with ratings (complete, partial, estimated, limited)
• Error Handling: Robust error handling and logging for failed scraping attempts
• Structured Output: Well-formatted JSON output compliant with defined schema
• Real-time Processing: Near real-time price updates from multiple sources

Data Sources

Global Sources

• GoldPrice.org
• Kitco
• TradingEconomics
• Investing.com
• GoldBroker
• BullionVault
• Metals Daily
• LBMA (London Bullion Market Association)
• Nasdaq

Indonesian Local Sources

• Logam Mulia (Antam)
• Galeri24
• Pegadaian
• Tokopedia
• Indogold
• Orori
• Lakuemas
• HargaEmas.org

API Sources

• GoldAPI.io
• Metals-API.com

Input

The actor accepts JSON input with the following parameters:

{
  "metals": ["gold", "silver"],
  "sources": ["global", "local", "api"],
  "use_proxy": true
}


Input Parameters
metals (array, optional): Types of metals to collect data for. Default: ["gold", "silver"]
Options: "gold", "silver", "platinum", "palladium"
sources (array, optional): Types of data sources to use. Default: ["global", "local"]
Options: "global", "local", "api"
use_proxy (boolean, optional): Whether to use proxy for scraping. Default: true
Output
The actor produces structured JSON data with the following format:

json
{
  "metalType": "gold",
  "source": "goldprice.org",
  "sourceUrl": "https://goldprice.org/",
  "priceBid": 1975.25,
  "priceAsk": 1976.75,
  "priceMid": 1976.0,
  "currency": "USD",
  "unit": "oz",
  "pricePerGram": 63.51,
  "pricePerKilo": 63510.0,
  "priceIDR": 30628000.0,
  "pricePerGramIDR": 984400.0,
  "pricePerKiloIDR": 984400000.0,
  "changePercent": 0.25,
  "changeValue": 4.92,
  "dayHigh": 1980.5,
  "dayLow": 1970.25,
  "yearHigh": 2080.0,
  "yearLow": 1810.5,
  "trend": "uptrend (moderate)",
  "trendReason": "Moderate upward movement with 0.25% increase",
  "volatilityScore": 0.35,
  "marketTime": "2023-11-29T17:45:00.000Z",
  "scrapedAt": "2023-11-29T17:45:30.000Z",
  "region": "Global",
  "market": "International",
  "exchange": "GoldPrice.org",
  "dataQuality": "complete",
  "status": "active"
}
Output Fields
metalType: Type of metal (gold, silver, platinum, palladium)
source: Name of the data source
sourceUrl: Original source URL
priceBid/priceAsk/priceMid: Bid/ask/mid prices
currency/unit: Price currency and unit
pricePerGram/pricePerKilo: Price per gram/kilo in original currency
priceIDR/pricePerGramIDR/pricePerKiloIDR: Prices in Indonesian Rupiah
changePercent/changeValue: Percentage and absolute price change
dayHigh/dayLow/yearHigh/yearLow: Daily and yearly high/low prices
trend/trendReason: AI trend analysis and reasoning
volatilityScore: Volatility score (0-1 scale)
marketTime/scrapedAt: Market and scraping timestamps
region/market/exchange: Market and exchange information
dataQuality: Data quality rating (complete, partial, estimated, limited)
status: Data status (active, error)
Installation
Using Apify Console
Create a new actor in Apify Console
Select the "Python Crawlee with Playwright" template
Replace the main.py code with this project's code
Upload the dataset_schema.json file to define the output schema
Local Development
Install Apify CLI:
bash

Line Wrapping
# Using Homebrew
brew install apify-cli

# Or using NPM
npm -g install apify-cli
Pull the actor from Apify Console:
bash

Line Wrapping

Collapse
1
apify pull <ActorId>
Run locally:
bash

Line Wrapping

Collapse
1
apify run
Usage
Via Apify Console
Navigate to the actor page in Apify Console
Click on the "Input" tab
Enter your desired input parameters
Click "Run" to start the scraping process
View results in the "Dataset" tab
Via API
bash

Line Wrapping

curl -X POST https://api.apify.com/v2/acts/<ActorId>/runs?token=<API_TOKEN> \
-d '{
  "metals": ["gold", "silver"],
  "sources": ["global", "local"],
  "use_proxy": true
}'
Example Use Cases
Investment Research: Collect comprehensive price data for precious metals investment analysis
Market Monitoring: Monitor real-time price movements across multiple exchanges
Price Comparison: Compare prices between global markets and Indonesian local markets
Trend Analysis: Leverage AI-powered trend analysis for trading decisions
Data Integration: Feed structured data into other financial systems or dashboards
Data Views
The actor provides multiple pre-configured dataset views:

Live Prices: Real-time prices across all sources
Best Prices: Find the lowest prices for each metal
Global Markets: International spot prices (USD/oz)
Indonesia Market: Local Indonesian prices (IDR/gram)
Price Movers: Biggest price movements sorted by percentage change
By Source: Compare prices from all sources
Gold Only: Gold prices from all sources
Trend Analysis: AI-powered trend analysis with volatility scores
Detailed View: Complete data with all fields
Errors: Failed scraping attempts
Error Handling
The actor includes comprehensive error handling:

Failed scraping attempts are logged with detailed error messages
Error responses are stored in the dataset with consistent structure
Data validation ensures only properly formatted data is stored
Retry mechanisms for temporary failures
Rate Limiting
To avoid overwhelming target websites:

Requests are distributed across multiple proxy servers
Delays are implemented between requests to the same domain
Concurrent requests are limited to prevent server overload
Contributing
This project is maintained by Supernesia Creative Technology. For contributions or issues, please contact the development team.

License
This project is proprietary software of Supernesia Creative Technology. All rights reserved.

Support
For support and inquiries:

Email: info@supernesia.id
Website: https://supernesia.id