Competitor keywords extractor and content Analyzer

Description

Crawl a competitor's website and get its most-used word phrases and topics. Moreover, find phrases related to given seed keywords.
This tool scrapes a website and extract the content terminology: keywords, words, ngrams and terms related to given seed keywords. This is useful to summarize the content of a website and identify the main topics covered and can be used for SEO keyword research, analyzing competitor content and find new ideas and trends.
The tool outputs a list of the most frequent keywords or phrases found with their frequencies. Unlike other tools, you don't need an account on any specific platform.

Main features

• Scrape a given website at a specific depth (extra domain links are ignored)
• Clean and process HTML and plain text pages
• Extract the most frequent words (single word terms) and most frequent ngrams (multiple word terms made of 2 to 4 words / bigrams, trigrams and quadrigrams)
• Extract keywords from the HTML metadata
• Identify terms similar to given seed keywords
• Merge all the extracted data, by language, for a global website analysis.
• Extract social media links and emails
• Output results in CSV/JSON formats and SVG wordcloud images

Supported formats

• HTML
• Plain text

Language identification

• The language is identified for each scraped page.
• The identified language is affected to the terms extracted from the page.
• Language stopwords (the most common words, short function words, such as the, is, at, which, etc for english) are used to filter the final term list.
• Stopwords are discarded in words, and forbidden as first or last word af an ngram.

Supported languages

French, English, German, Spanish, Italian, Portuguese

Examples of use cases

• Competitor Website Content Gap Analysis
• Extract Most Frequent Phrases From Any Website (SEO Audit)
• Competitor Marketing Site Terminology Report
• Blog Topic Ideas From Competitor Websites

Input

• The main input of the tool is a starting url for the website to process.
• A set of seed keywords. If provided, all terms (metadata keywords, ngrams or words) similar to one of the seed keywords will be identified and grouped together in a separate category (Seed Related).

Output

The result of analysis is:

• A dataset with the most frequent extracted terms. The data includes keywords, seed related terms, ngrams and words. For each term: value, frequency, language, type and seed keyword are given. The dataset can be found in Output and storage (terms.json)

Terms table view with seed keywords

• The scraped pages list is provided in JSON format. Each page is described by: url, title, description, author, date, keywords and language. The file can be found in storage (pages.json)
• The emails and social media links are provided in JSON format. The file can be found in storage (contact.json)
• A global file combining all the output (terms, contact and pages) can be found in storage (all.json)

• The extracted terms can be represented as wordcloud SVG images. The images can be found in storage (wordcloud..svg)

Wordcloud representation of the seed related terms

Seed keywords: web and scraper

WordCloud representation of the general terms (ngrams and words)

Input and Output samples

Extract Most Frequent terms or phrases From a Website

Input

{
  "start_url": "https://www.apify.com/"
  "max_depth": 1,
  "max_pages": 20,
  "clean_text": true,
  "max_display_number": 10
}

Output

[
    { "term": "tools for AI", "freq": 20, "type": "Keyword", "language": "English" },
    { "term": "AI agents", "freq": 20, "type": "Keyword", "language": "English" },
    { "term": "web scraper", "freq": 20, "type": "Keyword", "language": "English" },
    { "term": "web crawler", "freq": 20, "type": "Keyword", "language": "English" },
    { "term": "scraping", "freq": 20, "type": "Keyword", "language": "English" },
    { "term": "data extraction", "freq": 20, "type": "Keyword", "language": "English" },
    { "term": "API", "freq": 20, "type": "Keyword", "language": "English" },
    { "term": "apify actors", "freq": "39", "type": "NGram", "language": "English" },
    { "term": "web scraping", "freq": "39", "type": "NGram", "language": "English" },
    { "term": "web data", "freq": "38", "type": "NGram", "language": "English" },
    { "term": "ai agents", "freq": "36", "type": "NGram", "language": "English" },
    { "term": "data extraction", "freq": "26", "type": "NGram", "language": "English" },
    { "term": "web scraper", "freq": "25", "type": "NGram", "language": "English" },
    { "term": "tools for ai", "freq": "23", "type": "NGram", "language": "English" },
    { "term": "apify store", "freq": "21", "type": "NGram", "language": "English" },
    { "term": "web crawler", "freq": "20", "type": "NGram", "language": "English" },
    { "term": "real time", "freq": "18", "type": "NGram", "language": "English" },
    { "term": "apify", "freq": "268", "type": "Word", "language": "English" },
    { "term": "data", "freq": "240", "type": "Word", "language": "English" },
    { "term": "web", "freq": "166", "type": "Word", "language": "English" },
    { "term": "ai", "freq": "146", "type": "Word", "language": "English" },
    { "term": "scraper", "freq": "130", "type": "Word", "language": "English" },
    { "term": "actors", "freq": "91", "type": "Word", "language": "English" },
    { "term": "scraping", "freq": "87", "type": "Word", "language": "English" },
    { "term": "tools", "freq": "72", "type": "Word", "language": "English" },
    { "term": "agents", "freq": "59", "type": "Word", "language": "English" },
    { "term": "api", "freq": "59", "type": "Word", "language": "English" }
]

Find terms related to given seed keywords

Input

{
  "start_url": "https://www.apify.com/"
  "max_depth": 1,
  "max_pages": 20,
  "clean_text": true,
  "max_display_number": 10,
  "seed_keywords": [ "AI" ]
}

Output (excerpt)

[
    { "term": "tools for AI", "freq": "23", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "AI agents", "freq": "36", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "ai agent", "freq": "12", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "ai workflows", "freq": "6", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "ai models", "freq": "5", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "ai powered", "freq": "4", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "power your ai", "freq": "4", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "tools or ai", "freq": "4", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "tools or ai workflows", "freq": "4", "type": "Seed Related", "language": "English", "seed": "AI" },
    { "term": "ai agent can fetch", "freq": "3", "type": "Seed Related", "language": "English", "seed": "AI" }    
]

Find Blog Topic Ideas From Competitor Websites

Input

{
  "start_url": "https://blog.apify.com/",
  "max_depth": 1,
  "max_pages": 50,
  "clean_text": true,
  "max_display_number": 100,
}

Output (excerpt)

[
  { "term": "ai agents", "freq": "73", "type": "NGram", "language": "English" },
  { "term": "tool comparisons", "freq": "52", "type": "NGram", "language": "English" },
  { "term": "case studies", "freq": "49", "type": "NGram", "language": "English" },
  { "term": "developer community", "freq": "49", "type": "NGram", "language": "English" },
  { "term": "lead generation", "freq": "44", "type": "NGram", "language": "English" },
  { "term": "marketing and lead generation", "freq": "43", "type": "NGram", "language": "English" },
  { "term": "social media", "freq": "36", "type": "NGram", "language": "English" },
  { "term": "ecommerce and retail", "freq": "32", "type": "NGram", "language": "English" },
  { "term": "social media intelligence", "freq": "30", "type": "NGram", "language": "English" },
  { "term": "market research", "freq": "25", "type": "NGram", "language": "English" },
  { "term": "real estate", "freq": "24", "type": "NGram", "language": "English" },
  { "term": "web automation", "freq": "23", "type": "NGram", "language": "English" },
  { "term": "partner stories", "freq": "22", "type": "NGram", "language": "English" },
  { "term": "data processing and analysis", "freq": "20", "type": "NGram", "language": "English" },
  { "term": "job listings", "freq": "12", "type": "NGram", "language": "English" },
  { "term": "osint and public interest", "freq": "12", "type": "NGram", "language": "English" },
  { "term": "data quality", "freq": "10", "type": "NGram", "language": "English" },
  { "term": "ecommerce and retail pricing", "freq": "10", "type": "NGram", "language": "English" },
  { "term": "mcp server", "freq": "10", "type": "NGram", "language": "English" },
  { "term": "retail pricing strategy", "freq": "10", "type": "NGram", "language": "English" },
  { "term": "travel and hospitality", "freq": "10", "type": "NGram", "language": "English" }
]

Your feedback

If you’ve got any technical feedback, a bug to report or any suggestion to improve the actor usage, please create an issue on the Actor’s Issues tab.