# tarea_apify_bs4_model (`teal_contest/tarea-apify-bs4-model`) Actor

- **URL**: https://apify.com/teal_contest/tarea-apify-bs4-model.md
- **Developed by:** [Carmen Cruzado](https://apify.com/teal_contest) (community)
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### mcgee.cat scraper — búsqueda difusa de palabras + tópicos frecuentes

Actor de [web scraping](https://apify.com/web-scraping) en Python que recorre las páginas de [mcgee.cat](https://www.mcgee.cat/) (home, publications, projects y fun), las convierte a texto plano y las analiza de dos formas:

- **count_word**: cuenta cuántas veces aparecen una o más palabras o frases, con búsqueda aproximada ([RapidFuzz](https://github.com/rapidfuzz/RapidFuzz)), sin distinguir mayúsculas ni acentos, e incluyendo variantes y errores de tipeo.
- **top_topics**: detecta los tópicos más frecuentes del sitio agrupando oraciones con TF-IDF + KMeans ([scikit-learn](https://scikit-learn.org/)).

Los parámetros se pasan por input, definido en el [input schema](https://docs.apify.com/platform/actors/development/input-schema). El HTML se descarga con [HTTPX2](https://github.com/pydantic/httpx2), se procesa con [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) y el resultado se guarda en un [dataset](https://docs.apify.com/sdk/python/docs/concepts/storages#working-with-datasets).

### Included features

- **[Apify SDK](https://docs.apify.com/sdk/python/)** for Python: toolkit para construir [Actors](https://apify.com/actors) y scrapers en Python
- **[Input schema](https://docs.apify.com/platform/actors/development/input-schema)**: define y valida los parámetros del actor
- **[Dataset](https://docs.apify.com/sdk/python/docs/concepts/storages#working-with-datasets)**: almacena el resultado como datos estructurados
- **[HTTPX2](https://github.com/pydantic/httpx2)**: requests HTTP asíncronos
- **[Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)**: extracción de datos de HTML
- **[RapidFuzz](https://github.com/rapidfuzz/RapidFuzz)**: similitud de cadenas para la búsqueda aproximada
- **[scikit-learn](https://scikit-learn.org/)**: TF-IDF, KMeans y silhouette score para los tópicos

### Input

| Parámetro | Tipo | Default | Descripción |
|---|---|---|---|
| `url` | string | `https://www.mcgee.cat/` | URL base del sitio |
| `full` | boolean | `false` | Incluye también el texto de cada publicación |
| `words` | array | `["Engineer"]` | Palabras o frases a buscar |
| `threshold` | integer | `85` | Similitud mínima (0-100). Búsquedas de 3 caracteres o menos son siempre exactas |
| `scorer` | string | `ratio` | `ratio`, `partial_ratio`, `token_sort_ratio`, `token_set_ratio` o `WRatio` |
| `top_topics` | integer | `3` | Cantidad de tópicos a devolver |
| `n_clusters` | integer | `0` | Clusters de KMeans. `0` = automático por silhouette |
| `n_terms` | integer | `5` | Términos clave por tópico |
| `n_examples` | integer | `3` | Oraciones de ejemplo por tópico |
| `stop_words` | string | `english` | `english` o `none` |
| `extra_stop_words` | array | `[]` | Palabras adicionales a ignorar |

Ejemplo:

```json
{
  "url": "https://www.mcgee.cat/",
  "words": ["Engineer", "zero knowledge", "privacy"],
  "threshold": 85,
  "top_topics": 3
}
```

### Output

Una fila en el dataset con tres keys:

```json
{
  "count_word": {
    "Engineer": {
      "count": 4,
      "sentences": ["I'm a engineer from Ireland...", "💚 Engineer @ Aztec"],
      "variants": {
        "engineer":  {"count": 2, "score": 100.0},
        "engineers": {"count": 1, "score": 94.1}
      }
    }
  },
  "content": {
    "1": "texto plano de la home",
    "2": "texto plano de publications",
    "3": "texto plano de projects",
    "4": "texto plano de fun"
  },
  "top_topics": [
    {
      "rank": 1,
      "size": 6,
      "share": 0.545,
      "terms": ["privacy", "solana", "engineer"],
      "sentences": ["Engineer at Solana Foundation focused on privacy..."]
    }
  ]
}
```

- `count_word`: por cada búsqueda, el total de apariciones, las oraciones que la contienen y las variantes encontradas con su puntaje.
- `content`: texto plano numerado por página, en orden de visita.
- `top_topics`: clusters ordenados por cantidad de oraciones (`size`), con su proporción (`share`), términos clave y oraciones representativas.

### How it works

1. `Actor.get_input()` obtiene los parámetros del input.
2. `get_soup()` descarga cada página con `httpx2.AsyncClient` y la carga con `BeautifulSoup(response.content, 'lxml')`.
3. `page_text()` quita nav, header y footer, y convierte el HTML en texto plano con una línea por bloque. Con `full: true`, `publication_paths()` agrega las publicaciones a la lista de páginas.
4. `normalize()`, `tokenize()` y `split_sentences()` pasan el texto a minúsculas, le quitan los acentos y lo separan en palabras y oraciones.
5. `SentenceIndex` indexa los n-gramas de cada oración, para buscar tanto palabras sueltas como frases.
6. `fuzzy_count()` compara cada búsqueda contra el vocabulario con `rapidfuzz.process.extract` y arma `count_word`.
7. `find_topics()` vectoriza las oraciones con `TfidfVectorizer` (unigramas y bigramas), las agrupa con `KMeans`, elige el número de clusters por silhouette, toma los más grandes y los describe con `topic_terms()`. Así arma `top_topics`.
8. `Actor.push_data({...})` guarda el resultado en el dataset.

### Resources

- [BeautifulSoup Scraper](https://apify.com/apify/beautifulsoup-scraper)
- [Python tutorials in Academy](https://docs.apify.com/academy/python)
- [Web scraping with Beautiful Soup and Requests](https://blog.apify.com/web-scraping-with-beautiful-soup/)
- [RapidFuzz documentation](https://rapidfuzz.github.io/RapidFuzz/)
- [scikit-learn: text feature extraction (TF-IDF)](https://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction)
- [scikit-learn: KMeans clustering](https://scikit-learn.org/stable/modules/clustering.html#k-means)
- [Integration with Make, GitHub, Zapier, Google Drive, and other apps](https://apify.com/integrations)
- [Video guide on getting scraped data using Apify API](https://www.youtube.com/watch?v=ViYYDHSBAKM)

### Getting started

Para la información completa, [ver este artículo](https://docs.apify.com/platform/actors/development#build-actor-at-apify-console). En resumen:

1. Build the Actor
2. Run the Actor

Para correrlo localmente:

```bash
pip install -r requirements.txt
apify run --input '{"url":"https://www.mcgee.cat/","words":["Engineer"]}'
```

El resultado queda en `storage/datasets/default/`.

### Pull the Actor for local development

Si querés desarrollar localmente, podés bajar el actor desde la consola de Apify con Apify CLI:

1. Instalá `apify-cli`

   **Con Homebrew**

   ```bash
   brew install apify-cli
   ```

   **Con NPM**

   ```bash
   npm -g install apify-cli
   ```

2. Bajá el actor con su `<ActorId>`, que puede ser:

   - el nombre único del actor (ej.: "apify/hello-world")
   - o su ID (ej.: "E2jjCZBezvAZnX8Rb")

   Los dos aparecen al hacer clic en el título del actor, arriba de la página.

   Este comando copia el actor al directorio actual:

   ```bash
   apify pull <ActorId>
   ```

### Documentation reference

- [Apify SDK for Python documentation](https://docs.apify.com/sdk/python)
- [Apify Platform documentation](https://docs.apify.com/platform)
- [Join our developer community on Discord](https://discord.com/invite/jyEM2PRvMU)

# Actor input Schema

## `url` (type: `string`):

URL base del sitio a scrapear.

## `full` (type: `boolean`):

Descarga también el cuerpo de cada publicación.

## `words` (type: `array`):

Una por fila. Búsqueda aproximada (rapidfuzz), sin distinguir mayúsculas ni acentos. Acepta frases de varias palabras.

## `threshold` (type: `integer`):

Puntaje mínimo (0-100) para contar una coincidencia. 100 = exacta. Las búsquedas de hasta 3 caracteres siempre son exactas.

## `scorer` (type: `string`):

Función de rapidfuzz usada para comparar.

## `top_topics` (type: `integer`):

Devuelve los N clusters con más oraciones.

## `n_clusters` (type: `integer`):

Cantidad de clusters. 0 = automático (el k con mejor silhouette entre 'Cantidad de tópicos' y √oraciones, máximo 10).

## `n_terms` (type: `integer`):

Cuántos términos de mayor peso TF-IDF describen cada tópico.

## `n_examples` (type: `integer`):

Las oraciones más cercanas al centroide de cada cluster.

## `stop_words` (type: `string`):

Lista de palabras vacías a ignorar.

## `extra_stop_words` (type: `array`):

Palabras extra a ignorar (ej.: demo, github).

## Actor input object example

```json
{
  "url": "https://www.mcgee.cat/",
  "full": false,
  "words": [
    "Engineer"
  ],
  "threshold": 85,
  "scorer": "ratio",
  "top_topics": 3,
  "n_clusters": 0,
  "n_terms": 5,
  "n_examples": 3,
  "stop_words": "english",
  "extra_stop_words": []
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://www.mcgee.cat/",
    "words": [
        "Engineer"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("teal_contest/tarea-apify-bs4-model").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://www.mcgee.cat/",
    "words": ["Engineer"],
}

# Run the Actor and wait for it to finish
run = client.actor("teal_contest/tarea-apify-bs4-model").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://www.mcgee.cat/",
  "words": [
    "Engineer"
  ]
}' |
apify call teal_contest/tarea-apify-bs4-model --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,teal_contest/tarea-apify-bs4-model"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WxUM3W2xSefWaa4TK/builds/vpJntgqAM7P2FZG1o/openapi.json