Sendai Documentation Generator Actor - Updated

This Apify Actor crawls a website (defaulting to Sendai API documentation), processes the content and generates comprehensive markdown documentation answering a user's question using OpenRouter's API to access various LLMs. The Actor now accepts the URL of any website that exposes a sitemap.xml for crawling.

Features

• Crawls any website: Using a provided root URL and sitemap.xml, the Actor scrapes web content.
• Documentation Generation: Extracts content relevant to the user's question.
• OpenRouter Integration: Uses OpenRouter to access state-of-the-art LLMs to generate detailed markdown documentation. Leverages models with large context windows for complex queries.
• Flexible Models: Supports multiple LLMs via OpenRouter, allowing users to select the best model for their needs.
• Contextual Responses: Optionally includes a user context and previous AI response to refine results in iterative runs.
• Comprehensive Output: Generates well-structured markdown with tables, code examples, and lists.
• Persistence: Stores generated documentation in the Actor's default dataset and Key-Value store.

Input Schema

The Actor accepts the following input:

1{
2  "rootWebsite": "https://docs.sendai.fun",
3  "question": "What is the Sendai API and how do I use it?",
4  "userContext": "A developer familiar with JavaScript and React who wants to build a simple integration with the Sendai API.",
5  "previousResponse": "",
6  "model": "google/gemini-2.0-pro-exp-02-05:free",
7  "preview": false
8}

The input schema is defined in the actor and will be displayed in the Apify Console:

1{
2  "title": "Sendai Documentation Query",
3  "type": "object",
4  "schemaVersion": 1,
5  "properties": {
6    "rootWebsite": {
7      "title": "Root Website URL",
8      "type": "string",
9      "description": "The root website URL to crawl (will look for sitemap.xml at this URL)",
10      "editor": "textfield",
11      "default": "https://docs.sendai.fun"
12    },
13    "question": {
14      "title": "Question",
15      "type": "string",
16      "description": "The question you want to ask about the Sendai API documentation",
17      "editor": "textfield"
18    },
19    "userContext": {
20      "title": "User Context",
21      "type": "string",
22      "description": "Profile of the user (e.g., 'a developer familiar with JavaScript and React')",
23      "editor": "textfield",
24      "default": ""
25    },
26    "previousResponse": {
27      "title": "Previous Response",
28      "type": "string",
29      "description": "The previous response from the AI model (for iterative refinement)",
30      "editor": "textarea",
31      "default": ""
32    },
33    "model": {
34      "title": "Model",
35      "type": "string",
36      "description": "The OpenRouter model to use for answering the question",
37      "editor": "select",
38      "default": "google/gemini-2.0-pro-exp-02-05:free",
39      "enum": [
40        "anthropic/claude-3-opus",
41        "anthropic/claude-3-sonnet",
42        "anthropic/claude-3-haiku",
43        "openai/gpt-4-turbo",
44        "openai/gpt-4o",
45        "openai/gpt-3.5-turbo",
46        "google/gemini-2.0-pro-exp-02-05:free",
47        "meta-llama/llama-3-70b-instruct"
48      ],
49      "enumTitles": [
50        "Claude 3 Opus",
51        "Claude 3 Sonnet",
52        "Claude 3 Haiku",
53        "GPT-4 Turbo",
54        "GPT-4o",
55        "GPT-3.5 Turbo",
56        "Gemini Pro",
57        "Llama 3 70B"
58      ]
59    },
60    "preview": {
61      "title": "Preview",
62      "type": "boolean",
63      "description": "Whether this is a preview run (will not save documentation to file)",
64      "default": false
65    }
66  },
67  "required": ["question"]
68}

• rootWebsite (optional): The root URL of the website to crawl. The Actor will attempt to locate the sitemap.xml file at /sitemap.xml of this URL. Defaults to https://docs.sendai.fun.
• question (required): The topic or question you want documentation for about the specified API.
• userContext (optional): A description of the user asking the question. This helps the model generate more relevant and tailored responses. Example: "A beginner programmer learning JavaScript". Defaults to "".
• previousResponse (optional): The previous response from the AI model. Use this to refine the documentation in iterative runs. Defaults to "".
• model (optional): The OpenRouter model to use for generating the documentation. Default is "google/gemini-2.0-pro-exp-02-05:free" (2M token context window), which is recommended for its large context window. Available options include:
  • "google/gemini-2.0-pro-exp-02-05:free" (Recommended: Large Context Window)
  • "anthropic/claude-3-opus"
  • "anthropic/claude-3-sonnet"
  • "anthropic/claude-3-haiku"
  • "openai/gpt-4-turbo"
  • "openai/gpt-4o"
  • "openai/gpt-3.5-turbo"
  • "meta-llama/llama-3-70b-instruct"
• preview (optional): Whether this is a preview run. If set to true, the documentation will not be saved to a file when the webhook is triggered. Default is false.

Output

The Actor outputs comprehensive markdown documentation about the requested topic in the default dataset and Key-Value store. The documentation includes:

• A table of contents
• Well-structured sections with proper markdown formatting
• Code examples where appropriate
• Detailed explanations of the requested topic
• Considerations for specific user profiles included in userContext

You can access the documentation using the Apify API or the Apify Console.

Environment Variables

The Actor requires the following environment variables:

• OPENROUTER_API_KEY: Your OpenRouter API key. Get this from OpenRouter's website.

Local Development

1. Install dependencies:

npm install

2. Run the Actor locally:

OPENROUTER_API_KEY=your_api_key node main.js

You can also specify a custom input with the question:

OPENROUTER_API_KEY=your_api_key node -e "require('apify').main(async () => { await require('./main').default({ question: 'What is Sendai?', model: 'google/gemini-2.0-pro-exp-02-05:free' }); })"

The documentation will be generated as a string in the console.

Deployment to Apify

1. Install the Apify CLI:

npm install -g apify-cli

2. Log in to your Apify account:

apify login

3. Deploy the Actor:

apify push

4. Set the OPENROUTER_API_KEY environment variable in the Apify Console, under the "Env Vars" tab on in the Actor's settings.

5. Run the Actor with your question to generate comprehensive markdown documentation.

Sample Documentation Output

The generated documentation is:

1. Stored in the Actor's default dataset and Key-Value store.
2. Saved as a markdown file in the docs directory (when running via webhook, if preview is false).
3. Formatted with proper markdown syntax including:
   • Headers and subheaders
   • Code blocks
   • Tables
   • Lists
   • Links
   • Other markdown formatting as needed

This makes it easy to integrate the documentation into existing documentation systems or to publish it directly on platforms that support markdown.

Example Output: Assuming the question is "How do I authenticate with the Sendai API using JWTs?"

1# Authenticating with the Sendai API using JWTs
2
3## Table of Contents
4
5- [Introduction](#introduction)
6- [Prerequisites](#prerequisites)
7- [Obtaining an API Key](#obtaining-an-api-key)
8- [Generating a JWT Token](#generating-a-jwt-token)
9- [Using the JWT Token in API Requests](#using-the-jwt-token-in-api-requests)
10- [Example Code (JavaScript)](#example-code-javascript)
11- [Token Expiration and Refresh](#token-expiration-and-refresh)
12- [Security Considerations](#security-considerations)
13- [Troubleshooting](#troubleshooting)
14- [Conclusion](#conclusion)
15
16## Introduction
17
18The Sendai API uses JSON Web Tokens (JWTs) for authentication. JWTs are a standard method for securely representing claims between two parties. This documentation will guide you through the process of obtaining an API key, generating a JWT token, and using it to authenticate your requests to the Sendai API.
19
20## Prerequisites
21
22Before you begin, you'll need:
23
24- An account with Sendai.
25- Node.js and npm installed (if you plan to use the JavaScript example).
26- A JWT library for your preferred programming language (e.g., `jsonwebtoken` for JavaScript).
27
28## Obtaining an API Key
29
301.  Log in to your Sendai account.
312.  Navigate to the API Keys section in your profile settings.
323.  Generate a new API key.  Ensure you store this key securely, as it will be used to sign your JWTs.
33
34## Generating a JWT Token
35
36To generate a JWT token, you'll need to include the following claims:
37
38-   `iss`:  The issuer of the token (your Sendai user ID).
39-   `sub`: The subject of the token (typically your Sendai application ID).
40-   `iat`: The "issued at" timestamp (in seconds since epoch).
41-   `exp`: The expiration timestamp (in seconds since epoch).  Example: `iat + (60 * 60)` for one hour.
42
43Here's how you can generate a JWT token:
44
45```javascript
46const jwt = require('jsonwebtoken');
47
48const apiKey = 'YOUR_SENDAI_API_KEY';
49const userId = 'YOUR_SENDAI_USER_ID';
50const applicationId = 'YOUR_SENDAI_APPLICATION_ID';
51
52const payload = {
53  iss: userId,
54  sub: applicationId,
55  iat: Math.floor(Date.now() / 1000),
56  exp: Math.floor(Date.now() / 1000) + (60 * 60) // Token expires in 1 hour
57};
58
59const token = jwt.sign(payload, apiKey, { algorithm: 'HS256' });
60
61console.log(token);

Using the JWT Token in API Requests

To use the JWT token, include it in the Authorization header of your API requests as a Bearer token:

Authorization: Bearer <YOUR_JWT_TOKEN>

Example using curl:

curl -H "Authorization: Bearer YOUR_JWT_TOKEN" https://api.sendai.fun/v0/endpoint

... (Rest of the documentation) ...