# GraphQL Extractor (`jupri/graphql-extractor`) Actor πŸ’« Universal GraphQL Scraper - **URL**: https://apify.com/jupri/graphql-extractor.md - **Developed by:** [cat](https://apify.com/jupri) (community) - **Categories:** Developer tools - **Stats:** 27 total users, 0 monthly users, 100.0% runs succeeded, 2 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Welcome to GraphQL Extractor ![dont be sad readme is here](https://raw.githubusercontent.com/JupriGH/resources/main/cats/catframe.png) ### πŸ•ΈοΈ About GRAPHQL ![enter image description here](https://www.gravitee.io/hubfs/graphql.png) **GraphQL** is a data [query](https://en.wikipedia.org/wiki/Query_language "Query language") and [manipulation](https://en.wikipedia.org/wiki/Data_manipulation_language "Data manipulation language") language for [APIs](https://en.wikipedia.org/wiki/API "API") that allows a [client](https://en.wikipedia.org/wiki/Client%E2%80%93server_model "Client–server model") to specify what data it needs ("[declarative](https://en.wikipedia.org/wiki/Declarative_programming "Declarative programming") data fetching"). A GraphQL server can fetch data from separate sources for a single client query and present the results in a unified [graph](https://en.wikipedia.org/wiki/Graph_\(abstract_data_type\) "Graph (abstract data type)").[\[2\]](https://en.wikipedia.org/wiki/GraphQL#cite_note-2) It is not tied to any specific [database](https://en.wikipedia.org/wiki/Database "Database") or storage engine. ### πŸ•ΈοΈ About This Actor Trying to be Universal **GraphQL** Scraper Actor Web scraping is so much easier now, thanks to **GraphQL**. It’s like you ask websites for data, and they just serve it up on a silver platter, no fuss. Honestly, it's made scraping feel less like a chore and more like a friendly chat with the internet. Now, here's hoping [Amazon.com](Amazon.com) jumps on the **GraphQL** bandwagon soonβ€”fingers crossed! ![enter image description here](https://image.shutterstock.com/image-vector/vector-illustration-under-construction-sign-260nw-1665084250.jpg) ### πŸ•ΈοΈ Examples (Use Cases) #### Example #1: `https://apify.hashnode.dev` Scraping blogs from ##### URL : ``` https://apify.hashnode.dev/api/graphql ``` ##### Command : ```graphql query ($host: String!, $after: String, $first: Int!, $filter: PublicationPostConnectionFilter) { publication(host: $host) { id posts(after: $after, first: $first, filter: $filter) { edges { node { __typename id title subtitle slug publishedAt url brief } __typename } pageInfo { hasNextPage endCursor __typename } totalDocuments __typename } __typename } } ``` ##### Variables : ```yaml { "host": "apify.hashnode.dev" } ``` ##### ACTOR INPUT : ```yaml { "limit": 200, "url": "https://apify.hashnode.dev/api/graphql", "variables": "{ \"host\": \"apify.hashnode.dev\", \"first\": 10 }", "query": "query ($host: String!, $after: String, $first: Int!, $filter: PublicationPostConnectionFilter) {\r\n publication(host: $host) {\r\n id\r\n posts(after: $after, first: $first, filter: $filter) {\r\n edges {\r\n node {\r\n __typename id title subtitle slug publishedAt url brief\r\n }\r\n __typename\r\n }\r\n pageInfo { hasNextPage endCursor __typename }\r\n totalDocuments\r\n __typename\r\n }\r\n __typename\r\n }\r\n}", "cursor.step" : 25, "cursor.next" : "after", "cursor.limit" : "first", "parse.root" : "publication.posts", "parse.list" : "edges", "parse.item" : "node", "parse.total" : "totalDocuments", "parse.next" : "pageInfo.endCursor" } ``` ### πŸ•ΈοΈ GraphQL Quick Guide #### Lesson #1: BASIC ```php SYNTAX: { commandName(parameter: ARGUMENT) { FIELD_LIST } } ``` ***Example :*** ```php { search(name:"pants" count:10) { id name __typename } } ``` ***JSON Input :*** ```yaml { "query": "{ search(name:\"pants\" count:10) { id name __typename } }" } ``` #### Lesson #2: VARIABLES ```php SYNTAX: query ($VARIABLE: TYPE) { commandName(parameter: $VARIABLE) { FIELD_LIST } } ``` ***Example :*** ```php query ($text:String $limit:Int) { search(name:$text count:$limit) { id name __typename } } ``` ***JSON Input :*** ```yaml { "query": "query ($text:String $limit:Int){ search(name:$text count:$limit){id name __typename} }", "variables": { "text": "pants", "limit": 10 } } ``` #### Lesson #3: FRAGMENT ```php SYNTAX: query ($VARIABLE: TYPE) { commandName(parameter: $VARIABLE) { ... FRAGMENT_NAME } } fragment FRAGMENT_NAME on TYPE { FIELD_LIST } ``` ***Example: Without Fragment*** ```php query ($text:String $limit:Int) { search(name:$text count:$limit) { id name sku description url specifications __typename variations { id name sku description url __typename } similarProducts { id name sku description url __typename } } } ``` ***Example: With Fragment*** ```php query ($text:String $limit:Int) { search(name:$text count:$limit) { ... ProductInfo specifications variations { ... ProductInfo } similarProducts { ... ProductInfo } } } fragment ProductInfo on Product { id name sku description url __typename } ``` # Actor input Schema ## `limit` (type: `integer`): πŸ’‘ Number of results ## `url` (type: `string`): πŸ’‘ GraphQL URL ## `query` (type: `string`): πŸ’‘ GraphQL command or SHA1 Code (Persistent QUERY Code => 32 BYTES/64 HEX Char) ## `variables` (type: `string`): πŸ’‘ GraphQL input variables (JSON) ## `cursor.step` (type: `integer`): πŸ’‘ Maximum number of IPP (items-per-page) ## `cursor.index` (type: `string`): πŸ’‘ Pagination MODE 1️⃣: VARIABLES\[INDEX] value will increased by +1 ## `cursor.next` (type: `string`): πŸ’‘ Pagination MODE 2️⃣: VARIABLES\[NEXT] will set to cursor value from previous request ## `cursor.offset` (type: `string`): πŸ’‘ Pagination MODE 3️⃣: VARIABLES\[OFFSET] value will increased by +STEP ## `cursor.limit` (type: `string`): πŸ’‘ Pagination: VARIABLES\[LIMIT] will set to number of IPP (STEP) ## `parse.root` (type: `string`): πŸ’‘ Root of data ## `parse.list` (type: `string`): πŸ’‘ ROOT\[LIST]: Result List (relative to ROOT) ## `parse.item` (type: `string`): πŸ’‘ LIST\[ITEM]: Path to Item (relative to LIST) ## `parse.total` (type: `string`): πŸ’‘ ROOT\[TOTAL]: Total number of Data (relative to ROOT) ## `parse.next` (type: `string`): πŸ’‘ ROOT\[NEXT]: Cursor value for next page request ## `no_typename` (type: `boolean`): No description ## `dev_proxy_config` (type: `object`): πŸ’‘ Supported protocol:

HTTP(S), SOCKS5
{http|socks5}://{user:pass}@{hostname|ip-address}:port

Example: socks5://example.com:9000 ## `dev_custom_headers` (type: `array`): πŸ’‘ Additional HTTP Headers ## `dev_custom_cookies` (type: `array`): πŸ’‘ Additional HTTP Cookies ## `dev_transform_fields` (type: `array`): πŸ’‘ Transform the resulting output. Select only needed fields.

For nested object use DOT. For example: 
address.streetAddress

For nested array use NUMBER (index of array element starting from index=0). For example: 
images.0.url
## `dev_dataset_name` (type: `string`): πŸ’‘ Save results into custom named Dataset, use mask to customize dataset name

{ACTOR} = actor name
{DATE} = date (YYYYMMDD)
{TIME} = time (HHMMSS)

This masks can be used to autogenerate Dataset Name.

example: data-{DATE}
Depending on today date the dataset name will be: data-20230603

default: data-{ACTOR}-{DATE}-{TIME} ## `dev_dataset_clear` (type: `boolean`): Clear Dataset before insert/update. ## `dev_no_strip` (type: `boolean`): πŸ’‘ Keep/Save empty values (NULL, FALSE, empty ARRAY, empty OBJECT, empty STRING) ## Actor input object example ```json { "limit": 100, "url": "https://apify.hashnode.dev/api/graphql", "query": "query ($host: String!, $after: String, $first: Int!, $filter: PublicationPostConnectionFilter) {\r\n publication(host: $host) {\r\n id\r\n posts(after: $after, first: $first, filter: $filter) {\r\n edges {\r\n node {\r\n __typename id title subtitle slug publishedAt url brief\r\n }\r\n __typename\r\n }\r\n pageInfo { hasNextPage endCursor __typename }\r\n totalDocuments\r\n __typename\r\n }\r\n __typename\r\n }\r\n}", "variables": "{ \"host\": \"apify.hashnode.dev\", \"first\": 10 }", "cursor.step": 25, "cursor.next": "after", "cursor.limit": "first", "parse.root": "publication.posts", "parse.list": "edges", "parse.item": "node", "parse.total": "totalDocuments", "parse.next": "pageInfo.endCursor" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "limit": 100, "url": "https://apify.hashnode.dev/api/graphql", "query": `query ($host: String!, $after: String, $first: Int!, $filter: PublicationPostConnectionFilter) { publication(host: $host) { id posts(after: $after, first: $first, filter: $filter) { edges { node { __typename id title subtitle slug publishedAt url brief } __typename } pageInfo { hasNextPage endCursor __typename } totalDocuments __typename } __typename } }`, "variables": "{ \"host\": \"apify.hashnode.dev\", \"first\": 10 }", "cursor.step": 25, "cursor.index": "", "cursor.next": "after", "cursor.offset": "", "cursor.limit": "first", "parse.root": "publication.posts", "parse.list": "edges", "parse.item": "node", "parse.total": "totalDocuments", "parse.next": "pageInfo.endCursor" }; // Run the Actor and wait for it to finish const run = await client.actor("jupri/graphql-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "limit": 100, "url": "https://apify.hashnode.dev/api/graphql", "query": """query ($host: String!, $after: String, $first: Int!, $filter: PublicationPostConnectionFilter) {\r publication(host: $host) {\r id\r posts(after: $after, first: $first, filter: $filter) {\r edges {\r node {\r __typename id title subtitle slug publishedAt url brief\r }\r __typename\r }\r pageInfo { hasNextPage endCursor __typename }\r totalDocuments\r __typename\r }\r __typename\r }\r }""", "variables": "{ \"host\": \"apify.hashnode.dev\", \"first\": 10 }", "cursor.step": 25, "cursor.index": "", "cursor.next": "after", "cursor.offset": "", "cursor.limit": "first", "parse.root": "publication.posts", "parse.list": "edges", "parse.item": "node", "parse.total": "totalDocuments", "parse.next": "pageInfo.endCursor", } # Run the Actor and wait for it to finish run = client.actor("jupri/graphql-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "limit": 100, "url": "https://apify.hashnode.dev/api/graphql", "query": "query ($host: String!, $after: String, $first: Int!, $filter: PublicationPostConnectionFilter) {\\r\\n publication(host: $host) {\\r\\n id\\r\\n posts(after: $after, first: $first, filter: $filter) {\\r\\n edges {\\r\\n node {\\r\\n __typename id title subtitle slug publishedAt url brief\\r\\n }\\r\\n __typename\\r\\n }\\r\\n pageInfo { hasNextPage endCursor __typename }\\r\\n totalDocuments\\r\\n __typename\\r\\n }\\r\\n __typename\\r\\n }\\r\\n}", "variables": "{ \\"host\\": \\"apify.hashnode.dev\\", \\"first\\": 10 }", "cursor.step": 25, "cursor.index": "", "cursor.next": "after", "cursor.offset": "", "cursor.limit": "first", "parse.root": "publication.posts", "parse.list": "edges", "parse.item": "node", "parse.total": "totalDocuments", "parse.next": "pageInfo.endCursor" }' | apify call jupri/graphql-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,jupri/graphql-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/3ALyP9gVV5gx75ByE/builds/QGvYQdohPETzd30T9/openapi.json