QueryPath & Data Extraction

What is QueryPath?

QueryPath is an optional configuration parameter that extracts a specific subset of data from a JSON response instead of including the entire response body. This is particularly useful in background jobs where:

• The API response is deeply nested and you only need one field
• Processing large response payloads is inefficient or expensive
• You want to pass clean, focused data to downstream nodes
• Batch processing requires consistent data structure across thousands of items

QueryPath Syntax Reference

Output Structure with QueryPath

When QueryPath is configured, the extracted data is placed in the standardized parsed field of the node output:

{
  "status": "success",
  "statusCode": 200,
  "body": "...",
  "parsed": {
    "id": "user123",
    "email": "john@example.com",
    "profile": {
      "name": "John Doe"
    }
  }
}

{
  "status": "success",
  "statusCode": 200,
  "body": "...",
  "parsed": "john@example.com"
}

Background Job Example: Scenario

Processing 10,000 customer records daily. For each customer, your workflow:

1. Fetches customer profile from a CRM API
2. Extracts only the email address for email validation
3. Passes the email to a Loop node to queue bulk email jobs

The Problem Without QueryPath

Without QueryPath, every API response (even if it contains 50 fields) is passed in full to the next node. Over 10,000 iterations, this creates 500+ MB of unnecessary memory overhead.

// API returns ~5KB per customer
{
  "id": "cust_12345",
  "email": "jane@example.com",      // ← Only this is needed
  "name": "Jane Doe",
  "phone": "+1-555-0123",
  "address": {...},
  "billing": {...},
  "shipping": {...},
  "preferences": {...},
  "subscription": {...},
  ... 40 more fields
}

The Solution With QueryPath

Configure the HTTP Request node with queryPath: $.email to extract only the email address:

{
  "url": "https://api.crm.com/customers/{{ vars.customer_id }}",
  "method": "GET",
  "queryPath": "$.email",
  "auth_type": "bearer",
  "auth_credentials": { "credential_id": "crm-token" },
  "timeout_ms": 10000
}

Now the output is optimized:

{
  "status": "success",
  "statusCode": 200,
  "parsed": "jane@example.com"  // ← Clean, focused data
}

Real-World Background Job Scenarios

Scenario 1: Batch Product Inventory Check

Background Job: Every hour, check inventory levels for 5,000+ products from supplier API and update local database.

API Response (Complex):

{
  "success": true,
  "timestamp": "2024-06-20T15:30:00Z",
  "data": {
    "product": {
      "id": "PROD-789",
      "name": "Widget Pro Max",
      "sku": "WPM-001"
    },
    "inventory": {
      "warehouse_a": {
        "quantity": 450,
        "location": "Shelf B-12",
        "last_counted": "2024-06-19"
      },
      "warehouse_b": {
        "quantity": 230,
        "location": "Aisle 3",
        "last_counted": "2024-06-15"
      }
    },
    "pricing": { ... },
    "metadata": { ... }
  }
}

Extraction Goal:

Get only warehouse A inventory quantity to compare against threshold.

Configuration:

Output:

{
  "parsed": 450  // Just the number
}

Downstream Usage:

// In a Decision node
If Condition: @{output:checkInventory|parsed} < 100
  Route: LOW_STOCK (trigger alert)
Else: NORMAL_STOCK (continue)

Scenario 2: Batch User Data Extraction for Mailing List

Background Job: Daily sync of active users from SaaS platform. Extract emails for newsletter, ignore all other fields.

API Response (Array with Complex Objects):

{
  "page": 1,
  "total_pages": 50,
  "users": [
    {
      "id": "user_001",
      "email": "alice@company.com",     // ← Extract this
      "name": "Alice Johnson",
      "role": "admin",
      "last_login": "2024-06-19T10:30:00Z",
      "permissions": [...],
      "api_quota": {...},
      "2fa_enabled": true,
      "metadata": {...}
    },
    {
      "id": "user_002",
      "email": "bob@company.com",       // ← Extract this
      ...similar structure...
    }
  ]
}

Extraction Goal:

Extract all user emails from the array (not individual users, but all emails at once).

Configuration:

Output:

{
  "parsed": [
    "alice@company.com",
    "bob@company.com",
    "charlie@company.com"
  ]
}

Downstream Usage (Loop Node):

// Loop configuration
{
  "items": "parsed"  // Reference the email array
}

// Each iteration processes one email
// Access current: memory.SetVariable("email", current_item)

Complete Background Job Flow:

1. HTTP Request: Fetch users from SaaS API, QueryPath: $.users[*].email
2. Loop Node: items = "parsed" (the email array)
3. Email Send: For each email, queue newsletter send (inside loop body)
4. Database Update: After loop, update last-sync timestamp
5. Logging: Record how many emails were processed

Scenario 3: Nested Array Extraction - Order Items Processing

Background Job: Nightly order fulfillment job. For each order, extract line items to generate pick lists.

API Response (Heavily Nested):

{
  "order": {
    "id": "ORD-2024-001",
    "customer": {
      "id": "CUST-456",
      "name": "Acme Corp",
      "email": "orders@acme.com"
    },
    "metadata": {
      "created": "2024-06-19T08:00:00Z",
      "source": "web",
      "campaign": "summer_sale"
    },
    "items": [          // ← Target this array
      {
        "sku": "PROD-A1",
        "quantity": 5,
        "price": 29.99,
        "warehouse_location": "W1-A3-14"
      },
      {
        "sku": "PROD-B2",
        "quantity": 3,
        "price": 49.99,
        "warehouse_location": "W2-C1-08"
      }
    ],
    "shipping": {...},
    "billing": {...}
  }
}

Extraction Goal:

Extract only the items array for pick list generation, discard order/customer/shipping info.

Configuration:

Output:

{
  "parsed": [
    {
      "sku": "PROD-A1",
      "quantity": 5,
      "price": 29.99,
      "warehouse_location": "W1-A3-14"
    },
    {
      "sku": "PROD-B2",
      "quantity": 3,
      "price": 49.99,
      "warehouse_location": "W2-C1-08"
    }
  ]
}

Downstream Process:

1. Collection Operation: Sort items by warehouse_location
2. Loop Node: For each item, generate pick label
3. File Generation: Create PDF pick list with only needed fields

Scenario 4: Fallback Logic - Missing or Nested Data

Background Job: API sometimes returns nested data structure, sometimes returns it flat. Use QueryPath to normalize.

API Response Variation 1 (Nested):

{
  "data": {
    "user": {
      "contact": {
        "email": "user@example.com"  // ← Deep nesting
      }
    }
  }
}

API Response Variation 2 (Flat):

{
  "email": "user@example.com"  // ← No nesting
}

Configuration (handles both):

The QueryPath will only succeed for Variation 1. For Variation 2, the parsed field will be omitted, and you can use a Decision node to handle the fallback:

// In Decision node after HTTP Request
If: @{output:httpRequest|parsed} is not empty
  Route to: PROCESS_EMAIL
Else:
  Use fallback: FALLBACK_EXTRACTION  // Handle Variation 2

Performance Impact in Background Jobs

Error Handling & Edge Cases

QueryPath Not Found

If the path doesn't exist in the JSON response:

{
  "status": "success",
  "statusCode": 200,
  "body": "...",
  // "parsed" field is omitted if path not found
}

Handling in workflow: Use a Decision node to check if parsed exists:

If Condition: @{output:httpRequest|parsed} is not empty
  Process the extracted data
Else:
  Log error: "QueryPath did not match response structure"
  Route to error handler

Invalid JSON Response

If the response isn't valid JSON despite Content-Type: application/json:

{
  "status": "success",      // HTTP request succeeded
  "statusCode": 200,
  "body": "invalid json...",
  // "parsed" field not added, error logged
}

Empty or Null Values

If the QueryPath points to a null or empty value, it's still extracted:

// API response
{ "email": null }

// With queryPath: $.email
"parsed": null  // ← Null is preserved, not treated as error

Integration with Expression Framework

Access extracted data using the standard expression syntax:

Single Value

// Direct reference
@{output:httpRequest|parsed}

// In condition
if (@{output:httpRequest|parsed} = "active") { ... }

Array (from QueryPath like $.items[*])

// Count items
@{output:httpRequest|parsed|count}

// Access first item (if applicable)
@{output:httpRequest|parsed|[0]}

Object (from QueryPath like $.user.profile)

// Access nested property
@{output:httpRequest|parsed.email}

// In decision
if (@{output:httpRequest|parsed.status} = "verified") { ... }

Configuration Quick Reference