Declarative Data Sync Pipelines Without ETL Tools for SaaS

If you are writing Python or Node.js scripts to build a custom SaaS API connector and pull data from your customers' SaaS tools into your application, or deploying traditional ETL tools to handle the load, you have already lost the war on maintenance. Every new integration adds another cursor to track, another auth flow to debug, another pagination scheme to reverse-engineer. Building customer-facing integrations using imperative code or traditional Extract, Transform, Load (ETL) pipelines introduces heavy maintenance burdens, data latency, and severe compliance risks.

Declarative data sync pipelines without ETL tools for SaaS flip the model: you define what the data should look like in a versioned configuration, and a unified execution engine handles the extraction, transformation, and delivery. No Singer taps to fork. No visual workflow canvases to click through. No middleware database caching your customers' sensitive Personally Identifiable Information (PII) on someone else's servers. This architectural shift allows B2B SaaS companies to scale their integration catalogs without expanding engineering headcount.

This guide is for engineering leaders and PMs at B2B SaaS companies who are past the prototype stage and hitting the wall where custom integration code and heavy ETL frameworks stop scaling. We will break down why traditional ETL tools fail for customer-facing integrations, why visual iPaaS platforms create technical debt, what a declarative architecture actually looks like, and how zero data retention changes the enterprise procurement conversation.

The Problem With ETL Tools for Customer-Facing SaaS Integrations

Traditional ETL tools were designed for a very specific job: pulling massive volumes of internal enterprise data into a central data warehouse for Business Intelligence (BI) and analytics. Platforms like Fivetran, Airbyte, Stitch, and newer embedded ETL players like Hotglue inherit that DNA. They excel at batch loads into Snowflake, BigQuery, or Redshift.

When B2B SaaS companies attempt to repurpose ETL tools to power real-time, bidirectional, customer-scoped access to third-party APIs from inside their application, the architecture immediately shows its cracks. The mismatch shows up in four critical areas:

1. Data Privacy and the Caching Problem

ETL architectures inherently rely on intermediate storage. To transform data, the ETL engine must first extract it and hold it in a staging area or caching layer. If you are building an integration to pull employee records from a customer's HRIS into your platform, an embedded ETL tool will temporarily store that highly sensitive data on their servers.

For enterprise customers, this intermediate destination is a massive red flag. The moment a third-party integration tool stores customer data, it becomes a data sub-processor. This triggers extensive vendor risk assessments, Data Protection Impact Assessments (DPIAs), and prolonged procurement cycles. Enterprise InfoSec teams will hold up your deal until you can prove that intermediate copy either does not exist or lives strictly inside their tenant boundary.

2. Batch Processing vs. Real-Time Syncs

ETL tools are batch-oriented. They run on scheduled cron jobs (e.g., every 6 hours or once a day). Customer-facing SaaS integrations often require real-time or near-real-time bidirectional data flow. If a sales rep updates a deal stage in your application, they expect that change to reflect in their Salesforce instance immediately, not during the next nightly batch sync.

3. Bidirectional Write Paths

ETL is inherently one-way (Extract, Transform, Load to a destination). Customer-facing integrations need to push data back into Salesforce, create Jira tickets, or update Zendesk records. Bolting write operations onto an extract-first tool produces the ugliest, most brittle code in your repository.

4. The Maintenance Reality of Per-Customer Scoping

Warehouse ETL assumes one set of credentials per source. B2B SaaS integrations need thousands of OAuth-authorized connections, each scoped to a single customer, each with its own token refresh cycle.

Embedded ETL solutions often rely on open-source standards like the Singer specification, wrapping the tap/target spec in Python-based tenant isolation. The trade-off is that your team still owns and maintains Python scripts for every integration, plus the Kubernetes-scale infrastructure to run isolated jobs per tenant.

The average enterprise company now manages over 300 SaaS applications, according to Zylo's SaaS Management Index. If you rely on ETL pipelines, scaling to support those 300 applications means writing and maintaining 300 separate Python scripts. You have not eliminated integration-specific code; you have just moved it into a different repository. You need a better way to handle bulk extraction workflows without the overhead of traditional ETL.

Imperative vs. Declarative Data Pipelines: The Architectural Shift

To escape the maintenance trap, engineering teams must move from imperative data pipelines to declarative data pipelines.

An imperative pipeline describes step-by-step how to get data. You write code that says: Authenticate using this API key, request page one, parse the response, extract the contact_id, map it to your internal customerId, check for a next page cursor, request page two, handle the rate limit error, sleep for 5 seconds, retry, persist the offset.

This code is inherently brittle. When the upstream API changes its pagination cursor format, renames a field, or introduces a stricter rate limit threshold, your script fails. You write a code change, push it through CI, and redeploy. Recent research across SaaS engineering teams indicates that a single production-grade integration costs approximately $16,000 to build in year one, and maintaining 50 integrations costs upwards of $600,000 annually. The initial build represents only 20% of the total cost of ownership; the remaining 80% is entirely consumed by the integration maintenance tax, API versioning, and edge-case handling.

A declarative pipeline inverts this model. Instead of writing the execution steps, you describe what the output should be. You specify the source endpoint, the target schema, and a transformation expression. The underlying execution engine figures out how to paginate, when to refresh tokens, how to normalize errors, and how to enforce rate limits.

The difference maps cleanly onto how modern infrastructure has evolved everywhere else. Terraform replaced hand-rolled provisioning scripts. Kubernetes manifests replaced imperative deployment shell scripts. GraphQL schemas replaced hard-coded REST clients. Integration code is the last place where senior engineers still write imperative glue by hand.

flowchart TD
    subgraph Imperative ["Imperative Pipeline (Code-Heavy)"]
        A["Python / Node Script"] --> B{"Handle Auth"}
        B --> C["Fetch Page 1"]
        C --> D{"Transform Data"}
        D --> E{"Check Next Cursor"}
        E -->|Yes| C
        E -->|Rate Limit| F["Sleep & Retry"]
    end

    subgraph Declarative ["Declarative Pipeline (Config-Driven)"]
        G["JSONata Config"] --> H["Declarative Engine"]
        H --> I["Standardized Output"]
    end

Here is the concrete shape of a declarative mapping using JSONata, the JSON query and transformation language that has become the de facto standard for schema translation. This example maps a proprietary HubSpot payload into a standardized Common Data Model:

{
  "resource": "contact",
  "source": "hubspot",
  "mapping": {
    "id": "vid",
    "email": "properties.email.value",
    "first_name": "properties.firstname.value",
    "last_name": "properties.lastname.value",
    "company": "properties.company.value",
    "created_at": "$fromMillis(properties.createdate.value)",
    "custom_fields": "$sift(properties, function($v, $k) { $startsWith($k, 'custom_') })"
  }
}

That single JSON document replaces roughly 150 lines of Python that would otherwise handle HubSpot's nested properties object, coerce timestamps, and pluck custom fields dynamically.

Similarly, you can map Zendesk's disparate shapes using the exact same declarative approach:

{
  "id": upstream_id,
  "first_name": $split(name, " ")[0],
  "last_name": $split(name, " ")[1],
  "email": contact_info[type="work"].email,
  "status": is_active ? "ACTIVE" : "INACTIVE",
  "metadata": {
    "source": "zendesk",
    "original_created_at": created_at
  }
}

Because this mapping is a static configuration file, it lives alongside your application code in Git. It is diff-able, reviewable in a pull request, and swappable across customers without touching the runtime. You ship your integrations as configuration, not code.

Why Embedded iPaaS Isn't the Answer for Engineering Teams

Recognizing the pain of imperative scripts, many teams evaluate embedded Integration Platform as a Service (iPaaS) solutions like Prismatic, Alloy Automation, or Workato Embedded. These platforms attempt to solve the integration problem by offering visual, drag-and-drop workflow builders.

While visual builders demo well to non-technical stakeholders, they introduce severe operational friction for engineering teams.

1. Version Control is Theater Every mature engineering team runs code review, CI, and staged rollouts. Visual workflow builders force engineers into "click-ops"—manually clicking through a web interface to configure logic loops and mappings. Low-code workflows serialize into massive JSON blobs that technically live in Git, but diffing an auto-generated 5,000-line JSON file to find a single field mapping change is an operational nightmare. Rollbacks are ambiguous. Feature flags are a hack.

2. Programmatic Data Syncs Are Not Workflows A workflow builder is designed for event-driven "when X happens, do Y" logic. Bulk syncing 500,000 Salesforce contacts across 200 customer connections is not a workflow. It is a data pipeline, and pipelines belong in code or declarative config, not a visual DAG (Directed Acyclic Graph). You are still building bespoke point-to-point connections; you are just doing it with a mouse instead of a keyboard.

3. Cost Scales With Clicks, Not Value iPaaS platforms typically price on task or step execution. A pipeline that fans out across every customer connection multiplies your bill by the number of tenants and records. You end up either capping usage (creating a bad customer experience) or eating margin. For B2B SaaS engineering teams, declarative unified APIs provide a superior architecture.

Core Components of a Declarative SaaS Integration Pipeline

A production-grade declarative pipeline for B2B SaaS integrations needs five core components. If any one is missing, you fall back to writing custom code.

1. Unified Authentication and Token Lifecycle

Managing OAuth 2.0 token lifecycles across 50 different SaaS providers is historically one of the most error-prone aspects of integration engineering. Every provider implements the RFC 6749 specification slightly differently. Some access tokens expire in 1 hour; others live for 10 years. Some employ strict refresh token rotation.

A declarative platform maintains the durable state of the OAuth connection. It proactively schedules token refreshes ahead of their Time-To-Live (TTL) expiration, handles the long tail of vendor-specific quirks (Salesforce's short refresh windows, Google's per-user consent scopes, NetSuite's TBA), and securely injects the valid bearer token into the HTTP headers. Your application simply passes a tenant identifier, and you never see an expired token in your application logs.

2. Declarative Schema Mapping with JSONata

Third-party SaaS platforms use wildly different data models. The mapping layer must normalize these disparate shapes into your application's Common Data Model (CDM). JSONata handles the messy transformations real APIs demand: coercing types, flattening nested objects, filtering arrays, and picking custom fields dynamically. Crucially, the mapping layer must support per-tenant overrides so that Customer A's custom Salesforce field can map to your standard deal.stage while Customer B's picks a different field entirely, all without forking code.

3. Normalized Pagination Across Providers

Cursor-based, offset-based, page-token, Link header, next_url, opaque hashes: every API paginates differently. A declarative engine intercepts the upstream provider's specific pagination format and abstracts it into a single iteration primitive. Your configuration just says "give me all contacts," and the runtime handles the fifty-plus pagination patterns across your connector library.

4. Honest Rate Limit Handling

This is where the industry gets sloppy. Many integration platforms claim to "handle" rate limits by silently queueing and retrying requests when an upstream API returns an HTTP 429 Too Many Requests error. This looks nice in a demo but hides real problems: your app has no visibility into throttling, retries can cascade under load, and idempotency violations on writes become invisible failures.

The honest architecture is to pass HTTP 429s cleanly to the caller and normalize the metadata. Truto follows the strictly transparent approach by parsing proprietary upstream headers and standardizing them into the official IETF draft specification headers:

• ratelimit-limit: The total request quota.
• ratelimit-remaining: The remaining requests in the current window.
• ratelimit-reset: The time until the quota resets.

5. Webhook Normalization and Delivery

Inbound webhooks are just as messy as outbound requests. Signature verification schemes differ, event payload shapes differ, and replay semantics differ. A declarative platform verifies signatures, normalizes event shapes into your common data model, and delivers them to your app with retries and dead-letter handling.

Achieving Zero Data Retention in Enterprise Integrations

Here is the architectural shift that changes procurement outcomes. Traditional ETL and embedded iPaaS tools cache customer data in their own infrastructure. That intermediate copy is what turns your integration vendor into a data sub-processor. Enterprise InfoSec teams then run their vendor risk assessment against that sub-processor. That is where your six-figure deal stalls for six weeks.

A declarative pipeline can be architected as a pure pass-through proxy. In a pass-through architecture, the integration engine receives the request from your application, injects the OAuth token, applies the JSONata transformation to the payload in memory, and forwards the request to the upstream SaaS provider. The response is transformed in memory and immediately returned to your application.

sequenceDiagram
    participant App as "Your App"
    participant Truto as "Declarative Engine (Pass-Through)"
    participant Upstream as "Upstream API (e.g., Salesforce)"

    App->>Truto: GET /crm/contacts (Tenant ID)
    Note over Truto: Inject OAuth Token<br>No data stored to disk
    Truto->>Upstream: GET /services/data/v58.0/query
    Upstream-->>Truto: HTTP 200 (Proprietary JSON)
    Note over Truto: Apply JSONata Mapping<br>in memory
    Truto-->>App: HTTP 200 (Normalized JSON)

At no point does the declarative engine write the payload to a durable database, queue, or warehouse. When the InfoSec questionnaire asks "where is our customer data stored by this vendor?", the answer is "it isn't."

The practical implications of this zero data retention approach are massive:

• DPA scope shrinks: You do not need a Data Processing Agreement covering the integration vendor's storage of end-customer PII, because there is none.
• HIPAA and GDPR reviews accelerate: The vendor never becomes a covered entity holding PHI or PII.
• Regional compliance is trivial: If a customer requires EU-only data residency, a pass-through platform routing through EU infrastructure satisfies it without complex data-locality architectures.
• Breach blast radius is bounded: If the integration vendor is compromised, there is no historical dataset to leak.

Ship Integrations as Config, Not Code

The business mandate for B2B SaaS companies is clear: customers demand native integrations with their existing tech stacks. Buyers with five or more integrations consistently show lower churn, with 92% of B2B SaaS leaders observing that integrated customers are stickier.

Attempting to meet this demand by writing imperative Python scripts or deploying heavy ETL tools guarantees that your engineering team will eventually drown in maintenance debt. Visual iPaaS platforms offer a temporary band-aid, but they ultimately frustrate developers by removing version control and forcing bespoke workflow creation.

Declarative pipelines break the cost curve by decoupling integration logic from execution mechanics. Your engineers define the data shape using JSONata, and the infrastructure handles the authentication, pagination, rate limits, and schema normalization. This architecture makes three things possible:

1. Non-engineers can ship integrations: Solution engineers and PMs can author JSONata configs. Engineering reviews the changes, but does not author them.
2. Per-customer customization stops requiring code forks: When an enterprise customer needs a custom field mapped, you edit a config scoped to that tenant. No feature flags.
3. API changes become config changes: When an upstream vendor renames a field, one PR to a mapping file fixes it across every customer.

Stop building integration maintenance factories. Start shipping integrations as config.