---
title: "How to Stream SaaS Webhooks to Kafka & Redpanda (Architecture Guide)"
slug: how-to-stream-saas-webhooks-to-kafka-redpanda-architecture-guide
date: 2026-05-11
author: Roopendra Talekar
categories: [Engineering, Guides]
excerpt: "Architect a reliable pipeline for streaming third-party SaaS webhooks into Kafka or Redpanda. Learn how to handle verification, normalization, claim-checks, and rate limits."
tldr: "Kafka and Redpanda cannot ingest HTTP webhooks natively. A unified webhook architecture handles signature verification, payload normalization, and API enrichment before feeding a tiny HTTP listener that produces to your event broker."
canonical: https://truto.one/blog/how-to-stream-saas-webhooks-to-kafka-redpanda-architecture-guide/
---
# How to Stream SaaS Webhooks to Kafka & Redpanda (Architecture Guide)
If you are trying to pipe third-party SaaS webhooks—Salesforce, HubSpot, Jira, HiBob, Zendesk, Stripe—directly into Kafka or Redpanda for event-driven data pipelines, you have likely hit a fundamental architectural wall: message brokers do not speak HTTP.
Engineering teams building integrations for B2B SaaS products eventually outgrow API polling. Batch polling is the old default. You would schedule a cron job to hit a CRM's `/contacts` endpoint every 15 minutes, diff the result against your database, and emit change events. It worked—until you scaled past a handful of integrations and started burning 30-40% of your API quota on null deltas, while customers complained about unacceptable three-hour data lag.
As we've covered in our [guide to handling real-time data sync from legacy APIs](https://truto.one/how-to-handle-webhooks-and-real-time-data-sync-from-legacy-apis/), the business cost of this latency is severe. Gartner estimates that poor data quality costs organizations an average of $12.9 million per year, largely due to data decay and missed operational opportunities. Research from Marketing Sherpa backs this up, indicating that B2B contact data decays at roughly 2.1% per month. If your CRM or ERP integration relies on a nightly batch job, your customers are operating on measurably stale data.
The industry has moved on. As of 2025, 72% of enterprises use event-driven workflows to build scalable, loosely coupled systems. IDC research shows that 90% of the world's largest companies will be using realtime data by 2025. When real-time matters, Kafka and Redpanda are the obvious destinations. They are durable, partitioned, replayable, and battle-tested. LinkedIn uses partitioning and sharding in Kafka to process trillions of events daily, with event driven architecture underpinning its real-time feed and ads infrastructure.
If you are already running a Kafka or Redpanda cluster for internal events, routing third-party SaaS events into the same fabric is the natural play—one stream, one consumer model, one observability surface. But because Kafka and Redpanda are not HTTP-native, you cannot just point a SaaS provider's webhook URL at your broker. You need a listener in front of your event bus that handles signature verification, challenge handshakes, idempotency, and enrichment before a single byte hits a topic.
This guide breaks down the architectural patterns for ingesting third-party SaaS webhooks into event streaming platforms, the hidden costs of building custom HTTP shims, and how to use a unified webhook architecture to stream normalized, enriched data directly to your consumers.
## The Architectural Gap: Why Kafka and Redpanda Cannot Ingest Webhooks Directly
**Kafka and Redpanda are not HTTP-native brokers.** They expose a binary wire protocol over TCP, designed for high-throughput producers and consumers that authenticate via SASL/mTLS. They expect producers to maintain persistent TCP connections, serialize data into binary formats (like Avro or Protobuf), and handle partition routing. There is no `/webhook` endpoint waiting for a Salesforce POST.
Webhooks operate on entirely different assumptions. Every webhook ingestion architecture therefore needs an HTTP listener sitting in front of the broker. This seems trivial until you list what the listener actually has to do when a SaaS provider fires an event:
1. **Provide a public-facing HTTP endpoint:** The provider needs an accessible URL, usually secured via HTTPS. Your listener must terminate TLS and parse the raw body without mutating it, because cryptographic signatures are computed over byte-exact payloads.
2. **Verify cryptographic signatures:** You must verify the request actually came from the provider in whatever scheme they use—HMAC-SHA256 (Stripe, GitHub), JWT (Microsoft Graph), Basic Auth (legacy systems), Bearer tokens, or custom header schemes.
3. **Respond to challenge handshakes:** Some providers (like Slack, Zoom, or Microsoft Graph) require an initial verification handshake. When you register the webhook, they send a challenge string, and your endpoint must immediately echo it back to prove ownership before they will send any real events.
4. **Acknowledge fast with a synchronous 2xx response:** Most providers expect your server to acknowledge receipt within a strict timeout window (often 3 to 5 seconds). If you do not respond quickly enough, they retry aggressively. Miss the window repeatedly, and you get duplicates, or worse, the webhook gets auto-disabled.
5. **Enrich thin payloads:** Many APIs send "thin" webhooks containing only an ID and event type. HiBob's `employee.updated` event, for example, does not include the employee record. You have to call the API to fetch it before you have anything useful to put on a topic.
6. **Deduplicate:** Providers retry on network blips, so the exact same event can land two or three times within minutes.
Message brokers handle exactly zero of these requirements. Dumping raw POST bodies into a Kafka topic without any of this gives you a topic full of unverified, unauthenticated, partially-formed payloads that downstream consumers cannot trust.
If you try to use a generic HTTP-to-Kafka proxy (like the Confluent REST Proxy), you will immediately fail the security requirements. The proxy will happily ingest unverified, potentially malicious payloads, and it has no mechanism to respond to provider-specific challenge handshakes. Another naive workaround is to use something like Redpanda Connect (formerly Benthos), which ships with an `http_server` input that can write directly to a topic. The http_server input registers endpoints that support path parameters of the form /{foo}, and expects POST requests where the entire request body is consumed as a single message. You can even add HMAC verification with a Bloblang processor. But the moment you support more than two or three providers, that YAML config becomes its own massive integration codebase—and it still does not solve API enrichment, fan-out across tenants, or idempotency.
## The Hidden Costs of the 'HTTP Shim' Anti-Pattern
To bridge the gap between HTTP webhooks and TCP event streams, engineering teams typically build an "HTTP Shim"—a custom microservice (usually written in Node.js, Go, or Python) that sits behind an ALB, with provider-specific verification logic and a Kafka producer client.
At first, this looks like a weekend project. You write an Express.js route, verify an HMAC signature, and push the JSON payload to a Kafka topic. By month six, it mutates into a massive, unmaintainable API gateway liability.
Here is what happens when you scale the HTTP shim anti-pattern across dozens of integrations:
```mermaid
flowchart LR
A[Provider A: HMAC-SHA256] --> S[Custom HTTP Shim]
B[Provider B: JWT + Challenge] --> S
C[Provider C: Basic Auth] --> S
D[Provider D: Thin Payload] --> S
S --> V{Verify Security}
V -->|Fail| X[Drop & Log 401]
V -->|Success| E[Enrich via REST API]
E --> I[State Machine / Idempotency]
I --> K[(Kafka / Redpanda Topic)]
```
### 1. The Verification Wild West
Every SaaS provider handles webhook security as a snowflake. Each arrow into your shim is a per-provider quirk:
- **Stripe** rotates signing secrets and expects you to support two simultaneously during the rotation window.
- **GitHub** uses `X-Hub-Signature-256`.
- **Shopify** uses `X-Shopify-Hmac-Sha256` and Base64 encoding instead of hex.
- **Slack** signs a concatenation of the timestamp and body with a versioned scheme.
- **Salesforce** sends outbound messages as SOAP XML over CometD, requiring you to parse XML just to acknowledge it.
- **Microsoft Graph** requires you to validate a validation token during setup and then handle encrypted payloads at runtime.
Your HTTP shim quickly becomes bloated with provider-specific security logic. Managing the rotation of these webhook secrets across 50 different integrations becomes a security nightmare. For a deeper look at the pitfalls of managing this yourself, see our guide on [designing reliable webhooks in production](https://truto.one/designing-reliable-webhooks-lessons-from-production/).
### 2. The "Thin Webhook" Problem
Many SaaS APIs send "thin" webhooks. When an employee is created in a legacy HRIS, the webhook payload might look like this:
```json
{
"event": "employee.created",
"employee_id": "emp_88392"
}
```
This is useless to your downstream Kafka consumers. They need the employee's name, email, role, and department. To get that data, your HTTP shim must pause, authenticate with the provider's REST API (which means OAuth token management and refresh-on-expiry logic), fetch the full employee record, stitch it together with the webhook event, and *then* push it to Kafka.
If you do this synchronously, you will blow past the provider's 3-second webhook timeout. If you do it asynchronously, you have to build a complex state machine inside your shim to track which events are pending enrichment.
### 3. Schema Drift and Normalization
If you dump raw webhook payloads directly into a Kafka topic, you are just moving the integration problem downstream. Your Kafka consumers now have to write provider-specific logic to parse HubSpot JSON versus Salesforce XML. When a provider changes their webhook schema (which happens constantly), your downstream consumers break.
> [!WARNING]
> **The Trap:** Teams treat the HTTP shim as infrastructure and the verification logic as business code. In reality, the verification logic *is* the infrastructure—and it never stops changing. If each new provider takes one engineering week to add (verification, payload mapping, enrichment, tests, runbook), and you support 30 providers, you have burned almost a full year of a senior engineer just to keep the shim alive.
## How to Stream Normalized Webhooks to Kafka or Redpanda with Truto
A unified webhook architecture flips the paradigm. Instead of N custom HTTP shims per provider, modern engineering teams point every provider at a single ingestion surface that does verification, normalization, and enrichment as configuration. Truto acts as this ingestion layer, handling the messy HTTP edge cases and emitting a canonical event your Kafka or Redpanda producer can blindly subscribe to.
Here is how Truto sits in front of your event broker:
```mermaid
flowchart LR
P1[Salesforce] --> T
P2[HiBob] --> T
P3[Jira] --> T
P4[Stripe] --> T
T[Truto Unified Webhook Engine] --> V[Verify Signature: HMAC/JWT/Basic]
V --> M[JSONata Mapping to record:created]
M --> E[Enrich Thin Payloads via Unified API]
E --> O[Outbound Signed Delivery: X-Truto-Signature]
O --> L[Your Tiny Stateless HTTP Listener]
L --> K[(Kafka / Redpanda Topic)]
```
### Step 1: Declarative Ingestion and Verification
Truto provides a secure, public-facing URL for each integrated account. When a webhook arrives, Truto immediately returns a 200 OK to the provider to prevent retries.
Verification is declarative, not code. Each integration carries a config block describing how its signatures work. The same verification engine handles Stripe's HMAC, Microsoft Graph's JWT, and legacy Basic Auth using timing-safe equality checks to prevent side-channel attacks. Challenge handshakes are handled the same way: a JSONata expression inspects the payload, decides whether it is a handshake or a real event, and emits the correct echo response back to the provider without you writing any custom routing logic.
### Step 2: JSONata Normalization
Every provider has its own event taxonomy—`employee.joined`, `employee.left`, `contact.created`, `Contact.Updated`, `customer.subscription.updated`. Truto evaluates declarative JSONata expressions to map all of these onto a tight canonical vocabulary: `record:created`, `record:updated`, `record:deleted`, and `record:meta`.
Your Kafka consumers never see the raw provider schema. They only consume the unified data model. A consumer reading a Kafka topic only needs to handle four event types, not 400. You can learn more about this process in our [webhook normalization guide](https://truto.one/what-is-webhook-normalization-2026-integration-guide/).
### Step 3: Automatic Enrichment
If Truto receives a "thin" webhook containing only an ID, the engine automatically calls the underlying provider's API via the unified proxy layer, fetches the full resource, and maps it to the unified schema before the outbound event fires. Your Kafka consumer never sees a half-formed `{employee_id: "abc"}` payload; it sees a fully populated, enriched unified record.
### Step 4: The Downstream Contract
Outbound webhooks from Truto to your infrastructure are signed with HMAC-SHA256 using a per-subscription secret. The signature is delivered in an `X-Truto-Signature` header.
Your HTTP listener (the tiny stateless service that actually publishes to Kafka) only has to verify *one* signature scheme, regardless of how many upstream SaaS providers you have added. A minimal listener pushing to Redpanda looks like this:
```typescript
import { Kafka } from 'kafkajs'
import crypto from 'node:crypto'
const producer = new Kafka({
brokers: [process.env.REDPANDA_BROKERS!.split(',')].flat(),
}).producer()
await producer.connect()
export async function handleTrutoWebhook(req: Request) {
const body = await req.text()
const header = req.headers.get('x-truto-signature') ?? ''
const sig = header.split('v=')[1]
const expected = crypto
.createHmac('sha256', process.env.TRUTO_WEBHOOK_SECRET!)
.update(body)
.digest('base64')
if (!crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected))) {
return new Response('invalid signature', { status: 401 })
}
const event = JSON.parse(body)
await producer.send({
topic: `saas.${event.payload.resource.replace('/', '.')}`,
messages: [{
key: event.payload.integrated_account_id,
value: JSON.stringify(event),
headers: {
'event-type': event.event_type,
'raw-event-type': event.payload.raw_event_type,
},
}],
})
return new Response('ok', { status: 200 })
}
```
That is the entire "shim." Twenty lines of code, one signature scheme, and the only thing it does is route already-normalized events into the correct topic. Adding a 31st provider does not touch this code; it is merely a configuration change inside Truto.
> [!TIP]
> Use `integrated_account_id` as the Kafka partition key. This guarantees per-tenant ordering, which is almost always what you want for downstream consumers materializing state.
## Handling Large Payloads: The Claim-Check Pattern
One of the most dangerous edge cases in webhook-to-Kafka pipelines is message size limits. By default, Kafka's `message.max.bytes` is typically set to 1MB. Redpanda has similar default constraints.
SaaS webhook payloads can blow past this easily. A Notion page export might be 10MB. A Jira ticket webhook might contain 50 comments and massive HTML email threads. A Salesforce webhook might include Base64-encoded file attachments. If your HTTP listener tries to push a 5MB payload into a 1MB Kafka topic, the producer will throw a `RecordTooLargeException`, the message will be dropped, and your data pipeline will silently lose state.
Increasing the Kafka message size limit is a recognized anti-pattern. It degrades broker performance, increases memory pressure, and slows down replication.
To solve this, the industry-standard fix is the **claim-check pattern**. Instead of forcing massive payloads through the message queue, you decouple the payload from the event notification.
### How the Claim-Check Pattern Works
Truto applies this internally for outbound delivery, but when you adopt the same pattern between your HTTP listener and Kafka, you keep topics fast and consumers cheap:
1. **Store the Payload:** When your listener receives a massive webhook event, it writes the full JSON payload to durable object storage (like S3, GCS, or R2). The object is keyed by a unique event ID.
2. **Send the Metadata:** The listener then constructs a lightweight metadata message containing the event type, the integrated account ID, and the object storage reference ID.
3. **Produce to the Queue:** This lightweight message (usually less than 2KB) is sent to the Kafka or Redpanda topic.
4. **Consumer Retrieval:** Downstream consumers read the metadata message and use the reference ID to fetch the full payload directly from object storage only when they actually need to process it.
```typescript
// If payload exceeds threshold, offload to object storage
if (body.length > 512_000) {
const ref = await uploadToObjectStore(body)
await producer.send({
topic: `saas.${resource}`,
messages: [{
key: accountId,
value: JSON.stringify({ ...metadata, payload_ref: ref }),
}],
})
} else {
// Inline path for normal-sized events
}
```
This architecture guarantees that your topics remain lean and immune to `RecordTooLargeException` failures, regardless of how much data the SaaS provider attaches to the webhook.
## Managing API Rate Limits in Event-Driven Pipelines
Event streaming solves the HTTP ingestion problem, but it introduces a new bottleneck: upstream API rate limits.
As we explored in our [guide on how mid-market SaaS teams handle API rate limits at scale](https://truto.one/how-mid-market-saas-teams-handle-api-rate-limits-webhooks-at-scale/), the moment your pipeline needs to enrich a thin webhook (e.g., fetch the full record, look up related entities), you are back to dealing with HubSpot's 100 req/sec cap, Salesforce's daily quota, or HiBob's 60-per-minute throttle. If a customer runs a bulk import in their CRM, Truto will attempt to fetch the full records for all of those events simultaneously. This can easily trigger an HTTP 429 Too Many Requests error from the upstream provider.
Be honest about what your unified layer does and does not do here: **Truto does not automatically retry, throttle, or absorb upstream 429s.**
If the upstream API returns an HTTP 429, Truto passes that exact error down to the caller or includes it in the webhook failure event. What Truto *does* do is normalize the chaotic landscape of provider rate limit headers into standardized IETF draft headers:
- `ratelimit-limit`: The total number of requests allowed in the current window.
- `ratelimit-remaining`: The number of requests left in the current window.
- `ratelimit-reset`: The timestamp when the quota will be replenished.
### Architecting Your Consumer for Backoff
Because Truto passes the 429 error and the normalized headers to you, backoff and retry strategy is owned by the caller. This is a deliberate trade-off: it keeps the platform predictable, avoids hidden queue buildup, and lets you implement domain-aware retry policies.
If you are using a stream processor, you should implement the following pattern:
1. **Detect the 429:** When your consumer processes an event that resulted in a rate limit error, read the `ratelimit-reset` header.
2. **Pause the Partition:** Do not blindly retry and burn CPU. Use Kafka's `pause()` API to temporarily stop consuming from the specific partition associated with that integrated account.
3. **Schedule the Resume:** Set a timer to call the `resume()` API exactly when the `ratelimit-reset` timestamp is reached.
4. **Isolate Tenants:** Because your Kafka topics are partitioned by `integrated_account_id`, pausing Customer A's partition (who exhausted their Salesforce API quota) will not block Customer B's events from processing.
For a deeper dive into backoff strategies, read our guide on [handling API rate limits across multiple third-party APIs](https://truto.one/best-practices-for-handling-api-rate-limits-and-retries-across-multiple-third-party-apis/).
## Building a Resilient Event-Driven Data Pipeline
The winning architecture fundamentally decouples three concerns that the DIY HTTP shim conflates:
| Concern | Owned By | Why |
|---|---|---|
| HTTP ingestion, signature verification, challenge handshakes | Unified Webhook Layer | Provider-specific, configuration-shaped |
| Payload normalization and enrichment | Unified Webhook Layer | Schema mapping is data, not code |
| Stream routing, partitioning, retention, replay | Kafka / Redpanda | What brokers are actually good at |
| Business logic, materialization, fan-out | Stream Consumers | Your domain |
Kafka and Redpanda are excellent at the third row. They are terrible at the first two, not because the engineers who built them made bad choices, but because that is not what they are for. Forcing a streaming platform to also be an HTTP gateway is what produces the spaghetti most teams eventually rip out.
The other underrated benefit of this split: **replay becomes a first-class capability.** Because every normalized event lives on a Kafka topic with a retention window, you can spin up a new consumer (say, a vector DB indexer for RAG, or a new analytics pipeline) and replay the last 7-30 days of webhook traffic without re-fetching from any provider. That is impossible if your pipeline is a direct `webhook -> handler -> Postgres` flow.
> [!NOTE]
> **Architectural Rule of Thumb:** Anything that requires per-provider knowledge should be configuration. Anything that requires throughput, durability, and replay should be infrastructure. Do not mix them.
## Where to Go From Here
Attempting to build a custom HTTP shim to bridge the gap between HTTP webhooks and TCP brokers is a trap. You will spend months writing provider-specific signature verification logic, managing state machines for payload enrichment, and fighting Kafka message size limits. If you are at the stage of designing this pipeline, take these three concrete next steps:
1. **Audit your current webhook handlers:** Count the lines of code that exist purely to handle provider-specific verification and payload normalization. That is the surface area you can move to configuration.
2. **Prototype the listener:** Stand up a 30-line HTTP service that verifies one signature scheme (from your unified layer) and produces to a single Kafka or Redpanda topic. Confirm latency and partition behavior with your real consumers.
3. **Decide where enrichment lives:** Inside the unified layer (the engine fetches the full record before delivery) or downstream in a stream processor. The first is simpler; the second gives you replay-safe enrichment at the cost of more moving parts.
By leveraging a unified webhook architecture, you offload the entirety of the HTTP edge cases. Your Kafka and Redpanda infrastructure can then do exactly what it was designed to do: route high-throughput, standardized, reliable event streams to your microservices.
> Stop building custom HTTP shims for every SaaS integration. Let Truto handle the webhook verification, challenge handshakes, normalization, and enrichment so you can focus on your core product.
>
> [Talk to us](https://cal.com/truto/partner-with-truto)