The Architect's Guide to Bi-Directional API Sync (Without Infinite Loops)

Bi-directional sync between your product and a third-party API like Salesforce or HubSpot sounds simple on a whiteboard. System A writes to System B. System B writes back to System A. Both stay in sync. In production, it is anything but simple. The moment both systems can write to the same record, you open the door to infinite loops — phantom updates bouncing between systems, draining API quotas, polluting audit logs, and silently corrupting customer data.

These are what the Valence team calls "vampire records": data entries that bounce back and forth indefinitely, feeding on your API limits without ever dying. The financial impact is not hypothetical. Gartner estimates that poor data quality costs organizations an average of $12.9 million every year in wasted resources and lost opportunities. A Validity survey of over 1,250 companies found that 44% estimate they lose more than 10% in annual revenue from low-quality CRM data. When your bidirectional sync is the source of that bad data, the damage is immediate, visible, and strictly financial.

Architecting a bi-directional integration that prevents infinite loops without requiring custom code for every third-party application is one of the hardest engineering challenges in B2B SaaS. This guide breaks down exactly why these loops occur, why the traditional workarounds fail at scale, and how to build a loop-free architecture that works across dozens of integrations.

The "Vampire Record" Problem: Why Bi-Directional Syncs Loop

The mechanics of an infinite loop are deceptively simple. Consider a standard bi-directional sync between HubSpot and your internal database:

1. A sales rep updates a contact's phone number in HubSpot.
2. HubSpot fires a contact.updated webhook to your application.
3. Your application processes the payload and updates the corresponding record in your database.
4. Your application logic detects the internal change and fires a sync job to keep all connected systems current.
5. Your system sends a PATCH request back to HubSpot with the updated contact.
6. HubSpot registers this PATCH as a new mutation and fires another contact.updated webhook back to your application.
7. Go to step 3. Repeat forever.

sequenceDiagram
    participant Rep as Sales Rep
    participant HS as HubSpot
    participant You as Your App
    Rep->>HS: Updates contact
    HS->>You: Webhook: contact.updated
    You->>You: Update internal DB
    You->>HS: PATCH /contacts/:id (sync back)
    HS->>You: Webhook: contact.updated (echo!)
    You->>You: Update internal DB
    You->>HS: PATCH /contacts/:id (sync back)
    Note over HS, You: Infinite loop established

That sounds obvious on paper. In production, the loop hides behind boring field updates like owner_id, lifecycle_stage, status, or last_activity_at. Loops typically start for four reasons:

• You key off updated_at only. Delta sync without origin metadata will happily treat your own write as a fresh inbound change.
• You cannot distinguish human writes from machine writes. Not every API exposes reliable last_modified_by metadata in webhook payloads. Some apps collapse all automated edits under one service account. Others do not surface actor metadata at all.
• Your provider retries or reorders events. Stripe does not guarantee event ordering and can send duplicates. HubSpot retries failed webhooks up to 10 times and treats responses slower than 5 seconds as failures. Slack expects a valid response in 3 seconds and retries failed deliveries up to 3 times.
• Your write path enriches data and accidentally emits a secondary change. Skinny webhook payloads often force a read-after-webhook fetch, which is good for data quality but easy to wire incorrectly.

The damage compounds fast. Every bounce means one webhook delivery, one lookup, one upsert, and potentially one retry. In Salesforce, Enterprise Edition organizations are capped at a baseline of 100,000 daily API requests, plus 1,000 per user license. An infinite loop triggered by a bulk update of just 500 contacts can exhaust a customer's entire daily Salesforce API quota in minutes. When that quota is breached, every other integration in the org stops working and the customer's operations grind to a halt.

If you are building integrations for enterprise clients, you must architect defensive mechanisms against this exact scenario. For more context on handling CRM-specific rate limits, see our guide on Architecting Real-Time CRM Syncs for Enterprise.

3 Traditional Ways to Prevent Infinite Loops (And Why They Break)

Most teams discover the loop problem after their first production incident and reach for one of three workarounds. Each works in isolation. None scales to 20+ integrations.

1. Dedicated Integration Users

The most commonly recommended fix. Create a service account (e.g., integration-bot@yourcompany.com) in each connected app. Filter your inbound triggers to ignore any update made by that user. Workato explicitly recommends this pattern, and MuleSoft's bidirectional Salesforce templates expose integration.user.id settings for the same reason.

Why it breaks:

• You need a dedicated user license per connected app, per customer. At $25–$75/month per Salesforce user license, this gets expensive fast across a customer base.
• Not every SaaS API exposes the updated_by field in webhook payloads or change events. Some only give you the record data, not who changed it.
• Enterprise IT departments frequently reject this requirement during security reviews, preferring OAuth applications with scoped permissions over provisioning synthetic human accounts.
• The exact implementation steps vary per provider, meaning you are writing bespoke loop-prevention logic for every single integration.

2. Payload Hashing and Change Detection

Before writing an update to the target system, compute a hash of the payload. Store it. When you receive a webhook from the target, compare the incoming payload's hash to the stored one. If they match, the event is an echo — skip it.

Why it breaks:

• Third-party APIs aggressively mutate data upon ingestion. You send a boolean true, the provider normalizes it to an integer 1. You send an ISO 8601 timestamp, the provider truncates the milliseconds. The echo payload's hash will not match your stored hash.
• Timestamps, auto-generated fields like last_modified_at, and system-injected metadata mean the hash almost never matches even when the human-relevant data has not changed. You end up maintaining a growing exclusion list of fields to strip before hashing.
• This pattern fails if your transformation is not deterministic. Adding a last_synced timestamp makes the same record produce a different output each time, defeating hash comparison entirely.
• Race conditions between concurrent writes make the stored hash unreliable.

Hashing is useful as a deduplication layer, not as your entire contention strategy. If two systems both legitimately edit different fields on the same record, the hash changes, and the loop-prevention problem remains.

3. Custom Flag Fields

Create a custom field on your records (like sync_source=my_app or sync_required=yes/no) to distinguish between human updates and automated updates. Only trigger syncs when the flag indicates a human change. Valence documents both record-based markers and context-based markers, and MuleSoft's Salesforce-to-Salesforce templates extend objects with external ID fields to pair records across orgs.

Why it breaks:

• You need to create and manage a custom field in every connected app. Some APIs — especially HRIS systems — do not support custom fields at all.
• Relies on the triggering system correctly filtering on the custom field. Many webhook implementations do not support field-level trigger conditions.
• You need to ensure human edits correctly reset the flag. Otherwise valid changes get suppressed.
• You are polluting your customers' production schemas with integration plumbing. You cannot reasonably ask every enterprise customer to alter their Salesforce schema, add custom properties in HubSpot, and configure new fields in Zendesk just to install your integration. The onboarding friction will kill your conversion rate.

The common thread: every traditional approach requires integration-specific code and configuration. The loop prevention itself is not hard. Doing it differently for Salesforce, HubSpot, Jira, BambooHR, Zendesk, and 40 others is what kills your engineering team. If you want to understand the true cost of these architectural compromises, read our analysis on 3 models for product integrations: a choice between control and velocity.

The Role of Webhooks in Real-Time Bidirectional Sync

Webhooks should wake up your sync engine. They should not be the only source of truth.

Provider behavior is wildly inconsistent. Microsoft Graph validates subscriptions by POSTing a validationToken and expects you to echo it back. Slack sends a challenge for URL verification. HubSpot retries failed webhook notifications up to 10 times. Stripe does not guarantee ordering and can send duplicates. Shopify explicitly recommends idempotent processing and periodic reconciliation jobs because webhook delivery is never guaranteed.

A well-architected webhook ingestion layer handles five things before your product logic runs:

1. Verification challenges — Responding to initial handshake requests (Slack, Microsoft Graph) without forwarding them to your business logic.
2. Signature validation — Verifying event authenticity using timing-safe comparison. The format varies per provider: HMAC, JWT, Basic Auth, Bearer tokens.
3. Fast acknowledgment — Returning a 200 OK immediately and queuing the actual processing. If a slow database query causes a timeout, the provider retries the webhook, artificially inflating your event volume and triggering false loops.
4. Payload normalization — Transforming provider-specific event structures into a common schema before your sync logic sees them. A contact.updated event from HubSpot looks nothing like one from BambooHR.
5. Echo detection — Determining whether the event was triggered by a human action or an automated sync, so downstream processors can skip echoes.

flowchart LR
    A[Third-Party<br>Webhook] --> B[Ingestion Layer]
    B --> C{Verification<br>Challenge?}
    C -->|Yes| D[Respond to<br>Provider]
    C -->|No| E[Signature<br>Validation]
    E --> F[Payload<br>Transform]
    F --> G{Echo<br>Event?}
    G -->|Yes| H[Drop / Log]
    G -->|No| I[Normalize to<br>Unified Schema]
    I --> J[Enqueue for<br>Sync Processing]

The key insight: the webhook ingestion layer is where loop prevention belongs, not in your application code. If your ingestion layer can normalize events from any provider into a unified schema and tag their origin, your sync logic needs one set of rules — not one per integration.

If your source system is Salesforce, Change Data Capture is worth investigating over polling or basic webhooks. CDC change events include header metadata identifying the origin of the change, which Salesforce explicitly designed for ignoring changes generated by your own client. Salesforce retains CDC events for 72 hours, giving you a replay window when consumers fall behind.

For a deeper look at building reliable ingestion pipelines, see our guide on Designing Reliable Webhooks: Lessons from Production.

Practical Echo-Filter Patterns for Webhook Ingestion

The five-step ingestion pipeline above is conceptual. Here is how each echo-filtering approach works in practice, with language-agnostic examples you can adapt to any stack.

Actor Metadata Filtering

The most reliable echo filter: inspect provider-supplied metadata that identifies who or what triggered the change. Some providers make this straightforward. Salesforce CDC events include a changeOrigin header that contains the API client identifier - if your app with client ID MyApp made the change via REST API, the field reads something like com/salesforce/api/rest/60.0;client=MyApp. HubSpot exposes a changeSource field in webhook payloads that can be INTEGRATION, API, or CRM_UI - though the exact values are inconsistent between object types.

// Pseudocode: actor-metadata echo filter

function handle_webhook(event):
    actor = extract_change_actor(event)

    if actor matches YOUR_APP_CLIENT_ID
       or actor matches YOUR_OAUTH_APP_ID
       or actor == "INTEGRATION":
        log("Echo detected via actor metadata", event.id)
        return ACK_200

    enqueue_for_processing(normalize(event))
    return ACK_200

function extract_change_actor(event):
    // Provider-specific extraction - this is the one place
    // where per-integration logic lives
    if event.provider == "salesforce_cdc":
        return event.body.ChangeEventHeader.changeOrigin
    if event.provider == "hubspot":
        return event.body.changeSource
    if event.provider == "github":
        return event.headers["X-GitHub-Hook-Installation-Target-ID"]
    return null

This works well when the provider exposes actor metadata. When it does not - and many providers don't - you need a correlation-based approach.

Correlation ID Echo Detection

Before every outbound write, generate a correlation ID and store it in a short-lived cache keyed by (resource_type, record_id). When an inbound webhook arrives, check whether its record ID correlates with a recent outbound write.

// Pseudocode: correlation-based echo filter

RECENT_WRITES = TTLCache(default_ttl = 300)  // 5-minute window

function before_outbound_write(resource_type, record_id):
    key = resource_type + ":" + record_id
    RECENT_WRITES.set(key, now())

function handle_webhook(event):
    key = event.resource_type + ":" + event.record_id

    if RECENT_WRITES.exists(key):
        log("Probable echo - write correlation hit", event.id)
        RECENT_WRITES.delete(key)  // Consume the marker
        return ACK_200

    enqueue_for_processing(normalize(event))
    return ACK_200

The TTL is a trade-off. Too short and you miss echoes from slow providers (HubSpot can take 5+ seconds to fire a webhook after a write). Too long and you suppress legitimate rapid human edits. Start at 5 minutes and tune based on observed provider latency for your specific integrations.

Event-ID Deduplication

Some providers include an idempotency key or delivery ID in webhook payloads - Stripe's event.id, GitHub's X-GitHub-Delivery header. Store processed event IDs and reject duplicates:

// Pseudocode: event-ID deduplication

PROCESSED_EVENTS = PersistentSet(ttl = 86400)  // 24-hour retention

function handle_webhook(event):
    event_id = event.headers.get("x-webhook-delivery-id")
              or event.body.get("event_id")
              or event.body.get("id")

    if event_id and PROCESSED_EVENTS.contains(event_id):
        log("Duplicate event - already processed", event_id)
        return ACK_200

    PROCESSED_EVENTS.add(event_id)
    enqueue_for_processing(normalize(event))
    return ACK_200

This catches retries and true duplicates but does not catch echoes on its own - a provider will assign a new event ID to the echo webhook. Combine it with actor metadata or correlation ID detection for full coverage.

Origin Tagging: Tracking Where Every Write Comes From

Echo filtering at ingestion catches most loops. For defense-in-depth, your sync engine should know the origin of every data mutation it processes. Origin tagging means attaching metadata to every write that flows through your pipeline so downstream logic can make informed decisions about whether to propagate a change.

What to Tag

Every record mutation that passes through your sync pipeline should carry:

Where to Store Origin Tags

Option A: Dedicated sync metadata table (recommended). Store origin tags in a separate table keyed by (resource_type, record_id, external_system). This keeps your core data model clean and lets you query sync history independently.

CREATE TABLE sync_metadata (
    resource_type   TEXT,
    record_id       TEXT,
    external_system TEXT,
    sync_source     TEXT,
    sync_actor      TEXT,
    correlation_id  TEXT,
    synced_at       TIMESTAMP,
    field_hash      TEXT,
    PRIMARY KEY (resource_type, record_id, external_system)
);

Option B: Inline metadata on the record. Add _sync_source and _sync_updated_at fields directly to your domain records. Simpler to implement but mixes integration plumbing with business data - the same "schema pollution" problem we flagged with custom flag fields.

Option A is almost always better. It scales to many integrations without schema changes and gives you an audit trail for debugging loop incidents.

Propagating Origin Through the Pipeline

The origin tag should travel with the record from ingestion to write:

// Pseudocode: origin propagation

function process_inbound_event(event):
    record = normalize(event)
    record.meta = {
        sync_source:    event.provider,
        sync_actor:     classify_actor(event),
        sync_direction: "inbound",
        sync_timestamp: now()
    }

    upsert_to_internal_db(record)
    save_sync_metadata(record)

    // Key rule: never propagate machine-originated changes
    if record.meta.sync_actor == "sync_engine":
        return  // Stop here - do not fan out

    for target in get_sync_targets(record, exclude = event.provider):
        correlation_id = generate_uuid()
        RECENT_WRITES.set(record.type + ":" + record.id, correlation_id)
        record.meta.sync_direction = "outbound"
        record.meta.correlation_id = correlation_id
        write_to_target(record, target)
        save_sync_metadata(record)

The single most important rule: never propagate a change that originated from a sync engine to another sync target without explicit intent. This one rule eliminates the most common source of infinite loops.

Idempotent Writes: Key Patterns and Persistence

Idempotency is your last line of defense. If echo filtering and origin tagging both miss, an idempotent write ensures the echo produces no side effects. The goal: the same write operation, executed any number of times, always produces the same result.

External ID Upserts

Most third-party APIs support upsert-by-external-ID. Instead of separate create/update code paths, always upsert with a stable identifier:

// Pseudocode: external ID upsert

function sync_record_to_target(record, target_system):
    external_id = generate_external_id(record)

    response = target_system.upsert(
        resource         = record.type,
        external_id_field = "your_app_external_id",
        external_id_value = external_id,
        data             = record.fields
    )
    return response

function generate_external_id(record):
    // Use your internal ID directly
    return "myapp_" + record.id

    // Or composite key for multi-tenant scenarios
    return record.tenant_id + "_" + record.type + "_" + record.id

Salesforce supports upsert via external ID fields natively. HubSpot supports it via unique_property_key. For APIs that lack native upserts, implement search-then-create-or-update with optimistic concurrency.

Deterministic Idempotency Keys

For APIs that accept idempotency keys (Stripe, many payment and accounting providers), generate deterministic keys from the operation's content:

// Pseudocode: deterministic idempotency key

function generate_idempotency_key(operation):
    // Inputs: target system, resource, record, action, field values
    content = operation.target
              + ":" + operation.resource
              + ":" + operation.record_id
              + ":" + operation.action

    field_hash = sha256(canonical_json(operation.fields))

    return sha256(content + ":" + field_hash)

A deterministic key means retrying the same logical operation always hits the same idempotency slot. If the data has not changed, the write is a no-op.

Field-Level Diff Before Write

Before issuing any write, compare the fields you intend to send against the target's current state. Only send fields that actually differ:

// Pseudocode: field-level diff

function sync_with_diff(record, target_system):
    current = target_system.get(record.type, record.external_id)

    if current is null:
        return target_system.create(record.type, record.fields)

    changed_fields = {}
    for field, value in record.fields:
        if normalize_value(current[field]) != normalize_value(value):
            changed_fields[field] = value

    if changed_fields is empty:
        log("No field changes detected - skipping write", record.id)
        return SKIP

    return target_system.update(record.type, current.id, changed_fields)

function normalize_value(value):
    // Handle the type coercion quirks that break payload hashing
    if value is boolean:  return lowercase(string(value))
    if value is datetime: return truncate_to_seconds(value)
    if value is string:   return trim(value)
    return string(value)

The normalize_value function handles the type coercion mismatches that make whole-payload hashing unreliable (booleans vs. integers, timestamp precision, trailing whitespace). This is targeted normalization on fields you control, not a brittle full-payload hash.

How to Architect Loop-Free Syncs Using Truto RapidBridge

Bidirectional sync requires an architecture that inherently understands the concept of origin, time, and state — without relying on integration-specific code. Truto approaches this problem through two complementary mechanisms: a declarative webhook ingestion layer for real-time events, and RapidBridge for incremental polling-based reconciliation.

Filtering Echo Events at the Edge

Truto's unified webhook ingestion layer uses JSONata expressions to normalize and filter events before they reach your core logic. When a third-party webhook hits the Truto ingress, it passes through a transformation pipeline where you can inspect the raw payload and selectively drop events based on provider-specific metadata — all defined as configuration, not application code.

webhooks:
  hubspot: |
    (
      $is_echo := body.properties.last_modified_source = "INTEGRATION";
      $is_echo ? null : {
        "event_type": "updated",
        "raw_event_type": body.subscriptionType,
        "raw_payload": $,
        "resource": "crm/contacts",
        "method": "get",
        "method_config": {
          "id" : body.objectId
        }
      }
    )

The JSONata expression evaluates the raw payload. If it detects that the modification source was an integration, it returns null, dropping the event at the edge. Truto never forwards the echo to your system.

If the event is a valid human update, Truto enqueues it and triggers data enrichment. Many webhook payloads are "skinny" — they only contain an ID. Through the method_config block, Truto dynamically fetches the fully enriched unified data model directly from the provider API and delivers a standardized payload to your system. You always get the complete, up-to-date record without making secondary API calls from your application that could trigger rate limits.

sequenceDiagram
    participant Human
    participant Provider API
    participant Truto Edge
    participant Your SaaS
    
    Your SaaS->>Provider API: PATCH /resource (Sync)
    Provider API->>Truto Edge: Webhook (resource.updated)
    Note over Truto Edge: JSONata evaluates payload.<br>Detects automated source.
    Truto Edge--xYour SaaS: Event dropped. Loop prevented.
    
    Human->>Provider API: Manual Update
    Provider API->>Truto Edge: Webhook (resource.updated)
    Note over Truto Edge: JSONata evaluates payload.<br>Detects human source.
    Truto Edge->>Provider API: Fetch full enriched record
    Provider API-->>Truto Edge: Full record data
    Truto Edge->>Your SaaS: Unified Webhook Event

This architecture completely isolates your application logic from the nuances of provider-specific echo behavior. Your system only receives unified events that represent genuine, actionable data mutations. Adding a new integration's webhook support does not require deploying new application logic — it requires defining a new JSONata mapping expression.

Incremental Delta Syncs for Reconciliation

Webhooks handle the real-time path. But most production bidirectional syncs also need a polling-based reconciliation pass — a scheduled job that catches anything webhooks missed. And they will miss things; webhooks are best-effort delivery from most providers. Shopify's own documentation explicitly recommends reconciliation jobs for this reason.

This is where incremental delta syncs become the primary mechanism for loop prevention on the polling side. RapidBridge's native previous_run_date tracking allows declarative incremental syncs without writing custom timestamp logic:

{
  "resource": "crm/contacts",
  "method": "list",
  "query": {
    "updated_at": {
      "gt": "{{previous_run_date}}"
    }
  }
}

The previous_run_date is a system-managed attribute resolved at runtime to the timestamp of the last successful sync run for this specific resource and connected account. On the first run, it defaults to epoch (1970-01-01T00:00:00.000Z), pulling a full initial sync. Every subsequent run pulls only the delta.

This pattern inherently prevents loops in the polling path:

• Write-then-advance: After your sync job writes to the target, the echoed record's updated_at will be after the current previous_run_date but before the next run's watermark (since the current run completes and advances the marker). The echo falls in a narrow window and gets naturally filtered out.
• Idempotent writes: Combining delta sync with idempotent upserts using external IDs or composite keys means even if an echo slips through, it produces no net change to the target record.

For resources that depend on parent records (like ticket comments that depend on tickets), RapidBridge supports declarative dependencies:

{
  "resources": [
    {
      "resource": "ticketing/tickets",
      "method": "list",
      "query": {
        "updated_at": { "gt": "{{previous_run_date}}" }
      }
    },
    {
      "resource": "ticketing/comments",
      "method": "list",
      "depends_on": "ticketing/tickets",
      "query": {
        "ticket_id": "{{resources.ticketing.tickets.id}}"
      }
    }
  ]
}

No custom code to maintain timestamps. No per-integration watermark logic. No database tables you need to manage for tracking sync state.

Handling the Initial Sync After Account Connection

A common trap when solving the bulk extraction problem: a customer connects their Salesforce account, and you need to pull all existing contacts before the bidirectional sync starts. If you do not handle this carefully, the initial data load triggers webhooks on the target system, which bounce back and create duplicates.

Truto supports an event-driven kickoff. When an integrated account becomes active, a lifecycle event (integrated_account:active) is emitted. Your system listens for this event and triggers the initial sync job:

curl -X POST https://api.truto.one/sync-job-run \
  -H 'Authorization: Bearer <api_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "sync_job_id": "<sync_job_id>",
    "integrated_account_id": "<from_webhook_payload>",
    "webhook_id": "<webhook_id>"
  }'

Because the initial run uses previous_run_date set to epoch, it pulls everything. Subsequent runs pull only deltas. No special "initial load" code path required. You can also force a full re-sync at any time by passing ignore_previous_run: true — useful for recovery after incidents or permission changes.

Delta Sync Reconciliation: Watermarks, Scheduling, and Pseudocode

The sections above describe what to do. Here is the complete reconciliation job that ties watermarks, echo filtering, and conflict resolution together in one executable flow.

Watermark Format and Storage

A watermark tracks the high-water mark of the last successful sync. The minimum viable schema:

CREATE TABLE sync_watermarks (
    sync_job_id     TEXT,
    account_id      TEXT,
    resource_type   TEXT,
    last_successful TIMESTAMP,
    last_attempted  TIMESTAMP,
    status          TEXT,        -- 'completed', 'failed', 'running'
    record_count    INTEGER,
    PRIMARY KEY (sync_job_id, account_id, resource_type)
);

The last_successful timestamp only advances when a run completes without errors. If a run fails mid-way, the next run restarts from the same watermark - no data loss, no skipped records.

If you are using Truto's RapidBridge, you get this behavior for free. The previous_run_date attribute works exactly this way - it only advances on successful completion, and you can override it with ignore_previous_run: true for recovery scenarios.

Scheduling Strategy

How often you run reconciliation depends on your latency tolerance and API budget:

Avoid running reconciliation more frequently than every 15 minutes. Each run consumes API quota, and aggressive polling is the fastest way to hit rate limits on providers like Salesforce and HubSpot.

Full Reconciliation Job Pseudocode

// Pseudocode: delta sync reconciliation with conflict handling

function run_reconciliation(sync_job, account):
    watermark = load_watermark(sync_job.id, account.id, sync_job.resource)

    if watermark is null:
        since = EPOCH                          // First run - full sync
    else:
        since = watermark.last_successful

    run_started_at = now()
    records_processed = 0
    errors = []

    // Step 1: Fetch changed records from source since watermark
    try:
        changed_records = source_api.list(
            resource = sync_job.resource,
            filter   = { "updated_at": { "gt": since } },
            sort     = "updated_at:asc"        // Oldest first
        )
    except RateLimitError:
        log("Rate limited - will retry next cycle")
        save_watermark(sync_job.id, account.id, sync_job.resource, {
            last_attempted: run_started_at,
            status: "rate_limited"
        })
        return

    // Step 2: Process each changed record
    for record in changed_records:
        try:
            // Check if this record was recently written by us (echo)
            key = sync_job.resource + ":" + record.id
            if RECENT_WRITES.exists(key):
                log("Skipping echo record", record.id)
                continue

            // Step 3: Check target for conflicts
            target_record = target_api.get(
                sync_job.resource, record.external_id
            )

            if target_record is not null:
                resolution = resolve_conflict(record, target_record)
                if resolution == SKIP:      continue
                if resolution == MERGE:     record = merge_fields(record, target_record)
                // if OVERWRITE, proceed as normal

            // Step 4: Write with idempotency
            sync_with_diff(record, target_api)
            save_sync_metadata(record)
            records_processed = records_processed + 1

        except error:
            errors.append({ record_id: record.id, error: string(error) })
            continue   // Don't fail the whole job for one record

    // Step 5: Advance watermark only on full success
    if errors is empty:
        save_watermark(sync_job.id, account.id, sync_job.resource, {
            last_successful: run_started_at,
            last_attempted:  run_started_at,
            status:          "completed",
            record_count:    records_processed
        })
    else:
        save_watermark(sync_job.id, account.id, sync_job.resource, {
            last_attempted: run_started_at,
            status:         "partial_failure",
            record_count:   records_processed
        })
        alert("Reconciliation had errors", errors)

Two details worth highlighting. First, sorting by updated_at ascending means if the job dies partway through, you can resume from where you stopped without reprocessing earlier records. Second, the watermark advances to run_started_at (when the job began), not to the latest record's timestamp. This prevents a subtle gap: records modified between job start and job completion would be missed if you advanced to the latest record's updated_at.

A Reference Architecture for Loop-Free Bidirectional Sync

Putting it all together, here is the architecture that works in production:

flowchart TB
    subgraph Real-Time Path
        W1[Third-Party Webhook] --> IL[Unified Ingestion Layer]
        IL --> VF[Verify + Transform]
        VF --> EM[Event Mapping<br>via JSONata]
        EM --> EF{Echo<br>Filter}
        EF -->|Clean Event| Q1[Event Queue]
        EF -->|Echo| DROP[Drop]
    end
    subgraph Scheduled Path
        CRON[Scheduled Trigger] --> DS[Delta Sync Job]
        DS --> API[Unified API<br>with previous_run_date]
        API --> Q2[Data Queue]
    end
    Q1 --> SL[Your Sync Logic]
    Q2 --> SL
    SL --> TW[Write to Target<br>via Unified API]

Key principles:

• Two paths, one contract. Both the real-time webhook path and the scheduled delta path produce the same unified data schema. Your sync logic has one code path, not two.
• Filter at ingestion, not in business logic. Echo detection happens before events reach your application. Your engineers never see phantom updates.
• Watermarks, not wall clocks. The previous_run_date pattern ties reconciliation to successful completion, not arbitrary time windows. If a sync job fails, it retries from the same watermark — no data loss.
• Idempotent writes everywhere. Use external IDs or composite keys for upserts. If an echo sneaks through both filters, the write is a no-op.

Conflict Resolution: Choosing an Ownership Model

When two systems both write to the same record within a sync window, you have a conflict. Echo filtering and watermarks prevent loops, but they don't tell you which version of the data wins. That's the job of your ownership model.

Object-Level Ownership (Simplest)

One system is the authoritative source for an entire resource type. All contact changes originate in the CRM. All ticket changes originate in the helpdesk. Conflicts are impossible because writes only flow one direction per resource.

// Pseudocode: object-level ownership

OWNERSHIP = {
    "crm/contacts":       "hubspot",
    "ticketing/tickets":  "internal",
    "hris/employees":     "bamboohr"
}

function should_sync(record, source, target):
    owner = OWNERSHIP[record.type]
    return source == owner    // Only sync from the owner

This model works for most B2B integrations and eliminates entire categories of sync bugs. The trade-off: users cannot edit the record in the non-authoritative system and have changes flow back.

Record-Level Ownership

Each individual record has an owner. Contacts created in HubSpot are owned by HubSpot. Contacts created in your app are owned by your app. Ownership can transfer via explicit user action.

// Pseudocode: record-level ownership

function resolve_conflict(source_record, target_record):
    owner = get_record_owner(source_record.type, source_record.id)

    if owner == source_record.origin:   return OVERWRITE
    if owner == target_record.origin:   return SKIP
    return LAST_WRITE_WINS              // Fallback

Track ownership in your sync_metadata table with an owned_by column. Record-level ownership handles the common case where both systems create records, but adds complexity to conflict resolution.

Field-Level Ownership (Most Granular)

Different systems own different fields on the same record. HR owns job_title and department. Your app owns engagement_score and last_contacted. This is the most flexible model but also the hardest to debug.

// Pseudocode: field-level ownership

FIELD_OWNERSHIP = {
    "crm/contacts": {
        "hubspot":  ["first_name", "last_name", "email", "phone"],
        "internal": ["score", "segment", "last_contacted", "tags"]
    }
}

function merge_fields(source_record, target_record):
    merged = {}
    source_fields = FIELD_OWNERSHIP[source_record.type][source_record.origin]

    for field in all_fields(source_record, target_record):
        if field in source_fields:
            merged[field] = source_record.fields[field]
        else:
            merged[field] = target_record.fields[field]

    return merged

Field-level ownership is worth the complexity when your product genuinely enriches records that live in a customer's CRM. A sales intelligence tool that writes engagement scores to HubSpot contacts but never touches the contact's name or email is a natural fit.

Last-Write-Wins (Fallback)

When you cannot determine ownership, timestamp comparison is the pragmatic default. Compare updated_at from both systems and let the more recent write win.

The risk: clock skew between systems can cause the wrong version to win. Mitigate this by using your own sync_timestamp from origin tagging as the tiebreaker rather than relying on the provider's reported timestamp.

Testing, Monitoring, and Runbook for Loop Incidents

Testing Scenarios

Before shipping any bidirectional sync to production, run these scenarios in a staging environment:

1. Basic echo test. Write a record to System A via your sync engine. Verify the resulting webhook from System A is detected as an echo and dropped. Confirm no write is issued back to System A.
2. Rapid successive edits. Update the same record five times in 10 seconds. Verify your system processes all five without duplicates and without triggering a loop.
3. Simultaneous cross-system edit. Update the same record in both System A and your app within the same second. Verify your conflict resolution model produces the expected result.
4. Provider retry simulation. Send the same webhook payload three times with the same event ID. Verify only one is processed.
5. Watermark recovery. Kill a reconciliation job mid-run. Verify the next run resumes from the correct watermark without skipping or duplicating records.
6. Initial sync with existing data. Connect an account that already has 1,000+ records. Verify the bulk load completes without triggering echo loops.
7. Rate limit degradation. Trigger a sync that exceeds the provider's rate limit. Verify your system backs off, preserves the watermark, and resumes correctly on the next run.

Monitoring Checklist

Track these metrics in production to catch loops before they cause damage:

The single most reliable signal of a loop: outbound API calls per account spiking above baseline. If one account suddenly consumes 10x its normal API volume, you almost certainly have a loop.

Incident Runbook: Detecting and Stopping a Loop

When you suspect a loop in production:

Step 1: Confirm the loop. Pull webhook delivery logs for the affected account. Look for a repeating pattern: inbound webhook, outbound write, inbound webhook, outbound write - with timestamps separated by seconds, not minutes.

Step 2: Stop the bleeding. Pause the affected account's outbound sync immediately. Most loops are one-directional (System A → Your App → System A), so pausing outbound writes to the source system breaks the cycle.

Step 3: Assess the damage. Review affected records for data corruption. Common symptoms: fields overwritten with stale values, audit logs filled with bot edits, API quota consumption spiked.

Step 4: Identify the root cause. The usual suspects:

• A webhook mapping that does not filter on change origin
• A provider API update that changed the metadata format (e.g., modified_by moved from headers to body)
• A new field added to the sync that triggers updated_at changes on write
• The correlation ID cache expired before the echo arrived (TTL too short)

Step 5: Fix and validate. Deploy the fix to staging. Run the basic echo test and the rapid successive edit test from the list above. Only re-enable the account's outbound sync after confirming the loop is broken in staging.

Step 6: Post-mortem. Document which defense layer failed (echo filter, origin tag, idempotent write) and why. Add a regression test for the specific scenario.

Stop Writing Integration-Specific Code for Contention Resolution

Let's be honest about what the traditional approaches really require at scale.

If you support 30 integrations across CRM, HRIS, ATS, and ticketing categories using the dedicated-integration-user approach, you need 30 different user provisioning workflows (each vendor's admin panel works differently), 30 different trigger filter configurations (SOQL WHERE clauses for Salesforce, JQL for Jira, custom query parameters for everyone else), and 30 different test suites to verify loop prevention still works after each vendor's API update. That is not an integration strategy. That is a full-time job for a team doing nothing but maintaining loop-prevention boilerplate instead of building the integrations your B2B sales team actually asks for.

The alternative is pushing contention resolution into an abstraction layer that handles it generically. Every integration — whether Salesforce, HubSpot, BambooHR, or Linear — runs through the same execution pipeline. Webhook verification, payload normalization, event mapping, data enrichment, and outbound delivery all use declarative configuration (JSONata expressions and structured schema mappings) rather than per-integration code branches. This is where a unified API approach pays off — not because it magically eliminates the complexity, but because it concentrates it in one place instead of smearing it across 30 codebases.

This means:

• No custom SOQL queries to filter integration-user updates in Salesforce.
• No per-integration webhook handlers with bespoke loop-detection logic.
• No custom fields polluting your customers' CRM schemas.
• Adding a new integration means adding configuration, not deploying new code.

The honest caveat is that zero integration-specific code does not eliminate contention entirely. You still need to decide on ownership models — object-level, record-level, or field-level — for each synced resource. The difference is where you implement that logic. With a declarative integration layer, you implement it once at the platform boundary instead of scattering it across webhook handlers, cron jobs, and CRM triggers.

What to Do Next

If you are currently debugging an infinite loop in production, here is a triage checklist:

1. Pick an ownership model for each synced object. Decide whether the source of truth operates at the object level, record level, or field level.
2. Identify the echo source. Check your webhook logs for events that arrive within seconds of your outbound writes. The timestamp and event metadata will tell you if it is a bounce-back.
3. Split webhook ingestion from business logic. Verify, normalize, enqueue, and acknowledge fast. If your webhook handler does processing synchronously, a slow query will cause provider retries that amplify the loop.
4. Switch from full syncs to delta syncs. If your scheduled jobs pull the full dataset every time, you are multiplying echo risk and burning API quota. Implement updated_at > last_successful_run filtering immediately.
5. Evaluate your abstraction layer. If you are maintaining loop-prevention code for more than 5 integrations, the economics favor pushing that logic into a unified API layer. The maintenance cost of bespoke solutions grows linearly; a declarative approach stays flat.

Bi-directional sync is a hard problem. But it is a solved hard problem — the patterns exist, and you do not need to reinvent them for every integration your product supports.