How to Normalize API Pagination and Error Handling (With Code Examples)

You are evaluating an enterprise deal. The prospect demands integrations with their custom CRM, an obscure HRIS, and a legacy ticketing system. The initial authentication and GET requests take your engineering team a few hours. Then the reality of maintaining those connections sets in.

Every integration team eventually reaches the same conclusion: the hard part of building third-party integrations is not the first HTTP request. It is the ten thousand tiny inconsistencies that hide underneath. One API uses cursor pagination with base64-encoded tokens, another uses offset with a 10,000-row hard cap, a third returns HTTP 200 with an error nested three levels deep, and a fourth invents its own error envelope that changes shape depending on which endpoint you hit.

This guide breaks down the architectural patterns required to normalize third-party APIs. We will cover exactly how to normalize API pagination and error handling across dozens of third-party APIs with concrete code examples, the RFC 7807 standard, JSONata-based error expressions, and the architectural patterns that let you stop writing per-vendor plumbing. It is aimed at staff engineers, lead architects, and product engineers who own integration surface area and are tired of shipping bespoke retry logic for every new connector.

The Hidden Cost of Inconsistent APIs

When you integrate with a single API, writing custom logic to handle its specific quirks is entirely manageable. When you integrate with fifty APIs, that same approach creates a brittle, unmaintainable codebase.

According to industry research, companies use an average of 106 SaaS applications globally, and large enterprises regularly exceed 130. If your product needs to sync with even 10% of a customer's stack, you are already staring down a dozen integrations—each with its own quirks.

Consider a standard data synchronization pipeline. Your application needs to pull newly created contacts from a third-party CRM every five minutes.

• API A (Salesforce) uses SOQL queries with a query locator that expires.
• API B (HubSpot) uses cursor-based pagination with a vidOffset.
• API C (Legacy ERP) uses strict limit and offset integers, but fails if the offset exceeds 10,000.
• API D (Bespoke Marketing Tool) returns a 200 OK response, but embeds {"success": false, "error": "Too many requests"} inside the JSON body.

If your background workers contain specific if/else statements for each of these scenarios, your architecture is already failing. Every time an upstream provider changes their API version, introduces a new rate limit threshold, or modifies their error schema, your sync jobs will fail silently.

The operational tail is worse than the initial build. Engineers spend hundreds of hours annually per integration just diagnosing and resolving synchronization issues caused by unhandled edge cases, schema drift, and deprecated endpoints. Multiply that across a real integration portfolio and you have a full engineering team maintaining plumbing instead of building product.

The fix is not to "write better wrappers." The fix is to treat pagination and errors as normalized primitives at the platform layer, with declarative mappings per provider and a single, generic execution pipeline that never contains an if provider === 'salesforce' branch. To build scalable integrations, you must push the integration-specific logic to the absolute edge of your system and normalize the data into a generic format before it ever reaches your core application logic. For the operational side of this problem—retries, monitoring, breaking changes—read our companion piece on how to normalize pagination and error handling across 50+ APIs.

Normalizing API Pagination: Offset vs. Cursor

Pagination normalization is the process of exposing a single, consistent iteration interface to your callers, regardless of whether the upstream API uses ?page=2&limit=100, ?offset=200&limit=100, ?cursor=eyJpZCI6MTIzfQ, or a Link header with rel="next".

Before you can build an abstraction layer to solve how unified APIs handle pagination differences across REST APIs, you have to understand the four models in the wild:

The Problem with Offset Pagination

Offset pagination relies on two parameters: limit (how many records to return) and offset (how many records to skip). It is the default because it is easy to implement, but it silently degrades at scale.

When you request limit=50&offset=100000, the database engine must still walk 100,000 rows, discard them, and return the next 50. This leads to query timeouts and linear latency growth on massive datasets.

More dangerously, offset pagination is vulnerable to concurrent data mutations. If a record is inserted or deleted while you are paginating, the offset shifts. If a new row is inserted at position 50, the record that was at offset 100 is now at offset 101. This results in your sync job either skipping records entirely or processing duplicate records.

The Stability of Cursor Pagination

Cursor-based pagination (or keyset pagination) solves the offset problem by returning an opaque pointer to a specific indexed row in the database. The client passes this pointer (cursor, after, page_token) in the next request.

Because cursor pagination uses indexed sorting for fast, stable retrieval, it does not degrade at scale. Retrieval becomes an indexed seek instead of a scan, and results stay stable even when the underlying dataset changes. It is immune to data shifting caused by concurrent inserts or deletes, making it the superior choice for live feeds and data synchronization. Salesforce's Bulk API, Stripe, and GitHub all use variations of this pattern for exactly this reason.

The Normalization Goal

Your internal systems should only ever speak one pagination language. Your callers should not care which model the upstream uses. They should get:

• A consistent response envelope with data and next_cursor.
• An opaque cursor string they can pass back to fetch the next page.
• Deterministic behavior when next_cursor is null (end of results).

Everything else—offset math, base64 decoding, Link header parsing—lives inside your normalization layer.

Architecture 1: The Opaque Cursor Wrapper

The most effective initial pattern is to force all APIs into a cursor-based interface. For APIs that only support offset, you can encode the offset integer into a base64 string and treat it as an opaque cursor.

sequenceDiagram
    participant Worker as Background Worker
    participant Wrapper as Normalization Layer
    participant Upstream as Upstream API

    Worker->>Wrapper: GET /contacts?limit=100
    Wrapper->>Upstream: GET /api/v1/contacts?offset=0&limit=100
    Upstream-->>Wrapper: 200 OK (100 records)
    Wrapper-->>Worker: 200 OK + { next_cursor: "b2Zmc2V0PTEwMA==" }
    
    Worker->>Wrapper: GET /contacts?limit=100&cursor="b2Zmc2V0PTEwMA=="
    Note over Wrapper: Decodes cursor to offset=100
    Wrapper->>Upstream: GET /api/v1/contacts?offset=100&limit=100
    Upstream-->>Wrapper: 200 OK (100 records)
    Wrapper-->>Worker: 200 OK + { next_cursor: "b2Zmc2V0PTIwMA==" }

Below is a TypeScript implementation of an imperative normalization wrapper. This class intercepts requests from your internal workers, translates the generic cursor into the provider-specific format, executes the request, and normalizes the response.

interface NormalizedPage<T> {
  data: T[];
  next_cursor: string | null;
  has_more: boolean;
}
 
interface PaginationConfig {
  type: 'offset' | 'cursor';
  limitKey: string;
  cursorKey: string;
  responseCursorPath: string;
}
 
class UnifiedPaginationClient {
  constructor(
    private baseUrl: string,
    private headers: Record<string, string>,
    private config: PaginationConfig
  ) {}
 
  async fetchPage(limit: number, cursor?: string): Promise<NormalizedPage<any>> {
    const url = new URL(this.baseUrl);
    url.searchParams.set(this.config.limitKey, limit.toString());
 
    if (cursor) {
      if (this.config.type === 'offset') {
        // Decode the opaque cursor back to an offset integer
        const offsetValue = Buffer.from(cursor, 'base64').toString('ascii');
        url.searchParams.set(this.config.cursorKey, offsetValue);
      } else {
        // Pass the cursor directly
        url.searchParams.set(this.config.cursorKey, cursor);
      }
    } else if (this.config.type === 'offset') {
      // Initial request for offset pagination
      url.searchParams.set(this.config.cursorKey, '0');
    }
 
    const response = await fetch(url.toString(), { headers: this.headers });
    
    if (!response.ok) {
      throw new Error(`API Error: ${response.status}`);
    }
 
    const data = await response.json();
    return this.normalizeResponse(data, limit, cursor);
  }
 
  private normalizeResponse(data: any, limit: number, currentCursor?: string): NormalizedPage<any> {
    const records = data.results || data.items || data.data || [];
    let nextCursor: string | null = null;
    let hasMore = false;
 
    if (this.config.type === 'offset') {
      // Calculate the next offset and encode it as a cursor
      const currentOffset = currentCursor 
        ? parseInt(Buffer.from(currentCursor, 'base64').toString('ascii'), 10) 
        : 0;
      
      if (records.length === limit) {
        const nextOffset = currentOffset + limit;
        nextCursor = Buffer.from(nextOffset.toString()).toString('base64');
        hasMore = true;
      }
    } else {
      // Extract the provider's cursor using the configured path
      const rawCursor = this.extractValue(data, this.config.responseCursorPath);
      if (rawCursor) {
        nextCursor = rawCursor;
        hasMore = true;
      }
    }
 
    return {
      data: records,
      next_cursor: nextCursor,
      has_more: hasMore
    };
  }
 
  private extractValue(obj: any, path: string): string | null {
    return path.split('.').reduce((acc, part) => acc && acc[part], obj) || null;
  }
}

Architecture 2: The Declarative Execution Pipeline

While the imperative wrapper above is a huge step up from writing custom logic in your workers, it still relies on runtime code configurations. When you scale to dozens of APIs, writing and maintaining these classes becomes a bottleneck.

The ultimate pattern is to define a declarative pagination config per provider, then run every request through a single generic executor. The config is treated as data, not code. Adding a new provider means adding a row to a config store, not shipping a code release.

// Declarative pagination config - one per provider, stored as data, not code
type DeclarativePaginationConfig =
  | { type: 'offset'; offsetParam: string; limitParam: string; pageSize: number }
  | { type: 'cursor'; cursorParam: string; cursorPath: string; limitParam: string; pageSize: number }
  | { type: 'link_header'; rel: string; pageSize: number };
 
const configs: Record<string, DeclarativePaginationConfig> = {
  hubspot_contacts: {
    type: 'cursor',
    cursorParam: 'after',
    cursorPath: 'paging.next.after',
    limitParam: 'limit',
    pageSize: 100,
  },
  zendesk_tickets: {
    type: 'offset',
    offsetParam: 'page',
    limitParam: 'per_page',
    pageSize: 100,
  },
  github_issues: {
    type: 'link_header',
    rel: 'next',
    pageSize: 100,
  },
};
 
// Generic executor - one code path for every provider
async function fetchPage(
  resource: string,
  cursor: string | null,
): Promise<{ data: unknown[]; next_cursor: string | null }> {
  const cfg = configs[resource];
  const url = new URL(providerUrl(resource));
 
  if (cfg.type === 'offset') {
    // Here you would integrate the base64 decoding logic from Architecture 1
    const page = cursor ? parseInt(Buffer.from(cursor, 'base64').toString('ascii'), 10) : 1;
    url.searchParams.set(cfg.offsetParam, String(page));
    url.searchParams.set(cfg.limitParam, String(cfg.pageSize));
    const res = await httpGet(url);
    const nextPage = res.data.length === cfg.pageSize ? Buffer.from(String(page + 1)).toString('base64') : null;
    return { data: res.data, next_cursor: nextPage };
  }
 
  if (cfg.type === 'cursor') {
    if (cursor) url.searchParams.set(cfg.cursorParam, cursor);
    url.searchParams.set(cfg.limitParam, String(cfg.pageSize));
    const res = await httpGet(url);
    const next = getPath(res.body, cfg.cursorPath) ?? null;
    return { data: res.data, next_cursor: next };
  }
 
  if (cfg.type === 'link_header') {
    const target = cursor ?? url.toString();
    const res = await httpGet(target);
    const next = parseLinkHeader(res.headers.link)?.[cfg.rel] ?? null;
    return { data: res.data, next_cursor: next };
  }
 
  throw new Error('Unknown pagination type');
}

The caller always writes the identical loop:

let cursor: string | null = null;
do {
  const page = await fetchPage('hubspot_contacts', cursor);
  await processBatch(page.data);
  cursor = page.next_cursor;
} while (cursor);

This is the same architectural pattern Truto uses to run 100+ integrations through one generic execution pipeline. For a deeper walkthrough of the pagination modeling itself, see our guide on building a declarative pagination system.

Standardizing API Error Handling with RFC 7807

Pagination is only half the battle. When your background workers execute thousands of requests per minute, errors are inevitable. Tokens expire, permissions change, and malformed data triggers validation failures.

The challenge is that every SaaS provider returns errors differently. Here is the reality of the chaos you are normalizing away:

// Provider A: HTTP 200 with error nested in body
{ "status": "error", "errorCode": "AUTH_001", "message": "Token expired" }
 
// Provider B: HTTP 400 with a Salesforce-style array
[{ "errorCode": "INVALID_SESSION_ID", "message": "Session expired or invalid" }]
 
// Provider C: HTTP 401 with a nested envelope
{ "error": { "code": 401, "reason": "unauthorized", "details": [...] } }
 
// Provider D: HTML page because the load balancer intercepted the request
<html><body>502 Bad Gateway</body></html>

If your worker expects a 400 Bad Request to contain {"error": "invalid_email"}, but the provider actually returns Provider C's nested envelope, your worker will crash trying to parse the response. Without normalization, every caller has to know every shape. With normalization, they parse one envelope.

RFC 7807 (Problem Details for HTTP APIs) is the industry standard for normalizing API error responses into a consistent, machine-readable format. It defines a small, extensible schema that every client can parse without provider-specific glue code.

An RFC 7807 compliant error looks like this:

{
  "type": "https://api.example.com/errors/rate-limit",
  "title": "Rate limit exceeded",
  "status": 429,
  "detail": "You have exceeded 100 requests per minute for this API key.",
  "instance": "/v1/contacts"
}

The five core fields (type, title, status, detail, instance) are enough to build predictable retry, logging, and alerting logic. You can easily extend the object with custom fields like provider_error_code or reauth_required without breaking clients.

Code Example: Declarative JSONata Error Expressions

Writing imperative code (like if (data.messages) { ... } else if (data.error) { ... }) to normalize errors across fifty integrations will quickly bloat your codebase. The superior architectural approach is to use declarative mapping.

JSONata is a lightweight query and transformation language for JSON data. It allows you to define a mapping expression that transforms a messy upstream payload into a clean RFC 7807 object without writing integration-specific runtime code.

flowchart TD
    A["Upstream Error<br>(Bespoke JSON)"] --> B["JSONata Expression<br>(Declarative Mapping)"]
    B --> C["Normalized Error<br>(RFC 7807 Format)"]
    C --> D["Background Worker<br>(Standardized Retry Logic)"]

Imagine an upstream API returns a deeply nested validation error:

{
  "success": false,
  "error_payload": {
    "code": 422,
    "message": "Validation Failed",
    "fields": [
      {"field": "email", "issue": "format_invalid"}
    ]
  }
}

Instead of writing custom TypeScript to parse this, you can define a JSONata expression that maps it to RFC 7807:

{
  "type": "https://api.yourdomain.com/errors/validation-failed",
  "title": error_payload.message,
  "status": error_payload.code,
  "detail": "The following fields were invalid: " & $join(error_payload.fields.field, ", "),
  "instance": "urn:uuid:" & $uuid()
}

When you execute this expression against the upstream payload, the output is standard and predictable. But we can take this further. Here is a production-grade JSONata expression that handles the HTTP-200-with-error edge case and maps provider codes to canonical URIs:

(
  /* Detect error shape - HTTP 200 with nested error envelope */
  $isError := $exists(response.error) or $exists(response.errors);
 
  /* Extract raw fields */
  $rawCode := response.error.code ? response.error.code : response.errors[0].errorCode;
  $rawMsg  := response.error.message ? response.error.message : response.errors[0].message;
 
  /* Map provider codes to canonical types */
  $type := $rawCode = 'INVALID_SESSION_ID' or $rawCode = 'AUTH_001'
    ? 'https://errors.truto.one/reauth-required'
    : $rawCode = 'RATE_LIMIT_EXCEEDED'
      ? 'https://errors.truto.one/rate-limit'
      : 'https://errors.truto.one/upstream-error';
 
  /* Emit RFC 7807 envelope */
  $isError ? {
    'type': $type,
    'title': $rawMsg,
    'status': httpStatus,
    'detail': $rawMsg,
    'instance': requestPath,
    'provider_error_code': $rawCode,
    'reauth_required': $type = 'https://errors.truto.one/reauth-required'
  } : null
)

With one expression per provider stored as configuration, your runtime evaluates against the raw response and either returns null (success) or a normalized Problem Details object. A key benefit here is that re-auth detection becomes declarative. If the expression sets reauth_required: true, the platform can automatically trigger the OAuth refresh flow or notify the caller that the connection needs user intervention, without any provider-specific if-statements in the runtime.

For the mechanics of writing and testing these expressions, see our JSONata mapping tutorial.

Handling Rate Limits (HTTP 429) Without Masking Failures

One of the most common mistakes engineering teams make when building normalization layers is attempting to automatically absorb and retry rate limits on behalf of the client.

If you build a queue that holds requests when an upstream API returns a 429 Too Many Requests, you introduce massive state management complexity. If the upstream API goes down for three hours, your queue fills up, memory spikes, and your entire normalization service crashes. If your platform retries a 429 five times with exponential backoff, the caller sees a 30-second latency spike with no explanation, cannot make informed decisions about queue depth, and cannot distinguish "upstream slow" from "we are hammering an upstream that hates us." The caller's SLO becomes coupled to your internal retry policy.

A robust normalization layer does not retry, throttle, or apply backoff on rate limit errors automatically. Instead, it normalizes the rate limit information into standardized headers and explicitly passes the HTTP 429 error back to the caller. The client (your background worker) is responsible for maintaining the state and executing exponential backoff.

The better pattern:

1. Normalize the headers. Adopt the IETF draft standard for rate limit headers. Regardless of whether the upstream API uses X-RateLimit-Remaining, Rate-Limit-Left, Retry-After, or X-App-RateLimit, your normalization layer should translate them to standard headers:
   • ratelimit-limit: The total number of requests permitted in the current window.
   • ratelimit-remaining: The number of requests left in the current window.
   • ratelimit-reset: The time at which the rate limit window resets (normalized to a Unix epoch timestamp).
2. Pass the 429 through. When the upstream returns HTTP 429, the caller sees an HTTP 429 with normalized headers. No hidden retries, no swallowed errors.
3. Let the caller decide the backoff. A background sync job may want to sleep for ratelimit-reset seconds. An interactive request may want to fail fast and surface the error to the user. Only the caller has the context to make that call.

By normalizing the headers and returning the 429, you give your background workers the exact information they need to pause execution and resume safely. This transparency is critical when documenting rate limits for enterprise buyers.

Why Unified APIs Make Per-Integration Normalization Obsolete

The patterns above—declarative pagination configs, JSONata error expressions, normalized rate limit headers—all point in the same direction: integration-specific code in your runtime is a code smell. Every if provider === 'x' you write is a future bug and a future maintenance burden.

A properly designed unified API architecture pushes every provider-specific detail into declarative data:

flowchart LR
    A[Caller Request] --> B[Generic Executor]
    B --> C{Load Config}
    C -->|Pagination| D[Declarative Pagination Config]
    C -->|Error Mapping| E[JSONata Error Expression]
    C -->|Field Mapping| F[JSONata Response Transform]
    D --> G[Upstream API Call]
    E --> G
    F --> G
    G --> H[Normalized Response]
    H --> I[Caller Receives<br>Consistent Envelope]

Building a unified pagination wrapper, mapping errors to RFC 7807 with JSONata, and standardizing rate limit headers is the correct architectural approach. However, building and maintaining this infrastructure in-house requires dedicated engineering teams. Every time a SaaS provider deprecates an endpoint, alters their error schema, or changes their pagination limits, your team has to update the mapping configurations. This is why maintaining custom integrations consumes so much engineering bandwidth.

Unified APIs process hundreds of third-party integrations through a generic execution pipeline. They eliminate integration-specific code in your database and runtime logic entirely. Instead of writing custom pagination wrappers or error parsers, you execute a single standard request against the Unified API. The platform handles the declarative mapping, normalizes the pagination into a standard cursor, translates the errors into RFC 7807, and normalizes the rate limit headers.

Where to Go From Here

If you are staring at a growing integration roadmap and considering whether to keep building custom connectors, the honest answer depends on scale. For one or two integrations, custom code is fine. For ten or more, the normalization patterns above are the difference between a maintainable platform and a permanent engineering tax.

A practical path forward:

• Audit your current integrations. Count how many places you have provider-specific pagination or error-handling code. That number is your normalization backlog.
• Pick one integration and refactor. Extract pagination config into data, replace error handling with a declarative expression, normalize rate limit headers. Measure the delta in lines of code and test coverage.
• Decide build vs. buy. If you have 10+ integrations on the roadmap and no dedicated integration team, the math almost always favors a unified API platform that already implements these patterns.

The engineering pattern is the same whether you build it yourself or adopt a platform: declarative config, generic executor, no provider-specific runtime code. Get that right and the rest of your integration surface area becomes tractable. You stop writing boilerplate integration code and start focusing entirely on your core product logic.