Declarative Pagination system in Truto Unified Real-time API

This blog should serve as a guide to anyone who is looking to build a generic pagination system for their API integrations layer. It highlights how Truto has built its generic pagination system for it's real-time Unified API to handle various pagination formats without writing integration-specific code. And most importantly, it goes over the various edge cases and quirks it had to handle over the course of looking at and building over 250+ APIs and integrations.

Real-time Unified API in this context means that Truto is able to fetch data from various APIs, normalize it into a common format, and serve it to the end-user in real-time. This is done without storing any of the end-user data.

If you are caching the data from the API responses with you, either in a database or any kind of datastore, you might not need to read this blog because all of the cached data can be paginated in whatever format you think is best for your use case.

The blog will cover the following topics:

• Introduction about Truto

• What is pagination?

• Different formats of pagination

• How Truto normalizes the pagination formats

• Quirks and edge cases

This guide is intended for developers with prior experience in interacting with APIs and building API integrations.

Introduction about Truto

Truto is a Unified API platform that allows you to build native HTTP based API integrations within your application. It handles all the grunt work related to API integrations like authentication, pagination, and error handling. All of this in real-time without storing any of your end-user data.

What is pagination?

If you have ever worked with REST or GraphQL APIs, you might have come across pagination. Pagination is a technique used to break down a large set of data into smaller chunks. For example, if you have a list of 1000 contacts in a CRM, you might not want to or cannot fetch all of them at once in certain scenarios. Pagination allows you to fetch a subset of the data at a time, like 20 or 50 contacts at a time.

Different formats of pagination

Some of the most common pagination formats are:

1. Offset-based pagination: This format uses an offset and a limit to fetch the data. For example, to fetch the first 20 contacts, you would use an offset of 0 and a limit of 20. To fetch the next 20 contacts, you would use an offset of 20 and a limit of 20. The most common starting offset is 0, but some APIs might use 1 as the starting offset. Most APIs out there use query parameters to specify the offset and limit. For example:

GET /contacts?offset=0&limit=20 GET /contacts?offset=20&limit=20

Some of them might also specify it in the request body.

2. Page-based pagination: This format uses a page number and a page size to fetch the data. For example, to fetch the first page of 20 contacts, you would use page number 1 and a page size of 20. To fetch the next page of 20 contacts, you would use page number 2 and a page size of 20. The most common starting page number is 1, but some APIs might use 0 as the starting page number.

GET /contacts?page=1&pageSize=20 GET /contacts?page=2&pageSize=20

3. Cursor-based pagination: This format uses a cursor to fetch the data. The cursor is a unique identifier that points to a specific record in the dataset. To fetch the next set of data, you would use the cursor returned in the previous response.

The cursor can be opaque or transparent.

A transparent cursor is a direct reference to the record in the dataset, for example, the primary key of a record. Stripe uses transparent cursors in their APIs, where the cursor is the ID of the record and you would pass the ID of the last record fetched to get the next set of records.

An opaque cursor may or may not be a direct reference to the record in the dataset. It is usually a token that encodes the state of the query. You would pass this token to get the next set of records.

Cursor-based paginations also accept a limit to specify the number of records to fetch.

Most of the APIs use query parameters to specify the cursor and limit. For example:

GET /contacts?cursor=abc123&limit=20

The cursor can be found from the response of the previous request. It can be in the response body or in the response headers.

{ 
  "data": [ 
    { "id": 1, "name": "John Doe" } 
  ], 
    "cursor": "abc123" 
}

4. Link header pagination: This format uses the Link header to specify the next and previous URLs to fetch the data. The Link header contains the URL to the next and previous pages.

They usually have two query parameters, `page` and `pageSize` and can be thought of a special case of page-based pagination which has been standardized by RFC 5988.

Github uses this format in their APIs.

The link header looks like this:

Link: <https://api.example.com/contacts?page=2&pageSize=20>; rel="next", <https://api.example.com/contacts?page=1&pageSize=20>; rel="prev"

Here `rel="next"` specifies the URL to the next page and `rel="prev"` specifies the URL to the previous page.

How Truto normalizes the pagination formats

Simple answer: Cursor-based pagination. Truto uses cursor-based pagination to normalize the pagination formats.

Cursor-based pagination is a format which can be used to represent all the other formats of pagination mentioned above. We take the underlying pagination data and base64 encode it to create an opaque cursor. Opaque here means that the cursor is not a direct reference to the record in the dataset, but it encodes the state of the query. You can still base64 decode the cursor to get the underlying pagination data.

Truto's pagination format

Truto's pagination format contains two query parameters: `next_cursor` and `limit`. The `next_cursor` is the cursor to fetch the next set of data and the `limit` is the number of records to fetch in a single API call.

An example request to fetch the first 20 contacts from a CRM using the Unified API would look like this:

GET /api/unified/contacts?limit=20

The response from our Unified API would look like this:

{ 
  "results": [
    { "id": 1, "name": "John Doe" 
    }, 
    { "id": 2, "name": "Jane Doe" 
    }, 
    ... 
  ], 
  "next_cursor": "b2Zmc2V0PTIwJmxpbWl0PTIw" 
}

And the next request to fetch the next set of contacts would look like this:

GET /api/unified/contacts?next_cursor=b2Zmc2V0PTIwJmxpbWl0PTIw&limit=20

The next_cursor contains an opaque cursor which the client can use to fetch the next set of data.

Once the client has fetched all the data, the next_cursor will be null.

How the cursor is generated for various pagination formats

We'll now look at how the `next_cursor` is generated for various pagination formats.

Offset-based pagination

If you have an offset-based pagination which accepts an `offset` and a `page_size` query parameter, you can base64 encode the offset and limit to create a cursor.

Imagine the customer has 1000 contacts and you are fetching 20 contacts at a time. The first request to our Unified API would look like this:

GET /api/unified/contacts?limit=20

We map the `limit` query parameter of the Unified API to the `page_size` query parameter of the CRM API.

The response from our Unified API would look like this:

{
  "data": [
    {
      "id": 1,
      "name": "John Doe"
    },
    {
      "id": 2,
      "name": "Jane Doe"
    },
    ...
  ],
  "next_cursor": "b2Zmc2V0PTIwJnBhZ2Vfc2l6ZT0yMA"
}

The cursor is base64 encoded and contains the following string: `offset=20&page_size=20`.

As you can see, the cursor contains the offset and limit which can be used to fetch the next set of data.

The next request to fetch the next set of contacts would look like this:

GET /contacts?next_cursor=b2Zmc2V0PTIwJnBhZ2Vfc2l6ZT0yMA&limit=20

Since the `next_cursor` query parameter is now provided, Truto decodes the cursor to get the `offset` and `page_size` query parameters and passes them to the CRM API to fetch the next set of data.

Here `limit` query parameter of the Unified API is ignored.

The response from the CRM API would look like this:

{
  "data": [
    {
      "id": 21,
      "name": "John Doe"
    },
    {
      "id": 22,
      "name": "Jane Doe"
    },
    ...
  ],
  "next_cursor": "b2Zmc2V0PTQwJnBhZ2Vfc2l6ZT0yMA"
}

Knowing when to stop

Truto checks if the number of records returned by the CRM API is less than the limit specified by the client. If the number of records is less than the limit, it means that there are no more records to fetch and the next_cursor will be null.

Page-based pagination

If you have a page-based pagination which accepts a `page` and a `pageSize` query parameter, you can base64 encode the page and pageSize to create a cursor.

Imagine the customer has 1000 contacts and you are fetching 20 contacts at a time. The first request to our Unified API would look like this:

GET /api/unified/contacts?limit=20

We map the `limit` query parameter of the Unified API to the `pageSize` query parameter of the CRM API.

The response from our Unified API would look like this:

{
  "data": [
    {
      "id": 1,
      "name": "John Doe"
    },
    {
      "id": 2,
      "name": "Jane Doe"
    },
    ...
  ],
  "next_cursor": "cGFnZT0yJnBhZ2VTaXplPTIw"
}
 

The cursor is base64 encoded and contains the following string: `page=2&pageSize=20`. Here we assume the API is 1-indexed.

As you can see, the cursor contains the page and pageSize which can be used to fetch the next set of data.

The next request to fetch the next set of contacts would look like this:

GET /contacts?next_cursor=cGFnZT0yJnBhZ2VTaXplPTIw&limit=20

Since the `next_cursor` query parameter is now provided, Truto decodes the cursor to get the `page` and `pageSize` query parameters and passes them to the CRM API to fetch the next set of data.

Here `limit` query parameter of the Unified API is ignored.

The response from the CRM API would look like this:

{
  "data": [
    {
      "id": 21,
      "name": "John Doe"
    },
    {
      "id": 22,
      "name": "Jane Doe"
    },
    ...
  ],
  "next_cursor": "cGFnZT0zJnBhZ2VTaXplPTIw"
}

Knowing when to stop

Truto checks if the number of records returned by the CRM API is less than the pageSize specified by the client. If the number of records is less than the pageSize, it means that there are no more records to fetch and the next_cursor will be null.

Link header pagination

The link header pagination format is easy to handle because underlying APIs directly return the exact URL to fetch the next set of data.

So the approach here is to first parse the header from the response and then extract the URL with `rel="next"` and then just use the query parameters in that as the value of the `next_cursor`.

Imagine the customer has 1000 contacts and you are fetching 20 contacts at a time. The first request to our Unified API would look like this:

GET /api/unified/contacts?limit=20&foo=bar

We map the `limit` query parameter of the Unified API to the `pageSize` query parameter of the CRM API. Sometimes, even though the APIs use link header pagination, the query parameter name might be different, so Truto makes it configurable, but by default, it uses `pageSize`.

We are also passing an extra query parameter `foo=bar` to the CRM API, which is not used in the pagination.

The response from our Unified API would look like this:

{
  "data": [
    {
      "id": 1,
      "name": "John Doe"
    },
    {
      "id": 2,
      "name": "Jane Doe"
    },
    ...
  ],
  "next_cursor": "cGFnZT0yJnBhZ2VTaXplPTIwJmZvbz1iYXI"
}

The cursor is base64 encoded and contains the following string: `page=2&pageSize=20&foo=bar`.

As you can see, the cursor contains the page, pageSize, and the extra query parameter `foo`. The cursor is calculated from an link header like this:

Link: <https://api.crm.com/contacts?page=2&pageSize=20&foo=bar>; rel="next"

The next request to fetch the next set of contacts would look like this:

GET /contacts?next_cursor=cGFnZT0yJnBhZ2VTaXplPTIwJmZvbz1iYXI&limit=20

The `limit` query parameter of the Unified API is ignored.

Knowing when to stop

Truto checks if the link header contains a URL with `rel="next"`. If it does not contain a URL with `rel="next"`, it means that there are no more records to fetch and the next_cursor will be null.

Cursor-based pagination

Cursor-based pagination is easy to handle because underlying APIs directly return the cursor to fetch the next set of data.

As mentioned earlier, the cursor can be transparent or opaque, and it can be either found a field in the response body or needs to be calculated from the response body (Stripe, where the next_cursor is the ID of the last record in the response).

So Truto provides a JSON path to extract the cursor from the response.

For example, if the raw API response looks something like,

{
  "data": [
    {
      "id": 1,
      "name": "John Doe"
    },
    {
      "id": 2,
      "name": "Jane Doe"
    },
    ...
  ],
  "meta": {
    "pagination": {
      "next": "foobar"
    }
  }
}

Then the JSON path to extract the cursor would be `meta.pagination.next`.

In case of Stripe, the cursor is the ID of the last record in the response, so the JSON path would be `data[-1].id`. `-1` is used to get the last element of the array.

We just use the value found at the JSON path as the value of the `next_cursor`.

Knowing when to stop

Truto checks if the cursor (value found at the JSON path) is null. If the cursor is null, it means that there are no more records to fetch and the next_cursor will be null.

Some APIs also return a `has_more` or equivalent field which is typically a boolean, in the response body to indicate if there are more records to fetch. Truto also checks this field to know when to stop.

When APIs return the next link in the response body

In some cases, the APIs directly return the link to the next set of data in the response body regardless of the pagination format. So Truto considers them as cursor-based pagination and extracts the URL query parameters from the next link in the response body and encodes them into the `next_cursor`.

For example, if the raw API response looks something like,

{
  "data": [
    {
      "id": 1,
      "name": "John Doe"
    },
    {
      "id": 2,
      "name": "Jane Doe"
    },
    ...
  ],
  "next": "https://api.crm.com/contacts?page=2&pageSize=20"
}

Truto extracts the query parameters from the `next` field and encodes them into the `next_cursor`.

Declarative pagination configuration

Everything above is powered by a small piece of JSON attached to each integration. The runtime is one generic executor that reads the config, mutates outgoing requests, and extracts state from responses. There is no per-integration pagination code, which is what makes onboarding a new REST or GraphQL API a config change instead of a code change. This is the core answer to how a unified API reconciles REST pagination differences across providers: a shared execution pipeline reads a declarative description of each provider's pagination and translates it to and from the unified next_cursor contract.

The config attaches at two levels: the integration (as a default) and any individual resource method (as an override). Setting pagination: null on a method opts that method out entirely, even when the integration has a default strategy. That escape hatch matters for endpoints that return an unpaginated singleton payload inside an otherwise paginated API.

The declarative schema

Truto supports six pagination types. Each has a small set of fields that describe how to construct the next request and where to find the next-page signal in the response.

A few cross-cutting fields apply across all types:

• limit_param and a per-integration max_limit bound the caller's requested page size to what the upstream accepts.
• stop_on_short_page tells the runtime to treat a page smaller than the requested limit as end-of-stream. Useful for providers that never emit a null cursor.
• has_more_field names a boolean in the response body (like Stripe's has_more) that authoritatively signals end-of-stream even when a cursor is still present.
• pagination_path and include_path handle providers whose next-page URL uses a different endpoint from the first request.

Full JSON example

Here's a complete pagination block for an integration that mixes strategies across resources. The integration default is cursor-based, but two resources override it:

{
  "pagination": {
    "type": "cursor",
    "cursor_field": "paging.next.after",
    "cursor_query_param": "after",
    "limit_param": "limit",
    "max_limit": 100
  },
  "resources": {
    "contacts": {
      "pagination": {
        "type": "cursor",
        "cursor_field": "paging.next.after",
        "cursor_query_param": "after",
        "limit_param": "limit",
        "max_limit": 100
      }
    },
    "companies": {
      "pagination": {
        "type": "offset",
        "offset_param": "offset",
        "limit_param": "limit",
        "start_offset": 0,
        "max_limit": 200,
        "stop_on_short_page": true
      }
    },
    "activities": {
      "pagination": {
        "type": "page",
        "page_param": "page",
        "page_size_param": "per_page",
        "start_page": 1,
        "page_increment": 1,
        "max_limit": 50
      }
    },
    "owners": {
      "pagination": null
    }
  }
}

For a GraphQL Relay-style API, the dynamic type uses JSONata expressions to describe the cursor logic directly:

{
  "pagination": {
    "type": "dynamic",
    "next_cursor_expr": "response.data.**.pageInfo.endCursor",
    "has_next_expr": "response.data.**.pageInfo.hasNextPage",
    "apply_expr": "{ 'variables': { 'first': limit, 'after': pagination.cursor } }"
  }
}

That single block handles any Relay-style connection: first and after on the request, pageInfo.endCursor plus pageInfo.hasNextPage on the response. Different query names? The ** wildcard in the JSONata path matches any intermediate keys, so the same config works across contacts, deals, issues, and so on.

Mapping examples: unified cursor to provider request

The runtime always accepts ?next_cursor=…&limit=… from the caller. What it emits to the upstream depends on the declarative config. Here's the translation for each type.

Cursor (HubSpot-style)

Caller:

GET /unified/crm/contacts?limit=50&next_cursor=YWZ0ZXI9YWZ0ZXItdG9rZW4tMTIz

Config:

{
  "type": "cursor",
  "cursor_field": "paging.next.after",
  "cursor_query_param": "after",
  "limit_param": "limit"
}

Outgoing provider request:

GET https://api.hubapi.com/crm/v3/objects/contacts?after=after-token-123&limit=50

Offset

Caller cursor decodes to offset=100&limit=50. Config maps those to the provider's parameter names:

{
  "type": "offset",
  "offset_param": "skip",
  "limit_param": "top",
  "start_offset": 0
}

Outgoing:

GET https://provider.example.com/records?skip=100&top=50

Page

Cursor decodes to page=3&per_page=25:

{
  "type": "page",
  "page_param": "page",
  "page_size_param": "per_page",
  "start_page": 1
}

Outgoing:

GET https://provider.example.com/tickets?page=3&per_page=25

Link header (GitHub-style)

Cursor decodes to the query string of the previous rel="next" URL, for example page=2&per_page=100. The runtime replays that URL as-is, so the outgoing request is whatever the provider gave you last time:

GET https://api.github.com/repos/acme/things/issues?page=2&per_page=100

GraphQL Relay (dynamic)

Cursor is the provider-issued endCursor. The apply_expr places it into GraphQL variables:

{
  "query": "query($first:Int!,$after:String){ contacts(first:$first,after:$after){ edges{ node{ id name } } pageInfo{ endCursor hasNextPage } } }",
  "variables": { "first": 50, "after": "Y3Vyc29yOjEwMA==" }
}

Range

Cursor decodes to a domain-specific range like records=1000-1999. The runtime attaches it as an HTTP header:

GET https://provider.example.com/logs
Range: records=1000-1999

Provider response to unified cursor extraction

The reverse direction is symmetric. Each config type declares where the next-page signal lives; the runtime pulls it out and encodes an opaque next_cursor for the caller.

Cursor field in body

Provider response:

{
  "results": [{ "id": "1" }, { "id": "2" }],
  "paging": { "next": { "after": "after-token-456" } }
}

With cursor_field: "paging.next.after", extraction reads after-token-456 and base64-encodes it into next_cursor. If the path doesn't resolve, next_cursor is null.

Last-record ID (Stripe-style)

Provider response:

{
  "data": [{ "id": "cus_abc" }, { "id": "cus_xyz" }],
  "has_more": true
}

Config:

{
  "type": "cursor",
  "cursor_field": "data[-1].id",
  "cursor_query_param": "starting_after",
  "has_more_field": "has_more"
}

data [-1].id picks the last record. If has_more is false, extraction returns null even if cursor_field resolved to a value.

Offset and page

No response path is required. The runtime computes next_offset = current_offset + returned_count (or next_page = current_page + page_increment) itself. When the returned count is smaller than the requested limit, extraction returns null.

Link header

Provider response headers:

Link: <https://api.github.com/repos/acme/things/issues?page=3&per_page=100>; rel="next",
      <https://api.github.com/repos/acme/things/issues?page=1&per_page=100>; rel="prev"

Extraction parses the header, finds rel="next", and stores its query string in the opaque cursor. Missing rel="next" means next_cursor: null.

GraphQL Relay

Response:

{
  "data": {
    "contacts": {
      "edges": [{ "node": { "id": "1" } }],
      "pageInfo": { "endCursor": "Y3Vyc29yOjIwMA==", "hasNextPage": true }
    }
  }
}

next_cursor_expr returns Y3Vyc29yOjIwMA==, has_next_expr returns true. When hasNextPage is false, next_cursor is null regardless of what endCursor contains.

Validations and common misconfigurations

The pagination config is validated at integration-load time, but a handful of failure modes only show up under real traffic. Watch for these:

• Wrong cursor_field path. If the JSON path doesn't resolve, extraction returns undefined, which the runtime treats as end-of-stream. Symptom: the first page returns fine, but you never get a second. Fix: log the raw response and confirm the path against a JSONata evaluator before shipping.
• Off-by-one on start_page or start_offset. APIs disagree on whether numbering starts at 0 or 1. Setting start_page: 0 on a 1-indexed API double-fetches page one (or errors). Setting start_page: 1 on a 0-indexed API silently skips the first page.
• Missing has_more_field on APIs that recycle tokens. Some providers return the same cursor forever instead of null. Without has_more_field or stop_on_short_page: true, the loop never terminates. Truto also fingerprints response bodies with SHA-1 as a last-resort guard, but the correct fix is to configure the terminator field.
• limit above max_limit. If the caller sends ?limit=500 against an upstream that caps at 100, the runtime clamps to max_limit. Forget to set max_limit and the upstream will 400, or worse, silently truncate the page.
• Cursor in body vs. query. Setting cursor_in_body: true but omitting cursor_query_param is fine. Forgetting the body flag on an API that only accepts the cursor in the POST body, however, results in the cursor being appended to the URL and ignored by the server. Same failure mode as a missing cursor field: infinite first-page loop.
• Path-sensitive next URLs. Salesforce's SOQL API returns a nextRecordsUrl whose path differs from the initial query endpoint. Extracting only the query string and reusing the original path sends the next request to the wrong URL. Set include_path: true so the runtime preserves the URL from the response. Dropbox has a similar quirk with a dedicated continuation endpoint - configure pagination_path and ignore_limit_in_pagination for it.
• Pagination disabled on the wrong endpoint. pagination: null at the method level overrides the integration default. If a list endpoint accidentally inherits this from a copied config, you'll get a single page of data with no next_cursor, ever.
• Stale apply_expr on dynamic pagination. GraphQL Relay integrations sometimes rename their pagination variables (first becomes pageSize). The apply_expr has to match the current schema. The failure mode is a GraphQL 200 response with an errors array. Pair dynamic pagination with an error_expression on the method config so those surface as HTTP errors instead of empty result sets.

A safe rollout pattern: seed the config, run a two-page dry run against a sandbox account, log the outgoing provider request and the extracted next_cursor, and diff against expectations. Every failure mode above is visible in that trace.

Quirks and edge cases

We will now list a few quirks and edge cases Truto handles for the various pagination formats found in the wild —

• The API path for fetching the subsequent pages might be different from the initial request. There are two famous products which have slight variations of this

• Dropbox: Dropbox provides one URL for the initial request and a different URL for fetching the subsequent pages, and and and, you can't use any other query parameters in the subsequent requests. Truto solves this by making the pagination URL configurable (pagination_path) and adding a flag to ignore the query parameters in the subsequent requests (ignore_limit_in_pagination).

• Salesforce: In thier SOQL Query API, Salesforce returns the URL to the next set of records, but here it's not enough to just encode the query parameters, the path of the URL also needs to be considered. Truto solves this by adding a flag to include the path of the URL as well (include_path).

• Starting offset and page number might be 0 or 1. Truto handles this by making the starting offset and page configurable.

• Enforcing maximum limit. Truto enforces a maximum limit on the number of records that can be fetched in a single request. This is configurable based on the maximum limit of the underlying API.

• Some APIs might always require the limit or other query parameters to be present in the request. Truto handles this by always including the default values of the query parameters in the request.

• Not all APIs accept the pagination data in the query parameters. Some APIs accept the pagination data in the request body. Truto handles this by decoupling the pagination parsing and encoding from the request building.

• Some cursor based APIs never return a null cursor. So Truto checks if the number of records fetched in the response is less than the limit and stops fetching the records. It also employs a SHA-1 has of the request body to check if the same records are being fetched again.

Using next_cursor for ETL and bulk data extraction

The declarative pagination system pays off the moment you try to build an ETL or bulk extraction pipeline against multiple SaaS APIs. Because every integration collapses into the same next_cursor + limit contract, one worker loop can extract data from HubSpot, Salesforce, Zendesk, Notion, or any other integrated account without knowing which pagination format sits underneath. This section shows how to wire that up end to end - the extraction loop, checkpointing, parallelization, and retry behavior.

Short architecture diagram and data flow

Here's the flow of a single extraction run. Your worker owns state (checkpoints and logging). Truto stays stateless per request - the next_cursor it returns is all the state the worker needs to resume.

flowchart LR
    subgraph Client ["Your ETL worker"]
      A["Load checkpoint<br>(next_cursor, high_water_mark)"]
      B["Call Unified API<br>with next_cursor + limit"]
      C["Write batch to sink<br>(warehouse / lake / DB)"]
      D["Commit new checkpoint"]
    end
    subgraph Truto ["Truto Unified API"]
      E["Decode next_cursor"]
      F["Call third-party API<br>(auth, rate-limit, retry)"]
      G["Normalize response<br>Issue next next_cursor"]
    end
    A --> B
    B --> E
    E --> F
    F --> G
    G --> C
    C --> D
    D --> A

The important property here is that the client loop is the same regardless of upstream API. Adding a new integration to your pipeline is a config change - a new (model, resource, integrated_account_id) tuple - not a code change.

Copy-paste extraction worker (Node)

A minimal but production-shaped worker in Node. It streams pages, invokes an onBatch callback for each page (that's where you write to your sink), and returns when Truto stops issuing a next_cursor.

// Node 20+, uses global fetch
const BASE = 'https://api.truto.one'
const TOKEN = process.env.TRUTO_TOKEN
 
async function extractAll({
  model,           // e.g. 'crm'
  resource,        // e.g. 'contacts'
  integratedAccountId,
  query = {},      // e.g. { updated_at: JSON.stringify({ gt: '2026-05-01' }) }
  limit = 100,
  onBatch,         // async (records, { nextCursor }) => void
  checkpoint,      // { next_cursor?: string } | undefined
}) {
  let nextCursor = checkpoint?.next_cursor ?? null
  let pages = 0
 
  while (true) {
    const url = new URL(`${BASE}/unified/${model}/${resource}`)
    url.searchParams.set('integrated_account_id', integratedAccountId)
    url.searchParams.set('limit', String(limit))
    if (nextCursor) url.searchParams.set('next_cursor', nextCursor)
    for (const [k, v] of Object.entries(query)) url.searchParams.set(k, v)
 
    const res = await fetchWithRetry(url, {
      headers: { Authorization: `Bearer ${TOKEN}` },
    })
    const body = await res.json()
 
    await onBatch(body.result || [], { nextCursor: body.next_cursor })
    pages++
 
    if (!body.next_cursor) break
    nextCursor = body.next_cursor
  }
 
  return { pages }
}

Copy-paste extraction worker (Python)

Same contract in Python using requests. The on_batch callback is where you write to your sink and commit the checkpoint.

import os
import requests
 
BASE = "https://api.truto.one"
TOKEN = os.environ["TRUTO_TOKEN"]
 
def extract_all(
    model,                 # e.g. "crm"
    resource,              # e.g. "contacts"
    integrated_account_id,
    on_batch,              # def on_batch(records, meta): ...
    query=None,
    limit=100,
    checkpoint=None,
):
    session = requests.Session()
    session.headers["Authorization"] = f"Bearer {TOKEN}"
    next_cursor = (checkpoint or {}).get("next_cursor")
    pages = 0
 
    while True:
        params = {
            "integrated_account_id": integrated_account_id,
            "limit": limit,
            **(query or {}),
        }
        if next_cursor:
            params["next_cursor"] = next_cursor
 
        resp = fetch_with_retry(
            session, f"{BASE}/unified/{model}/{resource}", params
        )
        body = resp.json()
 
        on_batch(body.get("result", []), {"next_cursor": body.get("next_cursor")})
        pages += 1
 
        if not body.get("next_cursor"):
            break
        next_cursor = body["next_cursor"]
 
    return {"pages": pages}

Both workers work against /unified/* for normalized data or /proxy/* for raw integration payloads. The pagination contract is identical.

Checkpoint schema and resume logic

Keep the checkpoint minimal. Because Truto encodes offset, page, cursor, or link-header state into next_cursor, the worker doesn't have to track any of that itself. A checkpoint row looks like this:

{
  "run_id": "run_2026_06_01_hubspot_contacts",
  "integrated_account_id": "acc_abc",
  "model": "crm",
  "resource": "contacts",
  "next_cursor": "b2Zmc2V0PTIwJnBhZ2Vfc2l6ZT0yMA",
  "high_water_mark": "2026-06-01T00:03:12Z",
  "records_written": 4200,
  "started_at": "2026-06-01T00:00:00Z",
  "last_seen_at": "2026-06-01T00:03:12Z",
  "status": "running"
}

A reasonable SQL shape:

CREATE TABLE etl_checkpoint (
  run_id text PRIMARY KEY,
  integrated_account_id text NOT NULL,
  model text NOT NULL,
  resource text NOT NULL,
  next_cursor text,
  high_water_mark timestamptz,
  records_written bigint NOT NULL DEFAULT 0,
  status text NOT NULL,          -- running | done | failed
  updated_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (integrated_account_id, model, resource, run_id)
);

Resume logic follows four rules:

1. On worker start, load the row by (integrated_account_id, model, resource, run_id). If status = 'done', exit. Otherwise, take next_cursor from the row.
2. On each successful batch write to the sink, update next_cursor, records_written, and last_seen_at. Commit the sink write and the checkpoint update in the same transaction when possible, or write the checkpoint immediately after the sink write when the sink is append-only.
3. When Truto returns next_cursor: null, set status = 'done' and record high_water_mark (typically now() or the max updated_at seen in the batch).
4. On crash-restart, the loop naturally picks up from the last committed next_cursor.

Here's the Node onBatch for that flow:

async function onBatch(records, { nextCursor }) {
  await db.transaction(async trx => {
    await sink.upsertMany(trx, records)   // idempotent by primary key
    await trx('etl_checkpoint')
      .where({ run_id })
      .update({
        next_cursor: nextCursor ?? null,
        records_written: db.raw('records_written + ?', [records.length]),
        last_seen_at: new Date(),
        status: nextCursor ? 'running' : 'done',
      })
  })
}

For incremental syncs, drive the initial query from high_water_mark:

const query = checkpoint.high_water_mark
  ? { updated_at: JSON.stringify({ gt: checkpoint.high_water_mark }) }
  : {}

This pattern is idempotent under retries because sinks upsert by primary key and Truto returns the same next_cursor for the same logical page.

Pagination patterns mapped to next_cursor

The client loop above works for every integration because Truto has already collapsed six different pagination strategies into one contract. Here's how each upstream pattern surfaces in your worker:

The worker never sees the difference. If you switch a job from HubSpot to Salesforce, or add Notion to a knowledge-base ETL that already ingests Confluence, the loop code doesn't change - only the (model, resource, integrated_account_id) tuple does.

Parallelization and concurrency configuration

A single cursor chain is inherently sequential - page N+1 can't start until page N returns. There are two productive ways to parallelize an ETL pipeline built on top of the Unified API.

1. Shard by resource. Run one worker per (integrated_account_id, model, resource). Contacts, deals, companies, and tickets all extract concurrently, each with its own checkpoint row.

import pLimit from 'p-limit'
 
const limit = pLimit(8) // 8 concurrent resource extractions
const jobs = [
  { model: 'crm', resource: 'contacts' },
  { model: 'crm', resource: 'companies' },
  { model: 'crm', resource: 'deals' },
  { model: 'ticketing', resource: 'tickets' },
]
 
await Promise.all(
  jobs.map(job =>
    limit(() => extractAll({
      ...job,
      integratedAccountId,
      onBatch,
      checkpoint: loadCheckpoint(job),
    }))
  )
)

2. Shard by time window. For large historical backfills, split the range by updated_at and run windows concurrently. Each window keeps its own checkpoint keyed by (resource, window).

from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta, timezone
 
def month_windows(start, end):
    cur = start
    while cur < end:
        nxt = min(cur + timedelta(days=30), end)
        yield cur.isoformat(), nxt.isoformat()
        cur = nxt
 
start = datetime(2024, 1, 1, tzinfo=timezone.utc)
end = datetime(2026, 1, 1, tzinfo=timezone.utc)
 
with ThreadPoolExecutor(max_workers=6) as ex:
    for lo, hi in month_windows(start, end):
        query = {"updated_at": '{"gte":"%s","lt":"%s"}' % (lo, hi)}
        ex.submit(
            extract_all, "crm", "contacts", integrated_account_id,
            on_batch, query=query,
        )

A few practical notes on tuning concurrency:

• Set the concurrency ceiling based on the upstream integration's rate limit, not Truto's. Truto forwards the provider's rate-limit budget to you via standardized headers, so the provider is the real bottleneck.
• Per-account concurrency > 1 is usually fine for read-heavy backfills, but some providers rate-limit per user token rather than per app. Start at 2-4 and increase if ratelimit-remaining stays healthy.
• Cross-account concurrency scales linearly because each account has its own credentials and quota.

Retry/backoff and rate-limit handling

Truto standardizes rate-limit signals across integrations. Whether the upstream API uses X-RateLimit-Reset, Retry-After, or a custom header, your client always sees:

• HTTP 429 when the upstream is rate-limited
• Retry-After in seconds
• Optional ratelimit-limit, ratelimit-remaining, ratelimit-reset headers on both success and 429 responses

That means one retry helper works for every integration. Here's a Node version that handles rate limits, transient 5xx, and network errors with exponential backoff plus full jitter:

const sleep = ms => new Promise(r => setTimeout(r, ms))
const jitter = attempt =>
  Math.floor(Math.random() * Math.min(30_000, 500 * 2 ** attempt))
 
async function fetchWithRetry(url, opts, { maxAttempts = 6 } = {}) {
  let attempt = 0
  while (true) {
    attempt++
    let res
    try {
      res = await fetch(url, opts)
    } catch (err) {
      if (attempt >= maxAttempts) throw err
      await sleep(jitter(attempt))
      continue
    }
 
    if (res.status === 429) {
      const retryAfter = Number(res.headers.get('retry-after') || 1)
      await sleep(retryAfter * 1000)
      continue
    }
 
    if (res.status >= 500 && attempt < maxAttempts) {
      await sleep(jitter(attempt))
      continue
    }
 
    if (!res.ok) {
      const text = await res.text()
      throw new Error(`HTTP ${res.status}: ${text}`)
    }
    return res
  }
}

And the Python equivalent:

import random
import time
import requests
 
def fetch_with_retry(session, url, params, max_attempts=6):
    for attempt in range(1, max_attempts + 1):
        try:
            resp = session.get(url, params=params, timeout=60)
        except requests.RequestException:
            if attempt == max_attempts:
                raise
            time.sleep(random.uniform(0, min(30, 0.5 * 2 ** attempt)))
            continue
 
        if resp.status_code == 429:
            time.sleep(int(resp.headers.get("retry-after", "1")))
            continue
 
        if resp.status_code >= 500 and attempt < max_attempts:
            time.sleep(random.uniform(0, min(30, 0.5 * 2 ** attempt)))
            continue
 
        resp.raise_for_status()
        return resp

Two things worth knowing for ETL workloads:

• Resume is idempotent by construction. The checkpoint is only committed after a successful sink write. A retry replays the same next_cursor and produces the same page, so upserts on your sink make replays safe.
• 401s mean pause, not retry. Truto tags remote authentication failures with truto_is_remote_error: true. When you see repeated 401s from a specific account, it needs re-authentication - retrying won't help. Subscribe to the integrated_account:authentication_error webhook, pause runs for that account, and resume once you receive integrated_account:reactivated. That's a circuit breaker for the auth failure mode.

With the pagination normalization doing the heavy lifting, a couple hundred lines of client code is enough to build a multi-integration, multi-resource ETL pipeline with resume, checkpointing, concurrency, and rate-limit handling that behaves the same way whether the upstream is Salesforce, HubSpot, Notion, or Jamf.