How to Handle Third-Party API Rate Limits When AI Agents Scrape Data

If you're building AI agents that scrape, sync, or orchestrate data across external SaaS platforms, you've already hit the wall. Your LangChain or LangGraph setup works perfectly in local testing. But the moment you deploy it to production and point it at a customer's Salesforce, Jira, or Zendesk instance, the agent crashes with a 429 Too Many Requests error. Or worse — a 403, or a 200 OK with an error buried in the response body.

The reason is straightforward: autonomous agent loops consume API quotas at a rate that traditional applications never approached. And the retry logic you wrote for one provider doesn't work for the next one because every SaaS API signals rate limits differently.

The problem is architectural. You cannot rely on the native LLM rate limiters built into your agent framework. You need a proxy-side standardization layer that normalizes the chaotic rate limit headers of 50+ different SaaS providers into a single, predictable format, combined with client-side exponential backoff.

This post breaks down why agentic API traffic breaks traditional SaaS limits, the real cost of ignoring 429 errors, why standardizing rate limits across providers is so painful, and the exact architectural patterns that actually work - including adaptive concurrency algorithms, request prioritization, caching trade-offs, and production SLOs.

Why AI Agents Break Third-Party APIs (The Rate Limit Problem)

AI agents exhaust SaaS API quotas orders of magnitude faster than traditional applications because they execute autonomous, multi-step reasoning loops that chain dozens of API calls per task.

The core conflict is simple:

• Traditional apps trigger API calls based on slow, predictable human inputs — clicks, page loads — or scheduled batch ETL jobs. A human user clicking through a CRM might trigger 2-3 API calls per minute.
• AI agents execute autonomous, multi-step reasoning loops (like LangGraph) that recursively call tools until a condition is met, generating massive, unpredictable spikes in API traffic. An autonomous agent might chain 10-20 sequential API calls to complete a single task — tool lookups, retrieval-augmented generation queries, multi-step reasoning, and final completions — all in a rapid burst.

Consider a standard customer support triage agent. Its prompt instructs it to read a new inbound ticket, fetch the user's entire purchase history from Shopify, pull their active contracts from Salesforce, and check for open bugs in Jira. A human support rep might take ten minutes to gather this context. The agent attempts to do it in three seconds. If the data is paginated, the agent might recursively call the "next page" endpoint twenty times in a row. And that's just one agent run. Scale that to 50 customers, each with their own connected Salesforce org, and you're looking at thousands of calls per minute against a shared daily quota.

Traditional fixed windows and static thresholds aren't built for this kind of traffic. AI agent traffic patterns — high volume, bursty, automated — look remarkably similar to DDoS attacks or bot scraping. SaaS providers designed their rate limits for human-driven workflows, not for autonomous loops that can spin up, fan out, and retry without any natural pause.

This shift is not a niche concern. Gartner predicts that more than 30% of the increase in demand for APIs will come from AI and LLM tools by 2026. Industry analysts project the agentic AI market will surge from $7.8 billion today to over $52 billion by 2030, while Gartner predicts that 40% of enterprise applications will embed AI agents by the end of 2026. Legacy APIs enforce limits like "100 requests per minute." An agent can burn through that quota in the first five seconds of a complex reasoning task. The immediate result is an HTTP 429 Too Many Requests response — and if your agent isn't explicitly engineered to catch this specific status, parse the provider's retry headers, and pause its execution thread, the entire orchestration loop fails.

The Hidden Costs of Unhandled 429 Too Many Requests Errors

Failing to handle rate limit errors properly doesn't just drop a single request — it creates cascading failures that can crash your background workers, lock out customer SaaS accounts, and burn through cloud budgets.

Most engineering teams treat rate limiting as a deferred maintenance issue. In the context of agentic architectures, ignoring it carries a severe, immediate cost.

Here's what actually happens when an agent hits a 429 and your code doesn't respect it:

1. The infinite retry spiral. Because the agent's goal-oriented loop dictates that it must acquire the data to proceed, a naive implementation will fire the same tool call again. And again. An agent operating without proper restrictions can quickly turn a minor logical error or a malicious prompt into a catastrophic event. Consider a simple bug in a recursive loop, which could lead to an immediate and massive spike in API usage, resulting in a significant cost explosion that drains budgets in minutes. Hammering an API that has already returned a 429 only extends the penalty window.

2. Worker pool exhaustion. Each spinning retry holds a thread, a database connection, and memory. Within minutes, your background job queue is full of retrying tasks that will never succeed. New sync jobs can't start. Your core product's data freshness degrades across all customers — not just the one hitting the rate limit. We've documented this exact failure mode in our guide to Best Practices for Handling API Rate Limits and Retries Across Multiple Third-Party APIs.

3. Customer SaaS account lockout. SaaS providers actively monitor for abusive API patterns. Salesforce enforces a 100,000 daily API request limit for Enterprise Edition orgs, plus 1,000 additional requests per user license. The system also caps concurrent long-running requests at 25. An agent that burns through that daily budget by 10 AM means your customer's own sales team can't use their CRM for the rest of the day. That's not a bug report — that's a churned customer. If your agent repeatedly hammers a customer's HubSpot instance, HubSpot will revoke the OAuth token or temporarily ban the IP address. You've now broken your customer's internal workflows because your agent lacked basic rate limit awareness.

4. Financial damage. Cloud compute isn't free. Retry loops that spin for hours consume CPU, memory, and network I/O. Every failed request burns compute cycles, and every retry holds an execution thread open in your background worker pool.

Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls, according to Gartner. Rate limit mishandling is exactly the kind of "escalating cost" and "inadequate risk control" that kills agent projects before they ever reach production. If you're architecting AI agents that connect to SaaS systems, rate limit handling isn't a nice-to-have — it's table stakes.

Why Standardizing Rate Limits Across 50+ SaaS APIs is a Nightmare

There is no universally adopted standard for how SaaS APIs communicate rate limits. Every provider uses different HTTP status codes, different headers, and different semantics — making it impossible to write a single generic retry handler.

If you only need your agent to talk to one external API, writing custom rate limit handling is tedious but manageable. If you're building a B2B platform where your agents need to interact with dozens of different CRMs, ticketing systems, and HRIS platforms, custom handling becomes an architectural nightmare.

The IETF has been working on a draft standard (draft-ietf-httpapi-ratelimit-headers) since 2019, proposing a clean, predictable set of headers: RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset. Rate limiting of HTTP clients has become a widespread practice, especially for HTTP APIs. Typically, servers who do so limit the number of acceptable requests in a given time window. Currently, there is no standard way for servers to communicate quotas so that clients can throttle their requests to prevent errors. After 10 revisions, it remains a draft as of 2025. Almost no major SaaS provider actually uses it.

Here's what you actually encounter in the wild:

| Provider | Status Code | Rate Limit Headers | Retry Signal | Gotchas | |----------|------------|--------------------|--------------|---------|| | Salesforce | 403 (REQUEST_LIMIT_EXCEEDED) | Sforce-Limit-Info: api-usage=X/Y | No Retry-After; rolling 24-hour window | Uses 403, not 429. Per-org daily limit, not per-minute. | | Jira Cloud | 429 | X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, RateLimit-Reason | Retry-After (seconds) | Multiple limit types with different RateLimit-Reason values. | | Zendesk | 429 | x-rate-limit, ratelimit-limit, ratelimit-remaining, ratelimit-reset | Retry-After (seconds) | Additional endpoint-specific headers like zendesk-ratelimit-tickets-index. | | HubSpot | 429 | X-HubSpot-RateLimit-Daily-Remaining | Retry-After (seconds) | Separate daily and per-second limits, different header families. | | Shopify | 429 | GraphQL query cost in custom JSON extension | Leaky bucket algorithm | Limits tracked via request cost, not raw call count. |

Look at the Salesforce row. When you call the Salesforce REST API, responses include a Sforce-Limit-Info header that tells you consumption vs. limit. It doesn't return a 429. It doesn't return a Retry-After header. It uses a completely custom header format with a rolling 24-hour window. Your generic if status == 429: sleep(retry_after) handler will never trigger.

Jira is its own adventure. Different RateLimit-Reason headers indicate which limit was exceeded: jira-quota-global-based or jira-quota-tenant-based, jira-burst-based, or jira-per-issue-on-write. You need to parse the reason to know whether to back off for milliseconds or hours.

Worse, many legacy enterprise APIs don't even return standard HTTP status codes. It's incredibly common to query an older SOAP or REST API, exceed your quota, and receive a 200 OK response. The only indication that you've been rate-limited is a custom string buried inside the response payload, such as {"success": false, "error": "Quota exceeded"}.

A major interoperability issue in throttling is the lack of standard headers, because each implementation associates different semantics to the same header field names. If your agent talks to 10 SaaS providers, you need 10 different parsing and retry strategies. That's 10 sets of header-parsing logic, 10 different interpretations of "when can I retry," and 10 potential failure modes to test and maintain. If you attempt to handle this at the agent level, your tool definitions become bloated with integration-specific parsing logic, violating separation of concerns and making maintenance practically impossible.

Client-Side vs. Proxy-Side Rate Limiting for AI Agents

When your LangChain or LlamaIndex agent hits a rate limit on a third-party SaaS API, you have two architectural options. Both have real trade-offs.

Option 1: Client-Side Retry Logic (Inside the Agent)

When developers first encounter 429 errors in their agentic workflows, their instinct is to look for a solution within their framework. If you're using LangChain, you'll find documentation for the InMemoryRateLimiter and the .with_retry() method.

These are excellent utilities, but they solve the wrong problem.

A common issue when running large evaluation jobs is running into third-party API rate limits, usually from model providers. There are a few ways to deal with rate limits. If you're using LangChain Python chat models, you can add rate limiters to your model(s) that will add client-side control of the frequency with which requests are sent to the model provider API.

Note the scope: model provider API. LangChain's built-in rate limiter is designed for OpenAI, Anthropic, and similar LLM endpoints. This is an in memory rate limiter, so it cannot rate limit across different processes. The rate limiter only allows time-based rate limiting and does not take into account any information about the input or the output. It has no awareness of the rate limit headers coming back from Salesforce, Jira, or Zendesk. If your agent calls a search_zendesk_tickets tool, the framework executes the Python function you provided. If Zendesk returns a 429, the framework blindly passes that error back to the LLM, which usually hallucinates a fix or crashes.

The client-side approach means writing something like this for every single provider:

import time
 
def call_with_retry(api_func, max_retries=5):
    for attempt in range(max_retries):
        response = api_func()
        if response.status_code == 200:
            return response.json()
        elif response.status_code == 429:
            retry_after = int(response.headers.get("Retry-After", 2 ** attempt))
            time.sleep(retry_after)
        elif response.status_code == 403:  # Salesforce-style
            # Parse Sforce-Limit-Info, calculate backoff...
            time.sleep(3600)  # Good luck guessing
        else:
            raise Exception(f"Unexpected: {response.status_code}")
    raise Exception("Max retries exceeded")

This works for one provider. Now multiply it by 50. Each one has different headers, different status codes, and different semantics. Your agent tool layer becomes a graveyard of provider-specific if/elif branches. As we explored in Architecting AI Agents: LangGraph, LangChain, and the SaaS Integration Bottleneck, building resilience into the tool itself is mandatory — but the logic doesn't have to live there.

Option 2: Proxy-Side Normalization (Between the Agent and the API)

Instead of teaching every agent tool how every SaaS API signals rate limits, you put a normalization layer between the agent and the providers. This proxy intercepts the provider's response, detects rate limiting using provider-specific logic, and returns a standardized response to the agent.

The agent now only needs one retry strategy. It checks for 429, reads Retry-After, and backs off. It doesn't care whether the upstream provider returned a 403 with a custom header or a 200 with rate limit info buried in the response body.

How Truto Standardizes API Rate Limits for AI Agents

Truto takes the proxy-side approach and bakes rate limit normalization directly into its Unified API and Proxy API layers. The design philosophy: zero integration-specific code. Every provider's quirks are handled through declarative configuration, not custom handler functions.

Whether you're using Truto's Unified API (which normalizes data models across categories like CRM or ATS) or the Proxy API (which provides direct RESTful CRUD access to specific platforms), the execution pipeline handles rate limits uniformly.

Here's how it works at an architectural level:

1. Detection: The is_rate_limited Mapping

Because legacy APIs might return a 429, a 403, or even a 200 OK with an error payload, each integration has a configuration that includes a declarative JSONata expression to evaluate whether a given response constitutes a rate limit. For Salesforce, that expression checks for a 403 status with the Sforce-Limit-Info header pattern. For an API that returns a 200 with rate limit info in a custom header, the expression catches that too.

If no detection expression is configured for a given integration, Truto falls back to the standard: HTTP 429 means rate-limited, anything else doesn't.

Once detected, Truto immediately normalizes the response status. Regardless of what the provider sent, your AI agent will always receive a standard 429 Too Many Requests status code.

2. Normalization: Standardizing the Retry-After Header

Knowing that you hit a limit is only half the battle — your agent needs to know exactly how long to pause its execution loop. Some APIs return a Retry-After header in seconds. Others return an X-RateLimit-Reset header formatted as a Unix timestamp. Some return a complex HTTP-date string. Some return nothing at all.

A second JSONata expression extracts the "when to retry" information from whatever format the provider uses and converts it to a simple Retry-After value in seconds. The agent never needs to know the source format.

3. Standard Quota Headers

A third expression maps the provider's quota information into standardized headers, giving your agentic workflows maximum visibility into their remaining capacity:

This means your agent can inspect ratelimit-remaining on successful 200 OK responses and proactively slow down its loop before it ever hits a 429 error.

sequenceDiagram
    participant Agent as AI Agent (LangGraph)
    participant Truto as Truto Proxy Layer
    participant SaaS as 3rd-Party SaaS API

    Agent->>Truto: GET /crm/contacts (Tool Call)
    Truto->>SaaS: Forward Request
    SaaS-->>Truto: 200 OK <br> {"error": "Quota Exceeded"}
    
    Note over Truto: JSONata evaluates response<br>Detects rate limit condition<br>Extracts reset timestamp
    
    Truto-->>Agent: 429 Too Many Requests <br> Retry-After: 45 <br> ratelimit-remaining: 0
    
    Note over Agent: Agent reads standard header<br>Pauses execution for 45s<br>Resumes loop safely

Because both the Unified API and the Proxy API route through the exact same generic execution pipeline, this standardization applies automatically to all 100+ integrations on the platform. To understand why Truto can do this without writing per-integration code, see how the zero-code architecture works.

Building Resilient Agentic Data-Fetching Workflows

Once you have a proxy layer standardizing the rate limit responses, building resilient agents becomes straightforward. You no longer need provider-specific error handling. You only need one clean retry wrapper that respects the Retry-After header.

Exponential Backoff With Header-Aware Delays

The best approach combines the standardized Retry-After header (when available) with exponential backoff as a fallback. Using a retry library like tenacity, you can wrap your Truto-backed tool calls with strict retry caps — a maximum delay of 30-60 seconds and a hard stop after 5-7 attempts to avoid cascading failures:

import requests
import time
import random
from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type
 
class RateLimitException(Exception):
    def __init__(self, retry_after):
        self.retry_after = retry_after
 
def handle_rate_limit(retry_state):
    exception = retry_state.outcome.exception()
    wait_time = int(exception.retry_after)
    remaining = getattr(exception, 'remaining', 'unknown')
    print(f"Rate limited. Remaining: {remaining}. "
          f"Sleeping for {wait_time}s (attempt {retry_state.attempt_number})")
    time.sleep(wait_time)
 
@retry(
    retry=retry_if_exception_type(RateLimitException),
    wait=wait_exponential(multiplier=1, min=2, max=60),
    stop=stop_after_attempt(5),
    before_sleep=handle_rate_limit
)
def execute_truto_tool(endpoint, headers):
    response = requests.get(endpoint, headers=headers)
    
    # Truto guarantees this status code for ALL rate limits
    if response.status_code == 429:
        # Truto guarantees this header is always in seconds
        retry_seconds = response.headers.get('Retry-After', 10)
        raise RateLimitException(retry_after=retry_seconds)
        
    response.raise_for_status()
    return response.json()

This single function safely executes tool calls against Salesforce, NetSuite, BambooHR, and Jira. The agent framework remains completely ignorant of the underlying API quirks. No provider-specific branches. No header-parsing gymnastics.

Pre-Flight Quota Checks

If the standardized headers include ratelimit-remaining on successful responses (not just 429s), your agent can make smarter decisions before it runs out of quota:

def should_continue_scraping(response):
    """Check remaining quota from successful response headers."""
    remaining = response.headers.get("ratelimit-remaining")
    limit = response.headers.get("ratelimit-limit")
    
    if remaining and limit:
        utilization = 1 - (int(remaining) / int(limit))
        if utilization > 0.85:  # 85% consumed
            return "throttle"
        if utilization > 0.95:  # 95% consumed
            return "pause"
    return "continue"

Architectural Guardrails for Agent Loops

Beyond per-request retry logic, these structural patterns are essential in production:

• Hard iteration caps. Never let a LangGraph loop run unbounded. Set a max_iterations parameter on your agent executor. If the agent hasn't completed its task in 20 iterations, stop it and escalate.
• Per-account concurrency limits. If multiple agents are hitting the same customer's Salesforce org, use a shared semaphore or distributed lock to prevent them from collectively blowing through the daily quota.
• Circuit breakers. After 3 consecutive 429s on the same integration, trip a circuit breaker that pauses all requests to that provider for a configurable cooldown period. This prevents the thundering herd problem where dozens of retrying tasks slam the provider simultaneously.
• Observability. Log every rate limit event with the provider name, customer account, ratelimit-remaining value, and Retry-After delay. This data is gold for capacity planning and for having informed conversations with customers about their API tier.

Adaptive Concurrency: Dynamic Pacing With AIMD

Static concurrency limits are a blunt instrument. If you hardcode "5 concurrent requests per provider," you're either leaving throughput on the table when the provider has headroom, or still hitting 429s when quota is tight.

The better approach borrows from TCP congestion control: Additive Increase / Multiplicative Decrease (AIMD). The concept is proven at internet scale - it's how TCP has managed network congestion for decades. Applied to API rate limiting, it works the same way: when requests succeed and ratelimit-remaining shows healthy capacity, gradually increase parallelism. When you hit a 429, cut concurrency sharply.

This creates a sawtooth pattern that naturally converges on the maximum sustainable request rate for each provider without exceeding their limits.

How AIMD Maps to API Pacing

• Additive increase: After a streak of successful responses where quota utilization stays below 70%, add one concurrent worker. This probes for available capacity without overshooting.
• Multiplicative decrease: On any 429 (or normalized rate limit signal), immediately halve the worker count. Fast reaction to congestion prevents cascading failures and avoids extending penalty windows.
• Floor and ceiling: Never drop below 1 worker (the agent must still make progress) or exceed a configured maximum.

Here's a practical AIMD controller that reads standardized rate limit headers and dynamically adjusts your worker pool:

class AIMDConcurrencyController:
    """
    Adjusts parallel request count based on rate limit feedback.
    Additive increase on success with headroom.
    Multiplicative decrease on 429.
    """
    def __init__(self, min_workers=1, max_workers=20):
        self.workers = min_workers
        self.min_workers = min_workers
        self.max_workers = max_workers
        self.consecutive_ok = 0
        self.ramp_threshold = 10  # successes before adding a worker
 
    def on_success(self, headers):
        remaining = headers.get("ratelimit-remaining")
        limit = headers.get("ratelimit-limit")
 
        if remaining and limit:
            utilization = 1 - (int(remaining) / int(limit))
            if utilization < 0.7:
                self.consecutive_ok += 1
                if self.consecutive_ok >= self.ramp_threshold:
                    # Additive increase: cautiously add one worker
                    self.workers = min(self.max_workers, self.workers + 1)
                    self.consecutive_ok = 0
            else:
                # Utilization above 70% - hold steady
                self.consecutive_ok = 0
 
    def on_rate_limited(self):
        # Multiplicative decrease: cut concurrency in half
        self.workers = max(self.min_workers, self.workers // 2)
        self.consecutive_ok = 0
 
    def get_worker_count(self):
        return self.workers

This works because the proxy layer has already normalized the headers. Without standardized ratelimit-remaining and ratelimit-limit values, you'd need a separate AIMD controller per provider - each parsing different header formats.

Integrating AIMD With Your Agent's Task Dispatcher

The controller slots into your agent's task execution layer. Use it to size an asyncio semaphore that gates concurrent API calls:

import asyncio
 
controller = AIMDConcurrencyController(min_workers=2, max_workers=15)
 
async def dispatch_agent_tasks(tasks, endpoint, headers):
    results = []
 
    async def run_with_pacing(task):
        semaphore = asyncio.Semaphore(controller.get_worker_count())
        async with semaphore:
            response = await execute_truto_tool_async(endpoint, headers, task)
            if response.status_code == 429:
                controller.on_rate_limited()
                retry_after = int(response.headers.get("Retry-After", 5))
                await asyncio.sleep(retry_after)
                return await run_with_pacing(task)  # Retry with reduced concurrency
            controller.on_success(dict(response.headers))
            return response.json()
 
    results = await asyncio.gather(*[run_with_pacing(t) for t in tasks])
    return results

The AIMD controller manages how many requests fly concurrently. The exponential backoff logic from the previous section manages what happens to each individual failed request. They're complementary - use both.

This is how you tune scraping frequency to match API quotas automatically. Instead of guessing a static request-per-second cap for each provider, the controller discovers the right throughput empirically and adjusts in real time as conditions change.

Batching, Prioritization, and Backpressure Strategies

Adaptive concurrency controls how fast you send requests. Batching and prioritization control which requests you send and in what order. When quota is limited, these decisions determine whether your agent completes its high-value tasks or wastes calls on low-priority work.

Maximize Data Per Request

Most SaaS APIs charge rate limits per HTTP request, not per record returned. A single request fetching 200 contacts costs the same quota as one fetching 10. Always request the maximum page size the provider supports.

Agents frequently get this wrong. A naively-implemented pagination loop might use the API's default page size (often 10-25 records) when the maximum is 200. On a 100-requests-per-minute limit, that's the difference between processing 2,000 records per minute and 20,000.

Priority Queues: Writes Before Reads

Not all API calls carry equal weight. Write operations (creating a deal in Salesforce, updating a ticket in Jira) are typically more time-sensitive and harder to retry safely than reads. Structure your request queue with explicit priorities:

import heapq
from dataclasses import dataclass, field
from typing import Any
 
PRIORITY_WRITE = 0       # Highest - writes are hardest to retry
PRIORITY_READ_LIVE = 1   # User-facing reads
PRIORITY_READ_BULK = 2   # Background sync reads
PRIORITY_BACKFILL = 3    # Historical data backfill
 
@dataclass(order=True)
class PrioritizedRequest:
    priority: int
    item: Any = field(compare=False)
 
class AgentRequestQueue:
    def __init__(self, max_depth=500):
        self._queue = []
        self._max_depth = max_depth
 
    def enqueue(self, request, priority):
        if len(self._queue) >= self._max_depth:
            if priority > PRIORITY_READ_LIVE:
                raise BackpressureError("Queue full - shedding bulk work")
        heapq.heappush(self._queue, PrioritizedRequest(priority, request))
 
    def dequeue(self):
        return heapq.heappop(self._queue).item if self._queue else None

When rate limits tighten, your most important operations get through first. Bulk data backfills and non-urgent reads get shed under pressure.

Backpressure: Know When to Stop Accepting Work

Backpressure is the signal that flows upstream from a saturated consumer to a fast producer, telling it to slow down. In agent architectures, the third-party API is the slow consumer and your agent loop is the fast producer. Without explicit backpressure, the agent keeps scheduling API calls while the retry queue grows unboundedly.

Practical backpressure rules:

• Queue depth limits. Set a hard cap on pending requests per provider. When the queue hits 80% capacity, stop scheduling new agent tasks targeting that provider.
• Propagate delays upstream. If your AIMD controller has dropped to minimum concurrency and Retry-After headers indicate a 60-second pause, surface that delay to the agent orchestrator. The agent can switch to tasks targeting a different provider or perform local computation instead of blocking.
• Shed low-priority work first. Under pressure, drop backfill and bulk sync tasks. Keep writes and live user-facing reads flowing.

Decision Tree: Handling Write-Limited Endpoints

Write endpoints are almost always subject to stricter rate limits than reads. Some providers enforce entirely separate write quotas. The retry calculus is also different - retrying a failed write without idempotency can create duplicate records.

flowchart TD
    A[Agent needs to<br>write to SaaS API] --> B{Is the write<br>idempotent?}
    B -->|Yes| C[Safe to retry<br>with backoff]
    B -->|No| D{Can you add an<br>idempotency key?}
    D -->|Yes| C
    D -->|No| E[Single attempt +<br>queue for manual<br>review on failure]
    C --> F{ratelimit-remaining<br>> 10% of limit?}
    F -->|Yes| G[Execute immediately]
    F -->|No| H{Is the write<br>time-sensitive?}
    H -->|Yes| I[Execute with retry<br>and alert on failure]
    H -->|No| J[Defer to next<br>rate limit window]

The key insight: non-idempotent writes can't be blindly retried without risking duplicates. Either make them idempotent by attaching a unique request key, or limit to a single attempt and escalate failures to a review queue.

Pass-Through vs. Cached Architectures for AI Agents

When an AI agent queries third-party SaaS data, the request either hits the upstream API directly (pass-through) or reads from a locally cached copy. This architectural choice directly determines how - and how often - you encounter rate limits.

When Pass-Through Wins

Pass-through is the right choice when:

• Your agent writes data back to customer SaaS systems. Cached architectures can serve reads from local storage, but writes always hit the live API. A mixed architecture (cache for reads, pass-through for writes) adds complexity and risks read-your-own-write inconsistencies.
• Data freshness matters. If your agent triages a support ticket, it needs the customer's current subscription status - not a copy from 15 minutes ago. A stale cache hit that causes the agent to make the wrong decision silently is worse than a loud 429 error.
• Enterprise compliance requirements exist. Many enterprise customers won't approve vendors that cache their CRM or HRIS data in third-party infrastructure. Pass-through eliminates the data-at-rest surface entirely. See Zero Data Retention for AI Agents for a deeper analysis.

The trade-off is direct rate limit exposure. Every agent request hits the provider's quota. This is exactly where adaptive concurrency (AIMD) and standardized rate limit headers become essential - your agent needs real-time visibility into remaining quota to pace itself.

When Caching Wins

Caching makes sense when:

• Read-to-write ratio is extremely high. If your agent reads the same org chart or product catalog hundreds of times per day but never writes to it, syncing that data on a schedule and serving reads from cache dramatically reduces quota consumption.
• Latency matters more than freshness. A cache hit avoids the network round-trip to the third-party API entirely. For agents that need sub-second tool call responses to maintain conversational flow, caching stable reference data is worthwhile.
• You need to survive upstream outages. A cache provides a fallback when the third-party API is down or rate-limiting you aggressively. Pass-through architectures propagate upstream failures directly to your agent.

The Hybrid Approach

Most production systems land on a hybrid: cache stable reference data (org structures, product catalogs, user rosters) with appropriate TTLs, and pass-through for transactional reads and all writes. The key is setting TTLs that match the data's actual change frequency - a company's org chart doesn't change every minute, but a ticket's status might.

Truto's architecture supports both patterns. The Proxy API provides pass-through access with standardized rate limit headers, while webhooks and event-driven syncs can feed a local cache for high-read reference data.

Operational Checklist: Alerts, Circuit Breakers, and SLOs

The architectural guardrails described earlier - hard iteration caps, per-account concurrency, circuit breakers, observability - are the code-level foundation. But production agent systems also need explicit SLOs, alerting thresholds, and escalation procedures to catch problems before they cascade.

Define SLOs for Agent-to-API Interactions

Your SLOs should capture what actually matters to your users: does the agent complete its task? Start with these three:

• Task completion rate: ≥ 99% of agent tasks complete without a rate-limit-induced failure. This is your primary SLO.
• Rate limit error ratio: < 2% of all outbound API calls return a 429 (after normalization). Sustained rates above this indicate your pacing is too aggressive.
• Retry overhead: < 10% of total API calls should be retries. Higher percentages mean your concurrency settings need tuning.

Track these per provider and per customer account. A single customer's Salesforce org hitting its daily limit shouldn't distort your global metrics - but it should trigger an alert.

Alerting Thresholds

Set alerts at these boundaries:

The error budget burn rate is especially useful for catching slow-motion failures. If your SLO allows 1% task failures over a 30-day window and you're burning through that budget in 3 days, something has changed - maybe a customer connected more users, or a provider silently lowered their rate limits.

Circuit Breaker Configuration

The circuit breaker concept was introduced in the guardrails section above. Here's how to configure it for production, scoped per provider and per customer account:

• Trip threshold: 3 consecutive 429 responses, or a 429 rate exceeding 20% over a 1-minute window.
• Open duration: Start with the Retry-After value from the last 429, or default to 60 seconds.
• Half-open probe: After the open duration, allow a single request through. If it succeeds, close the circuit. If it returns another 429, reset the open timer with a doubled duration (capped at 5 minutes).

The per-customer-account scoping matters. A circuit breaker that trips globally would pause all customers' syncs because one customer's Salesforce org hit its daily limit.

Minimum Dashboard Metrics

At minimum, your ops dashboard should track:

• Per-provider 429 rate over the last hour, with trend lines
• Current ratelimit-remaining for each active customer-provider connection
• Active circuit breakers - which integrations are paused and when they'll retry
• AIMD controller state - current concurrency level per provider, trend over time
• Task completion funnel - tasks started vs. completed vs. failed-by-rate-limit vs. failed-other

This data serves double duty: it powers real-time alerting, and it gives you evidence for conversations with customers about upgrading their SaaS API tier when their quota consistently runs dry.

What This Means for Your Architecture

The gap between a working AI agent demo and a production-grade agent system is almost entirely infrastructure. The LLM reasoning works. The tool-calling interface works. What breaks is the connection to the messy reality of third-party SaaS APIs — their inconsistent rate limits, their undocumented edge cases, their surprise 403s when you expected 429s.

You have three paths forward:

1. Build it yourself. Write custom rate limit handling for every provider you integrate with. This is viable if you support 3-5 integrations and have dedicated engineering bandwidth. It becomes untenable at 20+.

2. Use a proxy layer that normalizes the chaos. This is what Truto does — every provider's rate limit signals get flattened into a single, predictable pattern your agent can handle with one retry function. The trade-off is adding a dependency on an intermediary service.

3. Avoid real-time API calls entirely. Use ETL pipelines to pre-sync data into your own datastore. This eliminates rate limits at query time, but introduces staleness and doesn't work for write operations or real-time agent workflows.

For most teams building AI agents that need live SaaS data, option 2 delivers the best balance of reliability and development speed. You cannot solve this by writing more if/else statements in your LangGraph nodes. The solution is architectural. By placing a normalization proxy between your agents and the SaaS platforms they interact with, you transform unpredictable, provider-specific rate limit errors into a single, standardized contract.

Your agents get the data they need, your background workers stay healthy, and your engineering team stops wasting sprints debugging bespoke API quotas.