OAuth at Scale: The Architecture of Reliable Token Refreshes

If you have ever built a Salesforce integration in a weekend, you know the feeling of triumph when that first 200 OK comes back. You store the access token, maybe toss the refresh token into a database column, and call it a day.

Then, three months later, your error logs light up.

Tokens are expiring mid-sync. Refresh requests are hitting race conditions because two background jobs tried to refresh the same token simultaneously. A customer changed their password, revoking all tokens, and your app is still hammering the API until you get rate-limited.

At Truto, we maintain connections to over 100 different SaaS platforms—from HRIS systems like Workday to CRMs like HubSpot. We process millions of API requests, and every single one relies on a valid, fresh credential.

We learned the hard way that managing OAuth token lifecycles is not a storage problem; it is a distributed systems problem.

Here is the engineering deep dive into how we architected a token refresh system that handles concurrency, proactive renewal, and graceful failure at scale.

The "Happy Path" is a Lie

The OAuth 2.0 spec provides a framework, but every provider implements it with their own chaotic flair.

• Expiry Times: Some tokens last 1 hour, some 24 hours, some never expire until used.
• Refresh Behavior: Some providers rotate the refresh token every time you use it (refreshing the refresh token). If you fail to capture the new one due to a network blip, you are locked out forever.
• Concurrency: If two processes try to refresh the same token at the exact same second, many providers will invalidate both requests, assuming a replay attack.

To handle this, we moved away from simple "check and refresh" logic to a multi-layered architecture involving proactive alarms, mutex locks, and self-healing state machines.

Layer 1: The Two-Pronged Refresh Strategy

Waiting for a token to expire before refreshing it is a recipe for latency and failed user requests. Conversely, refreshing it too aggressively wastes API quota. We use a hybrid approach: Proactive Alarms and Just-in-Time (JIT) Checks.

1. Proactive Refresh (The Background Worker)

Whenever a token is created or updated, we schedule work in our auth layer to run 60 to 180 seconds before the token expires - per account, not on a coarse global cron.

Why the randomization? To prevent thundering herds. If 10,000 accounts were connected at 9:00 AM, we don't want 10,000 refresh requests firing exactly at 9:59 AM. Spreading the load ensures stability.

When that scheduled refresh runs, it negotiates a new token, updates durable storage, and re-encrypts the new credentials.

2. Just-in-Time Safety Net

We cannot rely solely on background jobs. Clocks drift, schedulers miss edge cases, and sometimes a token expires faster than the provider claimed.

Before every single API request—whether it's a proxy call or a sync job—our infrastructure checks the token's validity. We use a 30-second buffer logic:

// Simplified logic: treat token as expired if it expires in the next 30s
if (token.expiresAt < (now + 30_seconds)) {
  await refreshCredentials();
}

This buffer is critical. Without it, a token might be valid when the check runs but expire 100ms later while the request is in flight to Salesforce.

Layer 2: Solving Concurrency with Mutex Locks

This is where most in-house integrations fail.

Imagine a scenario:

1. A scheduled sync job starts for Customer A.
2. Simultaneously, Customer A triggers a manual "Test Connection" in your UI.
3. Both processes see the token is about to expire.
4. Both processes fire a request to POST /oauth/token.

The Result: The provider receives two refresh requests. It processes the first, invalidates the old refresh token, and issues a new one. Then it processes the second request (using the now-invalid old refresh token), throws an error, and potentially revokes the entire grant for security reasons. Your customer is now disconnected.

The Solution: Per-account serialized refresh (mutex)

To prevent this, we wrap the refresh operation in a mutex (mutual exclusion) lock scoped to each integrated account.

Conceptually, each account has a single serializer for refresh work. When a refresh is requested:

1. The request attempts to acquire a lock for that specific integrated_account_id.
2. If no operation is in progress, it proceeds, arms a short watchdog timeout (e.g., 30s) so a stuck provider cannot block the account forever, and executes the refresh.
3. If a refresh is already running, the second request simply awaits the same in-flight operation and reuses its result.

This ensures that no matter how many concurrent callers need a fresh token, we only send one HTTP request to the provider. Everyone else gets the outcome of that single successful call.

Crucially, the lock is keyed by integrated_account_id, not global. Two refreshes for the same account are serialized. Two refreshes for different accounts run in parallel. This preserves throughput while eliminating the race condition.

Layer 3: Handling "Invalid Grant" and Re-Auth

Sometimes, refresh fails. The user might have uninstalled the app, changed their password, or an admin might have revoked the token.

When we receive a fatal error (like HTTP 400 invalid_grant or HTTP 401 Unauthorized), retrying is futile. We need to involve the human.

The needs_reauth State Machine

1. Detection: We catch invalid_grant errors specifically. Transient errors (HTTP 500s) trigger a retry with exponential backoff. Auth errors trigger a state change.
2. State Update: The account status is flipped from active to needs_reauth.
3. Notification: We fire a webhook event: integrated_account:authentication_error.

This allows our customers to listen for this event and immediately show a "Reconnect" banner in their UI.

Auto-Reactivation

We also support auto-reactivation. If an account is in needs_reauth but a subsequent API call succeeds (perhaps the user fixed the issue on the provider side, or it was a false positive from the API), we automatically flip the status back to active and fire a integrated_account:reactivated event.

Security at Rest

Storing thousands of access and refresh tokens requires paranoia. As part of our strict security standards, we never store tokens in plain text.

• Encryption: All sensitive fields (access_token, refresh_token, client_secret) are encrypted using AES-GCM before hitting the database.
• Masking: When developers list accounts via our API, these fields are masked. They are only decrypted inside the secure enclave of the refresh service just before being used.

How Truto's Unified Calendar API Works

Everything above - proactive refreshes, mutex locks, reauth detection - is the foundation that keeps Truto's Unified Calendar API running reliably. Here is the shape of it in one paragraph: you point your client at https://api.truto.one/unified/calendar/*, pass an integrated_account_id on every request, and Truto routes the call to the right provider (Google Calendar, Outlook, Calendly, Cal.com, and others), refreshes credentials if needed, applies the provider's rate-limit contract, maps the response into a normalized schema, and returns it in the same request cycle. No calendar data is cached or stored on our side.

That pass-through design means every single call depends on a valid OAuth token being available instantly. There is no fallback to a local cache if the token has expired. Calendar providers are a particularly demanding stress test for token management - they combine short-lived tokens (typically ~1 hour for both Google and Microsoft) with high-frequency access patterns like availability polling, event sync, and webhook-driven updates across potentially thousands of connected accounts.

The Unified Calendar API normalizes six core entities across providers:

• Calendars - the container for time-based entries ("Work", "Personal", "Team Holidays").
• Events - individual appointments, meetings, or blocked time slots. Full CRUD.
• Availability - calculated free/busy time windows for finding open slots without double-booking.
• EventTypes - pre-configured booking templates (highly relevant for Calendly and HubSpot Meetings) that define duration, location, and routing before an Event is formally booked.
• Contacts - attendees, organizers, or external guests associated with an Event.
• Attachments - files, agenda documents, or meeting materials linked to an Event.

You write your integration logic once against this schema; Truto handles the per-provider translation using JSONata expressions at runtime. If you switch a customer from Google Calendar to Outlook, no client code changes.

OAuth Flow and Required Scopes per Provider

Each calendar provider has its own OAuth configuration, scopes, redirect behavior, and token lifetime. Truto handles the full lifecycle - authorization redirect, token exchange, encrypted storage, and proactive refresh - so your application never touches provider credentials directly.

The end-to-end flow looks like this:

sequenceDiagram
    participant User
    participant YourApp as Your App
    participant Truto
    participant Provider as "Calendar Provider (Google/Microsoft/etc.)"

    User->>YourApp: Click "Connect Calendar"
    YourApp->>Truto: Create link token
    Truto-->>YourApp: link_token
    YourApp->>Truto: Redirect user to /oauth/{provider}
    Truto->>Truto: Build authorize URL with scopes, PKCE, state
    Truto->>Provider: 302 to provider's authorize endpoint
    User->>Provider: Grant consent
    Provider->>Truto: Redirect to /oauth/{provider}/callback with code
    Truto->>Provider: POST /token (code, client_id, client_secret, PKCE verifier)
    Provider-->>Truto: access_token + refresh_token + expires_in
    Truto->>Truto: Encrypt tokens (AES-GCM), schedule refresh alarm
    Truto-->>YourApp: Webhook: integrated_account:active
    YourApp-->>User: "Calendar connected"

Truto stores the state parameter with a 5-minute TTL, validates it on callback, and enforces PKCE (S256) where the provider supports it. If any step fails, the account never reaches active and no partial tokens are persisted.

Here are the scopes we recommend for each supported provider. You can trim these to the minimum your product actually needs when you bring your own OAuth app.

A few provider-specific notes:

• Google Calendar. Truto's default OAuth app is CASA Tier 2 certified - your customers can connect without hitting Google's unverified app warnings or user-cap limits. If you use read-only calendar data, drop calendar.events and calendar.calendars to reduce the consent prompt.
• Microsoft Outlook. Include offline_access explicitly, otherwise Microsoft will not issue a refresh token and every access token will expire in an hour with no way to renew. Truto handles both common, tenant-specific, and consumers authorization endpoints depending on the app registration.
• Calendly. The token endpoint returns a refresh token on every exchange but does not always rotate it. Truto's token merge logic preserves the previous refresh token if the provider omits it in the response, avoiding accidental lockout.
• Cal.com. No OAuth cycle at all. The API key is stored encrypted and injected into every request via the Authorization header. Skip Layers 1-3 mentally for this provider; the security and reauth logic still apply if the key is rotated or revoked.

Bring Your Own OAuth App

Truto provides a default OAuth app for each calendar provider so you can start building immediately. For production, you will likely want to use your own.

You supply your OAuth client credentials (client ID and secret) at the environment integration level - a pure configuration change, no code deployment. From that point, Truto uses your OAuth app for all authorization flows with that provider. The connection flow through Truto's Link SDK shows your app name and branding to the end user.

Why this matters for calendar integrations specifically:

• Your consent screen, your brand. Users see your app name when granting calendar access, not a third party's.
• Credential portability. If you ever switch providers, the OAuth tokens belong to your app. You migrate without forcing every customer to re-authenticate.
• Scope control. You decide exactly which calendar scopes to request. If your product only reads events, drop write scopes entirely.

The token refresh, encryption, and concurrency protections described in the layers above work identically regardless of whether you use Truto's OAuth app or your own.

Proactive Refresh and Mutex Logic in Calendar Practice

Layer 1 and Layer 2 are especially load-bearing for calendars because of how customers actually use them. A typical calendar-heavy workload looks like this:

• A background sync polls events every 5 minutes across every connected account.
• The end user opens a scheduling UI that fires 3-4 availability queries in quick succession.
• A provider webhook fires on an event change, which triggers a fetch to enrich the payload.

All three can happen inside the same second for the same account. Without a mutex, at least two of them will race on a token that is 40 seconds from expiry. With the per-account mutex:

1. The first caller to notice the token is within the 30-second expiry buffer acquires the lock and issues the refresh.
2. Callers 2 and 3 detect the in-flight refresh, await its promise, and reuse the resulting token.
3. The provider sees exactly one POST /token request, avoiding replay-attack heuristics that would revoke the grant.

Proactive refresh (the 60-180 second pre-expiry alarm) is what keeps you off the JIT path in the first place. On a healthy account, Layer 1 handles the refresh in the background before any request ever hits the 30-second buffer. Layer 2 is the fallback for the cases where an alarm was skipped, a token expired earlier than the provider's expires_in claim, or the account was just connected and the first sync ran before the initial alarm.

One detail worth calling out: new tokens are merged with existing tokens, not replaced. If Google returns a fresh access_token but omits refresh_token in the refresh response (which is common - many providers only issue the refresh token in the initial code exchange), the previous refresh_token is preserved. Blindly replacing the token object would silently lock every customer out on the second refresh.

Reauth UX and Error Handling for Calendar Apps

When a refresh finally does fail (revoked grant, changed password, deleted mailbox), the account status flips to needs_reauth and the integrated_account:authentication_error webhook fires. What you do with that event is the difference between a good calendar product and a broken one.

A solid reauth UX for calendars looks like this:

1. Subscribe to integrated_account:authentication_error. The webhook payload includes the integrated_account_id, the customer identifier, and a last_error message extracted from the provider's response (via error expressions).
2. Surface the state in your UI immediately. Do not wait for the user to notice broken sync. Show a banner or badge on any calendar view: "Your Google Calendar needs to be reconnected."
3. Provide a one-click reconnect. Re-open the Truto Link flow scoped to the same integrated_account_id. The user does not need to pick the provider again or re-enter identifiers.
4. Pause background work. Suspend polling for that account until you receive integrated_account:reactivated. Continuing to poll a needs_reauth account just burns rate-limit quota on 401 responses.
5. Handle auto-reactivation gracefully. If the user fixes the underlying issue on the provider side (for example, re-granting access from their Google security dashboard), the next successful API call will flip the account back to active and fire integrated_account:reactivated. Your UI should clear the reconnect banner on that event.

On the client side, treat every unified API response with an eye on two error shapes:

• HTTP 401 from Truto - the account has been marked needs_reauth. Do not retry. Route the user to reconnect.
• HTTP 429 from Truto - a normalized rate limit. Back off using Retry-After (see below).

Everything else (5xx, network errors) is safe to retry with exponential backoff.

Rate Limit Handling and Normalized Headers

Calendar providers signal rate limits in provider-specific ways. Google returns HTTP 429 with a Retry-After header. Microsoft Graph uses 429 with Retry-After plus custom RateLimit-* headers. Other providers get creative.

Truto normalizes all of this. Regardless of what the provider returns, your application sees the same headers:

If the provider signals a rate limit - even via a non-429 status code or a custom header - Truto always returns HTTP 429 to your application. One rate-limit handler covers Google, Microsoft, Calendly, and every other integration.

These headers are also returned on successful (2xx) responses, so you can monitor your rate limit headroom proactively without waiting to get throttled.

Retry and Backoff Strategy

When you receive a 429 from Truto, here is a practical client-side pattern:

async function callWithBackoff(
  request: () => Promise<Response>,
  maxRetries = 5
): Promise<Response> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await request();
    if (response.status !== 429) return response;
 
    const retryAfter = parseInt(
      response.headers.get('Retry-After') || '0', 10
    );
    const backoff = retryAfter > 0
      ? retryAfter * 1000
      : Math.min(1000 * Math.pow(2, attempt), 30_000);
 
    // Add jitter to avoid synchronized retry storms
    const jitter = Math.random() * 1000;
    await new Promise(r => setTimeout(r, backoff + jitter));
  }
  throw new Error('Max retries exceeded');
}

Key points:

• Always prefer Retry-After when present. The value comes from the provider, normalized to seconds by Truto, and reflects actual quota recovery timing.
• Fall back to exponential backoff when Retry-After is absent: 1s, 2s, 4s, 8s, 16s, capped at 30s.
• Add jitter if you have many workers hitting the same calendar provider. Without it, retries synchronize and create a second thundering herd.

Provider Capabilities: Proxy vs Unified

Not every calendar provider exposes the same operations. This matrix shows what is available through Truto's Unified Calendar API (normalized schema at /unified/calendar/*) vs the Proxy API (raw provider pass-through at /proxy/*):

"Unified + Proxy" means the operation is available through both the normalized Unified Calendar API and the raw Proxy API. The Proxy API gives you access to every provider-specific field and parameter. The Unified API normalizes responses into Truto's standard calendar schema so you can swap providers without changing client code. Operations marked "N/A" mean the provider does not support that concept at the API level.

Monitoring, Metrics, and Operational Expectations

Treat OAuth infrastructure the same way you treat a payment gateway: instrument it, alert on it, and know exactly what "healthy" looks like. Whether you build it yourself or use Truto, these are the signals worth watching.

Webhook signals from Truto. These are the fastest way to see auth health across your customer base:

• integrated_account:authentication_error - track the rate over time, per provider. A sudden spike usually means a provider rotated something, changed their token endpoint, or updated their consent policy.
• integrated_account:reactivated - the recovery counter. A healthy system shows reactivations following errors within minutes to hours, not days.
• integrated_account:active - fresh connections, useful for growth dashboards.

Response-level signals. On every unified API response, check:

• HTTP status distribution (2xx vs 401 vs 429 vs 5xx) segmented by provider.
• ratelimit-remaining headroom. If you consistently see this dropping below 10% of ratelimit-limit, you are one traffic spike away from throttling.
• Latency percentiles (p50/p95/p99) per provider. Google and Microsoft normally sit under 500ms p95 for calendar reads; sustained increases usually correlate with provider-side incidents.

Metrics we recommend tracking (client-side):

Alerting thresholds worth setting:

• Auth error rate exceeds 2x the trailing 7-day baseline for a single provider - probable provider-side change.
• Any single account sends more than 5 authentication_error events in an hour - stop retrying and page an operator.
• 429 rate above 1% of total requests for any provider - back off and check your polling strategy.

What to expect operationally. Truto's refresh architecture is designed so that a healthy account never sees an expired token. In practice:

• Proactive refresh completes 60-180 seconds before expiry, so under normal conditions no client request ever waits on a refresh.
• Mutex-serialized refreshes mean a burst of 100 concurrent requests against a nearly-expired token results in exactly one refresh call to the provider, not 100.
• Failed refresh attempts against transient errors (HTTP 5xx) are retried automatically on a 3-hour cadence, capped so a permanent revocation doesn't loop forever.
• Non-retryable errors (HTTP 401, 403, invalid_grant) skip retry entirely, flip the account to needs_reauth, and emit the webhook within seconds.

We publish live availability on our status page and communicate incidents there. For contractual SLAs on your specific deployment, talk to us directly.

Best Practices for High-Frequency Calendar Access

Calendar workloads punish naive polling. A scheduling assistant checking availability every minute across 5,000 accounts is 300,000 requests per hour before the user has even done anything. Here is how to keep that workload well-behaved.

Prefer webhooks over polling. For Google, Outlook, and Calendly, Truto normalizes provider webhooks into unified events (record:created, record:updated, record:deleted) delivered to your endpoint. Use these to invalidate a local mirror instead of re-polling every N minutes. Webhooks are dramatically cheaper on rate limit budget and lower-latency for the user.

Batch availability queries. If a user is looking at a week view, ask for the whole week in one call, not seven daily calls. Google's freebusy.query and Microsoft Graph's getSchedule both accept multiple calendars and long time windows.

Cache free/busy results with tight TTLs. Availability rarely changes within a 30-60 second window for the same user. A short in-memory cache in front of Truto cuts request volume without materially affecting freshness. Invalidate the cache on any Event webhook for that calendar.

Use updated_after filters for incremental sync. The Unified Calendar API supports incremental fetches. Poll ?updated_after=<last_sync> instead of paginating the full event list every cycle. Combine with sync tokens where the provider supports them (Google's syncToken is exposed via the Proxy API).

Spread background work. If you run a nightly sync across all accounts, jitter the start times over a window (e.g., 30 minutes) instead of firing them all at midnight. This mirrors the same reasoning behind our randomized 60-180 second refresh window: avoid synchronized load on both your infrastructure and the provider's.

Respect the account-level mutex. Because refreshes are serialized per account, extremely bursty traffic against a newly-expired token will queue behind a single in-flight refresh. This is by design and correct behavior. If you see latency spikes in the p99 during token expiry moments, that is the refresh flight completing and unblocking everyone. It should be sub-second in normal conditions.

Separate hot and cold reads. If your product mixes user-facing scheduling (needs to be fast) with background analytics (can tolerate lag), route them differently. Real-time scheduling should always hit the Unified API. Analytics can query a synced local copy or use the truto_super_query mechanism to hit pre-synced data without touching the provider at all.

Summary: The Checklist for Reliable Auth

If you are building this in-house, ensure your architecture covers these bases:

1. Buffer your expiry checks: Don't wait for 0 seconds remaining.
2. Serialize your refreshes: Never let two threads refresh the same token.
3. Handle revocation gracefully: Distinguish between "API is down" (retry) and "Token is dead" (alert user).
4. Secure your storage: Encrypt at rest, always.
5. Instrument everything: Auth failures should be a first-class metric, not a log line.

Or, you can offload this entirely. At Truto, we treat authentication infrastructure as a core product, so you can focus on the data, not the handshake.