Does Truto automatically handle Helicone API rate limits?

No. Truto does not retry, throttle, or apply backoff on rate limit errors. If Helicone returns an HTTP 429, Truto passes that error directly to your application along with standardized IETF rate limit headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset). Your application code is responsible for handling the retry and backoff logic.

Can I use Truto's Helicone tools with LangGraph or CrewAI?

Yes. Truto's /tools endpoint returns standardized JSON schemas that describe the available Helicone methods. These schemas can be dynamically loaded and bound to any major AI framework, including LangChain, LangGraph, CrewAI, and the Vercel AI SDK.

Do I need to teach the LLM how to write Helicone ClickHouse filters?

No. By using Truto's unified tools, the complex nested JSON structures required by Helicone's query endpoints are abstracted behind strict, declarative JSON schemas. The LLM simply provides the arguments defined by the schema, and Truto handles the validation before passing it to the API.

Can my AI agent trigger actual LLM calls through Helicone?

Yes. By exposing the helicone_ai_gateway_create_chat_completion tool, your agent can route secondary prompts through the Helicone AI Gateway, allowing you to take advantage of Helicone's pass-through billing, caching, and logging.

Connect Helicone to AI Agents: Route AI Traffic & Analyze Sessions

You want to connect Helicone to an AI agent so your system can independently route AI traffic, query session analytics, evaluate trace quality, and manage prompt versions based on historical context. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to manually build custom API wrappers or maintain fragile authentication flows.

Giving a Large Language Model (LLM) read and write access to your Helicone instance is an engineering headache. You either spend weeks building, hosting, and maintaining a custom connector that maps out Helicone's complex ClickHouse-backed analytics queries, or you use a managed infrastructure layer that handles the boilerplate for you. If your team uses ChatGPT, check out our guide on connecting Helicone to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting Helicone to Claude. For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for Helicone, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex AI observability workflows. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

The Engineering Reality of Custom Helicone Connectors

Building AI agents is easy. Connecting them to external SaaS APIs is hard. Giving an LLM access to external data sounds simple in a prototype. You write a Node.js function that makes a fetch request and wrap it in an @tool decorator. In production, this approach collapses entirely, especially with an observability platform like Helicone.

If you decide to integrate Helicone yourself, you own the entire API lifecycle. Helicone's API introduces several highly specific integration challenges that break standard LLM assumptions.

The Custom Query Language Trap

Helicone relies on a highly flexible, ClickHouse-backed query system for filtering requests, sessions, and dashboard metrics. You do not just hit a simple GET /v1/requests?status=200 endpoint. Instead, the API requires deeply nested JSON payloads containing specific logical operators to filter data.

For example, to find requests made to gpt-4o that cost more than $0.01, you must construct a payload that looks like this:

{
  "filter": {
    "left": {
      "request": {
        "model": {
          "equals": "gpt-4o"
        }
      }
    },
    "operator": "and",
    "right": {
      "response": {
        "cost": {
          "gte": 0.01
        }
      }
    }
  }
}

If you hand-code this integration, you have to write massive system prompts to teach the LLM the exact syntax of this custom filtering language, including when to use equals, contains, gte, and how to nest left and right conditions. When the LLM inevitably hallucinates a key or misplaces an operator, the API rejects the payload with a generic 400 error. The agent fails, and the workflow stops.

Prompt Versioning Complexity

Managing prompts via the Helicone API is not a simple CRUD operation. Prompts in Helicone are structured around versions, environments, and inputs. When you want an agent to fetch the "current production prompt", it cannot just call a single endpoint. It must first query the prompt, fetch its designated production version ID, resolve the environment variables, and then retrieve the specific compiled body for that version.

Exposing these raw, multi-step relational operations directly to an LLM wastes context window space and significantly increases the chance of logical errors. The agent gets confused about whether it is updating the base prompt metadata or committing a new minor version.

The Gateway vs Observability Divide

Helicone serves two primary functions: acting as an AI Gateway (routing traffic, pass-through billing) and acting as an observability platform (logging, scoring, analytics). The API schemas for creating a chat completion via the gateway look entirely different from the schemas used to query request metadata.

When an LLM sees helicone_ai_gateway_create_chat_completion alongside helicone_requests_submit_score, it has to understand that one is a synchronous proxy to OpenAI/Anthropic, and the other is an asynchronous mutation on historical data. Managing these divergent schemas requires a strict validation layer.

Why a Unified Tool Layer Matters for Agent Safety

Before writing a line of integration code, decide what layer your agent talks to. This choice determines how safe your production system will be.

Direct API tools (one tool per raw Helicone endpoint) look convenient but they push provider quirks into the LLM's context. The model has to remember the nested JSON query structures and versioning rules.

A unified tool layer collapses these complexities behind strict JSON schemas. Your agent sees stable, declarative function definitions that Truto automatically normalizes.

flowchart TD
    Agent["AI Agent (LangChain / CrewAI)"]
    TM["Truto Tool Manager<br>(Schema Validation)"]
    API["Helicone API<br>(ClickHouse Backend)"]

    Agent -->|"Calls list_all_helicone_requests"| TM
    TM -->|"Validates strict JSON Schema"| API
    API -->|"Returns normalized response"| TM
    TM -->|"Feeds context back to LLM"| Agent

That gives you concrete safety wins:

Deterministic input validation. Every tool has a strict JSON schema. Invalid arguments (like a malformed filter payload) are rejected locally before they hit the Helicone API, so a broken tool call fails fast with a clear error message that the LLM can auto-correct.
Normalized error handling. Truto normalizes upstream errors. The agent does not have to interpret Helicone-specific status codes or HTML error pages.
Guaranteed Rate Limit Transparency. Truto does not swallow rate limits. If Helicone returns an HTTP 429, Truto passes the error back to the caller along with standardized IETF rate limit headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset). This allows your application framework to orchestrate its own intelligent backoff strategy rather than guessing why a request hung.

Helicone Hero Tools for AI Agents

Instead of exposing the entire Helicone API surface, you provide the LLM with specific, high-leverage tools. Here are the core "hero" tools you will use to build autonomous observability and gateway workflows.

List All Helicone Requests

list_all_helicone_requests

This tool allows the agent to query and list LLM requests using Helicone's advanced filtering. It is essential for auditing traffic, finding failed requests, or identifying high-latency calls.

"Find all LLM requests from the past 24 hours that used the gpt-4-turbo model and had a latency higher than 5000 milliseconds. Return their request IDs and token usage."

Create a Helicone Eval

create_a_helicone_eval

Agents can autonomously evaluate the outputs of other models. This tool attaches an evaluation score (e.g., relevance, toxicity, or accuracy) to a specific request ID in Helicone.

"Evaluate request ID 9a8b7c6d based on our internal safety guidelines. If it passes, create a Helicone eval named 'safety_check' with a score of 100. If it fails, submit a score of 0."

Gateway Create Chat Completion

helicone_ai_gateway_create_chat_completion

This tool turns Helicone into an active router. The agent can trigger secondary chat completions through the Helicone gateway, inheriting Helicone's caching, rate limiting, and pass-through billing configurations.

"Take this user query and run it through the Helicone AI Gateway using the claude-3-5-sonnet-20241022 model. Return the exact text response generated by the model."

Get Single Prompt by ID

get_single_helicone_prompt_by_id

Retrieves the metadata, tags, and version history of a specific prompt template. Useful when an agent needs to audit which prompt is currently designated for production.

"Look up the prompt template with ID 12345. Tell me what tags are currently applied to it and when it was last modified."

List Dashboard Metrics

list_all_helicone_dashboard

Queries high-level aggregated metrics from Helicone's dashboard, such as total cost, average latency, or total tokens over a specific time range.

"Pull the Helicone dashboard metrics for the 'customer_support' team over the last 7 days. Summarize the total cost and average latency across all models."

List Session Metrics

helicone_sessions_list_metrics

Helicone supports session-level tracking (grouping multiple LLM calls into one conversational trace). This tool fetches aggregated metrics for these sessions.

"Find the top 5 longest sessions from yesterday and list their total token consumption and session durations."

To view the complete inventory of available Helicone tools and their strict JSON schemas, visit the Helicone integration page.

Workflows in Action

Individual tools are useful, but the real power of AI agents comes from chaining them together to execute multi-step revenue operations and DevOps tasks.

Workflow 1: Automated Trace Evaluation and Alerting

Quality assurance teams need to ensure that customer-facing LLMs are not hallucinating. You can configure an agent to continuously audit recent traffic and programmatically score it.

"Fetch the last 10 requests from the 'support_bot' user. Read the inputs and outputs. If the model provided an inaccurate refund policy, create an eval score of 0 named 'policy_accuracy'. If it was correct, score it 100. Finally, summarize the results."

list_all_helicone_requests: The agent queries Helicone for recent requests filtered by the specific user ID (support_bot).
create_a_helicone_eval: The agent analyzes the response bodies in its own context window. For each request, it calls this tool to attach a numeric score.
Agent Logic: The agent aggregates the scores and outputs a final textual summary of the audit.

Workflow 2: Cost Anomaly Detection

Platform engineering teams need to track down runaway LLM costs before the end of the billing cycle.

"Analyze the dashboard metrics for the past 24 hours. If total costs exceed $50, query the session metrics to find the specific session IDs responsible for the highest spend. List those IDs and their associated costs."

list_all_helicone_dashboard: The agent fetches aggregated cost metrics for the past 24 hours.
helicone_sessions_list_metrics: Upon detecting the $50 threshold was breached, the agent queries session-level metrics, sorting by cost to identify the outliers.
Agent Logic: The agent formats the offending session IDs and cost figures into an alert payload.

sequenceDiagram
    participant User as User / Scheduler
    participant Agent as AI Agent
    participant Truto as Truto Tool Layer
    participant Helicone as Helicone API

    User->>Agent: "Find expensive sessions > $50"
    Agent->>Truto: call list_all_helicone_dashboard(time_range)
    Truto->>Helicone: GET /v1/dashboard
    Helicone-->>Truto: { total_cost: 85.50 }
    Truto-->>Agent: { total_cost: 85.50 }
    
    Note over Agent: Threshold exceeded, drilling down.
    
    Agent->>Truto: call helicone_sessions_list_metrics(sort: cost)
    Truto->>Helicone: POST /v1/sessions/query
    Helicone-->>Truto: [ { session_id: "xyz", cost: 40.00 } ]
    Truto-->>Agent: [ { session_id: "xyz", cost: 40.00 } ]
    Agent-->>User: "Session xyz cost $40.00."

Building Multi-Step Workflows

To run these workflows in production, you need to bind Truto's proxy tools to your agent framework. Truto provides the truto-langchainjs-toolset which handles the dynamic fetching of schemas from the /integrated-account/<id>/tools endpoint.

This architecture is framework-agnostic. While the example below uses LangChain, the JSON schemas returned by Truto can be loaded into LangGraph, CrewAI, or the Vercel AI SDK.

Implementing the Tool Manager

First, install the required packages:

bun install @langchain/openai @trutohq/truto-langchainjs-toolset

Next, initialize the agent. Notice the explicit error handling block. Because Truto does not retry or swallow HTTP 429 rate limit errors, you must intercept the tool execution results and handle backoff logic locally using the standardized ratelimit-remaining and ratelimit-reset headers.

import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "@trutohq/truto-langchainjs-toolset";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";
import { ChatPromptTemplate } from "@langchain/core/prompts";
 
async function runHeliconeAgent() {
  // 1. Initialize the LLM
  const llm = new ChatOpenAI({
    modelName: "gpt-4o",
    temperature: 0,
  });
 
  // 2. Initialize the Truto Tool Manager
  // You need your Truto PAT and the specific Helicone integrated account ID
  const toolManager = new TrutoToolManager({
    trutoPat: process.env.TRUTO_PAT!,
  });
 
  const accountId = process.env.HELICONE_INTEGRATED_ACCOUNT_ID!;
 
  // 3. Fetch all Helicone tools available for this account
  console.log("Fetching Helicone tools...");
  const tools = await toolManager.getTools(accountId);
  console.log(`Loaded ${tools.length} Helicone tools.`);
 
  // 4. Bind the tools to the prompt and agent
  const prompt = ChatPromptTemplate.fromMessages([
    ["system", "You are an AI Ops engineer. Use the provided tools to query Helicone data. If a tool fails due to a rate limit, advise the user."],
    ["human", "{input}"],
    ["placeholder", "{agent_scratchpad}"],
  ]);
 
  const agent = createToolCallingAgent({
    llm,
    tools,
    prompt,
  });
 
  const executor = new AgentExecutor({
    agent,
    tools,
    maxIterations: 5,
  });
 
  // 5. Execute the multi-step workflow
  const query = "Find the total cost of all requests in the last 24 hours. If it is over $10, list the top 3 longest sessions.";
  
  try {
    console.log(`Executing query: ${query}`);
    const result = await executor.invoke({ input: query });
    console.log("\nAgent Output:\n", result.output);
    
  } catch (error: any) {
    // Explicit rate limit handling
    if (error.response?.status === 429) {
      const resetTime = error.response.headers.get('ratelimit-reset');
      console.error(`Helicone Rate Limit Hit. Caller must retry after ${resetTime} seconds.`);
      // Implement your custom backoff logic here
    } else {
      console.error("Workflow failed:", error.message);
    }
  }
}
 
runHeliconeAgent();

When you execute this script, the LLM parses the natural language prompt, decides to call list_all_helicone_dashboard, evaluates the total cost returned, and subsequently decides whether to call helicone_sessions_list_metrics. The entire orchestration happens autonomously, bounded by the strict schemas provided by Truto.

Unblocking AI Ops Teams

Connecting AI agents to Helicone shifts your team from reactive dashboard-watching to proactive observability. Instead of manually clicking through logs to find why a prompt failed, an agent can continually monitor request streams, evaluate outputs, and flag cost anomalies in real-time.

By leveraging a unified tool layer, you eliminate the friction of maintaining complex ClickHouse filter queries, parsing undocumented errors, and writing endless validation schemas. Your engineers can focus on building intelligent system prompts and agentic behaviors, while the underlying integration layer ensures structural integrity and deterministic execution.