Connect Helicone to Claude: Track LLM Costs & Evaluate AI Responses

If you need to connect Helicone to Claude to track LLM costs, evaluate AI responses, and manage prompt iterations, you need a Model Context Protocol (MCP) server. This server acts as the translation layer between Claude's tool calls and Helicone's REST API. You can either build, host, and maintain this infrastructure yourself, or use a managed integration platform like Truto to dynamically generate a secure, authenticated MCP server URL. If your team uses ChatGPT, check out our guide on connecting Helicone to ChatGPT or explore our broader architectural overview on connecting Helicone to AI Agents.

Giving a Large Language Model (LLM) read and write access to your observability and telemetry stack is an engineering challenge. You have to handle API authentication, map complex nested JSON schemas into flat tool definitions, and deal with strict filtering formats. Every time Helicone updates an endpoint or adds a new prompt management feature, you have to update your server code, redeploy, and test the integration. This guide breaks down exactly how to use Truto to generate a secure, managed MCP server for Helicone, connect it natively to Claude, and execute complex LLM monitoring workflows using natural language.

The Engineering Reality of the Helicone API

A custom MCP server is a self-hosted integration layer. While the open MCP standard provides a predictable way for models to discover tools, the reality of implementing it against Helicone's API presents distinct challenges. Helicone is both an AI gateway and an observability platform, which means its API surface is split between high-volume telemetry reads and precise prompt management writes.

If you decide to build a custom MCP server for Helicone, you own the entire API lifecycle. Here are the specific challenges you will face:

Complex Filtering and ClickHouse Aggregations Helicone is built to handle massive volumes of LLM requests. Endpoints like the bulk request listing and session metrics rely on complex, nested filter objects targeting a ClickHouse backend. You cannot simply pass a user_id query parameter; you must construct deeply nested JSON logic (e.g., filter: { left: { request: { user: { equals: "user_123" } } }, operator: "and", right: "..." }). Mapping these dynamic schema requirements to static MCP tool definitions requires heavy manual curation. Truto automatically parses Helicone's documentation and translates these structures into LLM-friendly schemas.

Schema Drift in Prompt Configurations Helicone's prompt management API handles versioning, environments, and variable inputs. When you update a prompt version, the payload requires a specific chat completion format based on the underlying model provider (OpenAI, Anthropic, etc.). Hardcoding these payload shapes in a custom MCP server leads to fragile integrations that break when new model parameters are introduced. A documentation-driven tool generation system dynamically adapts to these schema changes.

Rate Limits and High-Volume Telemetry When your AI agent attempts to run automated evaluations across thousands of historical requests, it will hit rate limits. Helicone protects its endpoints against abuse. If you build a custom server, you must decide how to handle these limits. Truto takes a pass-through approach: it does not retry, throttle, or apply backoff on rate limit errors. When Helicone returns an HTTP 429, Truto passes that error to the caller, while normalizing upstream rate limit info into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) per the IETF spec. Your agent is responsible for executing intelligent retry and backoff logic.

How to Generate a Helicone MCP Server

Truto dynamically generates MCP tools based on the Helicone API documentation and integration configurations. You can create a self-contained, authenticated MCP server URL via the Truto UI or programmatically via the API.

Method 1: Via the Truto UI

For teams who prefer visual configuration, you can generate an MCP server directly from your Truto dashboard.

1. Log in to Truto and navigate to your Integrated Accounts.
2. Select your connected Helicone account.
3. Click the MCP Servers tab.
4. Click Create MCP Server.
5. Select your desired configuration. You can filter by HTTP methods (e.g., read, write) or by tags (e.g., prompts, observability).
6. Copy the generated MCP server URL. It will look like https://api.truto.one/mcp/a1b2c3d4e5f6....

Method 2: Via the API

For platform engineers looking to provision AI agent tools programmatically, you can POST to the Truto API. The system validates that the integration has available tools, generates a cryptographically secure token, stores it in Cloudflare KV, and returns the endpoint.

Endpoint: POST /integrated-account/:id/mcp

curl -X POST https://api.truto.one/integrated-account/<helicone_account_id>/mcp \
  -H "Authorization: Bearer <YOUR_TRUTO_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Helicone Prompt Engineering Agent",
    "config": {
      "methods": ["read", "write"],
      "tags": ["prompts"]
    }
  }'

The response returns the ready-to-use URL:

{
  "id": "mcp_abc123",
  "name": "Helicone Prompt Engineering Agent",
  "config": { "methods": ["read", "write"], "tags": ["prompts"] },
  "expires_at": null,
  "url": "https://api.truto.one/mcp/a1b2c3d4e5f6..."
}

How to Connect the MCP Server to Claude

Once you have the Truto MCP URL, you need to expose it to your LLM framework. The Truto MCP server implements a standard JSON-RPC 2.0 protocol over HTTP POST, meaning it works out-of-the-box with any standard MCP client.

Method 1: Via the Claude UI (Desktop/Web)

If you are using Claude's interface (or ChatGPT's developer settings):

1. In Claude, navigate to Settings -> Connectors -> Add custom connector.
2. In ChatGPT, navigate to Settings -> Apps -> Advanced settings -> Developer mode -> Custom connectors -> Add.
3. Name your connection (e.g., "Helicone Telemetry").
4. Paste the Truto MCP server URL into the URL field and click Add.
5. The model will automatically execute an initialize handshake and call tools/list to discover the Helicone API endpoints.

Method 2: Via Manual Configuration File (Claude Desktop)

For local development or custom agent deployments, you can configure Claude Desktop using the claude_desktop_config.json file. Because Truto uses standard HTTP POST for MCP, you simply point the standard SSE transport client to your URL.

{
  "mcpServers": {
    "helicone": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-sse",
        "https://api.truto.one/mcp/a1b2c3d4e5f6..."
      ]
    }
  }
}

Save the file and restart Claude Desktop. The tools will immediately appear as available functions.

Hero Tools for Helicone

Truto maps every documented Helicone API endpoint to a callable tool. The MCP router handles flattening the input namespace, parsing the agent's flat argument object, and splitting it appropriately between query parameters and the JSON body based on the derived JSON schemas. Here are the highest-leverage operations for observability and prompt management.

list_all_helicone_requests

This tool is the primary entry point for querying LLM request logs. It accepts complex filter objects to isolate requests by model, time range, latency, cost, and custom properties. Claude can use this to identify failing inputs or analyze usage spikes.

"Find the last 20 requests sent to gpt-4-turbo where the cost exceeded 0.05 dollars, and summarize the prompt inputs that drove the cost up."

helicone_requests_submit_score

Essential for LLM-as-a-judge workflows. Once Claude evaluates a specific LLM request (either fetched previously or provided in context), it can attach a numeric evaluation score to that request in Helicone for tracking.

"Evaluate the request ID 'req-789' for tone and correctness on a scale of 1 to 10. Once evaluated, submit the score back to Helicone under the metric 'claude_tone_eval'."

helicone_sessions_list_metrics

Helicone sessions group multiple requests into a single trace (e.g., a LangChain agent run). This tool allows Claude to aggregate cost, token usage, and duration at the session level to monitor overarching task efficiency.

"Pull the session metrics for the last 24 hours grouped by session_id, and tell me which three multi-agent sessions consumed the most tokens."

create_a_helicone_prompt

Claude can architect its own system prompts or create specific templates for other models, committing them directly into Helicone's prompt registry. It accepts a name, tags, and the initial prompt body in OpenAI chat completion format.

"Create a new Helicone prompt named 'Support Triage Agent'. Use tags 'support' and 'v1'. The prompt body should instruct the model to categorize tickets into Billing, Technical, or Enterprise."

update_a_helicone_prompt_by_id

This is the core tool for iterative prompt engineering. Claude can modify an existing prompt based on evaluation data and push a new minor or major version to the registry, complete with a commit message.

"Take the prompt ID 'pr-456', add an instruction to format the output as JSON, and push a new minor version with the commit message 'added JSON formatting constraints'."

list_all_helicone_evals

Retrieves a list of evaluation scores and criteria defined in the Helicone environment. This helps Claude understand what metrics the engineering team currently cares about before running an automated scoring pass.

"List all evaluation criteria currently active in Helicone. I want to see if we are currently tracking 'hallucination_rate' or if I need to create a new eval."

For the complete inventory of tools, including asset attachments, gateway routing, and ClickHouse bulk exports, see the Helicone Integration Page.

Workflows in Action

By exposing Helicone to Claude, you transition from manually debugging dashboards to deploying autonomous prompt engineers. Here is how Claude executes real-world observability workflows using the generated tools.

Workflow 1: Automated Prompt Evaluation and Scoring

Engineering teams often struggle to backtest prompt changes against historical user inputs. You can task Claude with finding specific failure modes in production logs and scoring them.

"Search Helicone for the last 10 requests where the model returned an HTTP 400 error. Evaluate the inputs to determine if the prompt template failed to sanitize the variables. For each request, submit a score of 0 for 'prompt_safety' if it failed sanitization, and 1 if it was a different error."

1. list_all_helicone_requests: Claude constructs a filter for requests where the status code is 400 and limits the response to 10 records.
2. Analysis: Claude processes the inputs and template parameters returned in the payload to identify the failure root cause.
3. helicone_requests_submit_score: Claude loops over the identified requests, calling this tool for each one to attach the prompt_safety score.

Claude handles the iterative tool calling automatically, turning a tedious manual audit into a single natural language command.

sequenceDiagram
    participant User as User
    participant Claude as Claude Desktop
    participant Truto as Truto MCP Server
    participant Helicone as Helicone API

    User->>Claude: "Evaluate 400 errors for safety..."
    Claude->>Truto: Call list_all_helicone_requests
    Truto->>Helicone: Proxy API Request
    Helicone-->>Truto: Return request logs
    Truto-->>Claude: Return JSON array
    Claude->>Claude: Analyze inputs for sanitization failures
    Claude->>Truto: Call helicone_requests_submit_score (x10)
    Truto->>Helicone: Proxy POST requests
    Helicone-->>Truto: Return success codes
    Truto-->>Claude: Acknowledge scores submitted

Workflow 2: Prompt Iteration based on Latency Spikes

When a specific prompt version starts taking too long to generate completions (impacting user experience), Claude can isolate the version and rewrite it for brevity.

"Find the production version for the prompt ID 'pr-111'. Check the average latency for requests using this version over the last 3 days. If it's above 4 seconds, update the prompt to remove the redundant context examples and push a new major version."

1. helicone_prompts_get_production_version: Claude retrieves the current active version ID and full prompt body for 'pr-111'.
2. list_all_helicone_requests: Claude filters requests by the retrieved prompt_version_id and a time range of the last 3 days, reading the latency metrics.
3. Analysis: Claude determines the average latency exceeds the threshold and rewrites the retrieved prompt body to be more concise.
4. update_a_helicone_prompt_by_id: Claude pushes the optimized prompt as a new major version, committing the change directly to the Helicone registry.

Security and Access Control

Giving AI agents API access to your production telemetry requires strict boundaries. Truto provides four mechanisms to lock down your Helicone MCP server:

• Method Filtering: Restrict the MCP server to specific HTTP methods. Passing methods: ["read"] ensures the LLM can only query logs and metrics, preventing accidental prompt modifications or metric deletions.
• Tag Filtering: Scope the available tools to specific functional areas. Applying tags: ["observability"] exposes logging endpoints while completely hiding prompt management and gateway routing tools.
• Extra Authentication: Enable require_api_token_auth: true. The MCP client must provide a valid Truto API token in the Authorization header. This ensures that even if the MCP server URL is leaked, unauthorized users cannot execute tools.
• Expiration: Set an expires_at timestamp when generating the server. Truto uses Cloudflare KV TTLs and Durable Object alarms to permanently destroy the server and its tokens once the time expires, perfect for temporary agent debugging sessions.

Wrapping Up

Building custom MCP infrastructure to connect Claude to Helicone forces your engineering team into a perpetual cycle of reading vendor docs, maintaining complex JSON schemas, and debugging rate limit behavior. By using a managed MCP server, you offload the boilerplate. Truto dynamically converts Helicone's comprehensive observability endpoints into structured tools, normalizes the pagination, and securely routes the API traffic.

Whether your agents are evaluating historical responses, monitoring prompt token costs, or iterating on system prompts directly in the registry, a managed approach allows you to focus on AI workflows rather than API plumbing.