Connect Langfuse to AI Agents: Automate Scores and Evaluations

You want to connect Langfuse to an AI agent so your system can independently read trace data, run automated model evaluations, score user sessions, and update prompt versions based on historical context. Here is exactly how to do it using Truto's /tools endpoint and SDK, bypassing the need to manually build and maintain a complex, highly-relational integration.

Giving a Large Language Model (LLM) read and write access to your Langfuse instance is a significant engineering challenge. You either spend weeks reading the Langfuse documentation to build, host, and maintain a custom API connector, or you use a managed infrastructure layer that handles the boilerplate for you. If your team uses ChatGPT, check out our guide on connecting Langfuse to ChatGPT, or if you are building on Anthropic's models, read our guide on connecting Langfuse to Claude. For developers building custom autonomous workflows, you need a programmatic way to fetch these tools and bind them to your agent framework.

This guide breaks down exactly how to fetch AI-ready tools for Langfuse, bind them natively to an LLM using LangChain (or any framework like LangGraph, CrewAI, or Vercel AI SDK), and execute complex LLM observability and evaluation workflows. For a deeper look at the architecture behind this approach, refer to our research on architecting AI agents and the SaaS integration bottleneck.

The Engineering Reality of Custom Langfuse Connectors

Building AI agents is the easy part. Safely connecting them to external LLM engineering platforms is where systems fail. Giving an LLM access to external telemetry data sounds simple in a Jupyter notebook. You write a Python function that makes an HTTP GET request to a trace endpoint and wrap it in an @tool decorator. In production, this approach collapses entirely, especially with a relational ecosystem as complex as Langfuse.

If you decide to build this integration yourself, you own the entire API lifecycle. Langfuse's API introduces several highly specific integration challenges that break standard LLM assumptions.

The Nesting and Pagination Trap

Langfuse relies heavily on deeply nested, relational concepts. A Session contains Traces. A Trace contains Observations. An Observation can be a Span, an Event, or a Generation. To calculate the cost of a specific user interaction, an agent cannot simply hit a single endpoint. It must list traces for a session, paginate through them, fetch the specific generation observations associated with those traces, and parse the polymorphic usageDetails and costDetails fields.

If you hand-code this integration, you have to write complex system prompts to teach the LLM how to manage cursor-based pagination for the v2 Observation endpoints, while managing offset pagination for older v1 endpoints. When the LLM hallucinates a cursor string or requests a nested object type that does not exist, the workflow crashes.

Polymorphic Schema Complexity

Langfuse's architecture supports immense flexibility, particularly in how it handles scoring and evaluations. When you hit the /api/public/v3/scores endpoint (or its deprecated v2 predecessor), the schema for the value field is polymorphic. Depending on the dataType (NUMERIC, BOOLEAN, CATEGORICAL), the value might be a float, a true/false primitive, or a string.

Large Language Models struggle with polymorphic schemas in function calling. If you pass the raw Langfuse OpenAPI spec to an agent, it will frequently attempt to pass a string to a NUMERIC score, resulting in a 422 Unprocessable Entity error. The agent then gets stuck in a retry loop trying to correct its own schema violation.

Unstable Endpoints and API Drift

Langfuse is a fast-moving open-source project. Features like Evaluators and Evaluation Rules are incredibly powerful but live under the /api/public/unstable/ path hierarchy. These endpoints are subject to rapid evolution. Relying on handwritten fetch wrappers for unstable endpoints guarantees integration debt. Your engineering team will be paged when a schema change breaks your agent's ability to trigger automated LLM-as-a-judge evaluations.

A unified tool layer collapses these complexities. Truto normalizes the endpoint schemas into discrete, strictly typed tools. The LLM only ever chooses from stable function names and predictable JSON schemas. Invalid arguments are rejected locally before they hit the network, and pagination boilerplate is abstracted away.

Essential Langfuse Tools for AI Agents

Instead of overwhelming your LLM with a 10,000-line OpenAPI specification, Truto's /tools endpoint provides discrete, purpose-built functions. This drastically reduces the context window overhead and prevents hallucinated API parameters. Here are the highest-leverage Langfuse operations available to your AI agents.

list_all_langfuse_public_traces

This tool allows an agent to search and filter traces across a Langfuse project. It supports filtering by userId, sessionId, tags, and time ranges. This is critical for agents acting as automated reviewers, allowing them to pull subsets of traces that match specific error states or user behaviors.

"Find all traces in the production environment from the last 24 hours that contain the tag 'hallucination-suspected'. Retrieve their trace IDs and latency metrics."

get_single_langfuse_public_trace_by_id

Once an agent identifies a problematic trace, it needs full context. This tool retrieves the complete trace object, including its nested observations (events, spans, generations), input/output payloads, metadata, and aggregated metrics like totalCost and latency.

"Fetch the complete details for trace ID 'trc-8f92a1b-44c2'. Extract the raw output text from the primary LLM generation observation."

create_a_langfuse_public_score

This is the engine for autonomous LLM-as-a-judge workflows. This tool allows the agent to create a score and attach it to a trace or session. Because Truto handles the schema enforcement, the agent knows exactly how to format NUMERIC, BOOLEAN, or CATEGORICAL scores.

"The generation in this trace contained harmful content. Create a categorical score named 'safety-eval' with the value 'failed' and attach it to trace 'trc-8f92a1b-44c2'."

list_all_langfuse_public_prompts

Agents tasked with optimizing prompts need to read the current prompt registry. This tool fetches Langfuse prompt names, their current versions, labels (like 'production' or 'staging'), and the actual prompt templates.

"Retrieve the 'customer-support-system-prompt'. Return the exact text of the version currently labeled as 'production'."

create_a_langfuse_unstable_evaluation_rule

For advanced autonomous observability, agents can dynamically create evaluation rules in Langfuse. If an agent detects a new edge case in the data, it can provision a new rule targeting specific observation names to trigger automated evaluators going forward.

"We are seeing increased latency on the 'sql-generation' span. Create a new evaluation rule that triggers the 'latency-monitor' evaluator whenever an observation named 'sql-generation' is logged."

list_all_langfuse_public_dataset_run_items

When evaluating a new model version, agents need to retrieve the results of dataset runs. This tool lists the run items for a specified dataset and run name, allowing the agent to compare expected outputs against actual generated outputs.

"List all run items for the dataset 'onboarding-qna' under the run name 'gpt-4o-baseline'. Compare the outputs and calculate an accuracy percentage."

create_a_langfuse_annotation_queue_item

Sometimes an AI agent cannot make a definitive judgment and needs human intervention. This tool allows the agent to push a specific trace or observation into a Langfuse annotation queue for human review.

"This trace scored 0.4 on the helpfulness metric, which is below the threshold for auto-resolution. Add this trace to the 'human-review-tier-2' annotation queue."

To view the complete inventory of Langfuse capabilities, including SCIM user management, dataset manipulation, and blob storage integrations, visit the Langfuse integration page for comprehensive schemas and configuration details.

Real-World Use Cases

Exposing these tools to an agent transforms passive observability into active, autonomous engineering operations. Here are specific examples of how engineering teams orchestrate Langfuse tool calling in production.

1. Autonomous Prompt Optimization Pipeline

Prompt engineering is highly iterative. Instead of a human manually reviewing traces to find edge cases, an agent can run on a cron job, identify failing traces, and draft prompt variations.

"Analyze yesterday's traces for the 'rag-retrieval' operation. Find any traces where the 'relevance' score is below 0.5. Extract the user inputs from those traces, fetch the current 'rag-system-prompt', and write a new prompt version that specifically addresses these failure modes."

Step-by-step execution:

1. The agent calls list_all_langfuse_public_traces filtering by the time range and tags=rag-retrieval.
2. The agent loops through the results, calling get_single_langfuse_public_trace_by_id to inspect the nested scores for values under 0.5.
3. The agent calls list_all_langfuse_public_prompts to pull the active 'rag-system-prompt' template.
4. The LLM processes the failed inputs against the prompt template to deduce the failure reason.
5. The agent drafts a report or calls create_a_langfuse_public_prompt (if permitted) to stage a new version for human approval.

2. LLM-as-a-Judge Anomaly Detection

Running heavy evaluations on every single trace in production is cost-prohibitive. Instead, teams use agents to act as intelligent samplers, running deep evaluations only on traces that exhibit specific risk markers.

"Fetch the last 100 traces from the 'finance-bot' session. If any trace shows a totalCost greater than $0.05, analyze the full generation payload. If it contains financial advice, apply a boolean score of 'compliance-violation=true' and escalate it to the manual review queue."

Step-by-step execution:

1. The agent calls list_all_langfuse_public_traces for the 'finance-bot' environment.
2. It filters the returned array in memory for traces exceeding the cost threshold.
3. For each expensive trace, it calls get_single_langfuse_public_trace_by_id to read the generation content.
4. If the LLM identifies prohibited financial advice, it calls create_a_langfuse_public_score with dataType=BOOLEAN and value=true.
5. Finally, it calls create_a_langfuse_annotation_queue_item to push the offending trace to compliance officers.

Building Multi-Step Workflows

To build these workflows, you need to bind Truto's tools to an agent framework. The following architecture works across LangChain, LangGraph, Vercel AI SDK, and CrewAI.

First, you fetch the definitions from the Truto API and bind them. Because Truto normalizes the schema, you do not need to write custom Zod validation schemas for every Langfuse endpoint.

sequenceDiagram
    participant App as "Your Agent Application"
    participant Truto as "Truto /tools API"
    participant LLM as "LLM (OpenAI/Claude)"
    participant Langfuse as "Langfuse API"

    App->>Truto: GET /integrated-account/{id}/tools
    Truto-->>App: Returns strictly-typed JSON schemas
    App->>LLM: Bind tools to model and pass user prompt
    LLM-->>App: Emits tool_call (e.g., list_all_langfuse_public_traces)
    App->>Truto: Execute tool call with arguments
    Truto->>Langfuse: Translates to Langfuse REST request
    Langfuse-->>Truto: Raw JSON response
    Truto-->>App: Normalized result payload
    App->>LLM: Return tool result to context window

Handling API Rate Limits Deterministically

When building autonomous agents, error handling is the difference between a prototype and a production system. A common misconception is that middleware layers magically absorb all API rate limits.

Factual requirement: Truto does not automatically retry, throttle, or apply backoff on rate limit errors. When the upstream Langfuse API returns an HTTP 429 Too Many Requests error, Truto immediately passes that 429 error back to the caller.

However, tracking rate limits across dozens of different APIs is a nightmare because every vendor uses different headers (e.g., x-ratelimit-reset, Retry-After, X-Rate-Limit-Remaining). Truto solves this by normalizing upstream rate limit information into standardized headers according to the IETF specification:

• ratelimit-limit: The total request quota.
• ratelimit-remaining: The number of requests remaining.
• ratelimit-reset: The timestamp when the quota refreshes.

Your agent loop must intercept the 429, read the normalized header, and pause execution.

import { ChatOpenAI } from "@langchain/openai";
import { TrutoToolManager } from "truto-langchainjs-toolset";
 
async function runLangfuseAgent(prompt: string, integratedAccountId: string) {
  // 1. Initialize the Truto Tool Manager
  const toolManager = new TrutoToolManager({
    trutoToken: process.env.TRUTO_API_KEY,
  });
 
  // 2. Fetch Langfuse tools using the Integrated Account ID
  const tools = await toolManager.getTools(integratedAccountId);
 
  // 3. Bind tools to the LLM
  const llm = new ChatOpenAI({ 
    modelName: "gpt-4o",
    temperature: 0 
  }).bindTools(tools);
 
  let messages = [{ role: "user", content: prompt }];
  
  // 4. The Agent Execution Loop
  while (true) {
    try {
      const response = await llm.invoke(messages);
      messages.push(response);
 
      if (!response.tool_calls || response.tool_calls.length === 0) {
        // Agent has finished its task
        return response.content;
      }
 
      // Execute requested tool calls
      for (const toolCall of response.tool_calls) {
        const selectedTool = tools.find(t => t.name === toolCall.name);
        const toolResult = await selectedTool.invoke(toolCall.args);
        
        messages.push({
          role: "tool",
          tool_call_id: toolCall.id,
          name: toolCall.name,
          content: JSON.stringify(toolResult)
        });
      }
 
    } catch (error) {
      // 5. Deterministic Rate Limit Handling
      if (error.status === 429) {
        // Read the IETF standardized headers provided by Truto
        const resetTime = error.headers['ratelimit-reset'];
        const waitSeconds = Math.max(0, parseInt(resetTime) - Math.floor(Date.now() / 1000));
        
        console.warn(`Rate limit hit. Agent pausing for ${waitSeconds} seconds.`);
        await new Promise(resolve => setTimeout(resolve, waitSeconds * 1000));
        // Loop will re-attempt the previous invocation
      } else {
        throw error;
      }
    }
  }
}

By normalizing the rate limit headers, your engineering team can write a single backoff utility that works identical for Langfuse, Salesforce, Zendesk, or any other API your agent touches.

Agentic Observability Requires Structured Interfaces

Building an AI agent that can actively debug, evaluate, and optimize other AI models is the future of LLM engineering. But an agent is only as intelligent as the tools it has access to. Forcing an LLM to generate raw REST requests against complex, evolving SaaS APIs is a recipe for hallucinations and brittle pipelines.

By placing a unified tool layer between your agent frameworks and Langfuse, you enforce strict schema validation, standardize authentication, and normalize rate limit logic. Your agents spend less time fighting 422 schema errors and more time executing high-value observability tasks.