Connect Langfuse to Claude: Audit Experiments & Datasets

If you need to connect Langfuse to Claude to audit LLM tracing pipelines, manage datasets, and automate prompt engineering workflows, you need a Model Context Protocol (MCP) server. This server acts as the translation layer between Claude's tool calls and the Langfuse REST API. You can either build and maintain this infrastructure yourself, or use a managed integration platform like Truto to dynamically generate a secure, authenticated MCP server URL. If your team uses ChatGPT, check out our guide on connecting Langfuse to ChatGPT or explore our broader architectural overview on connecting Langfuse to AI Agents.

Giving a Large Language Model (LLM) read and write access to your observability and evaluation stack is an engineering challenge. You have to handle API key lifecycles, map complex polymorphic JSON schemas to MCP tool definitions, and deal with upstream rate limits. Every time Langfuse updates an endpoint - migrating from v1 observations to v2, or shifting unstable evaluation endpoints - you have to update your server code, redeploy, and test the integration.

This guide breaks down exactly how to use Truto to generate a secure, managed MCP server for Langfuse, connect it natively to Claude Desktop, automate scale-intensive tasks, and execute complex evaluation and auditing workflows using natural language.

The Engineering Reality of the Langfuse API

A custom MCP server is a self-hosted integration layer. While the open MCP standard provides a predictable way for models to discover tools, the reality of implementing it against the Langfuse API is painful. You are not just integrating a simple CRUD app - you are interfacing with a high-volume telemetry, evaluation, and tracing system.

If you decide to build a custom MCP server for Langfuse, you own the entire API lifecycle. Here are the specific challenges you will face:

Polymorphic Scoring Schemas In Langfuse, a score is not just a simple integer. The dataType of a score dictates its structure - it can be NUMERIC, BOOLEAN, CATEGORICAL, or a CORRECTION. When exposing this to an LLM, passing a raw, unstructured schema often causes the model to hallucinate string values for numeric fields or vice versa. Truto maps these complex, polymorphic API definitions into rigid, MCP-compliant JSON schemas, ensuring Claude knows exactly what data types to provide when evaluating traces.

Evolving Endpoints and Cursor Pagination Langfuse is a rapidly evolving platform. The original v1 observations endpoint relied on standard pagination, but the newer v2 observations endpoint introduces complex filtering and cursor-based pagination designed for massive datasets. If you expose raw pagination parameters to Claude, the model will frequently guess cursor values or misunderstand how to iterate through pages of experiment items. Truto normalizes this across all Langfuse endpoints into a standard limit and next_cursor schema. The tool description explicitly instructs the LLM to pass cursor values back unchanged, preventing hallucinated tokens.

Handling Telemetry Rate Limits Langfuse enforces API rate limits to protect its ingestion and querying infrastructure. Truto does not retry, throttle, or apply backoff on rate limit errors. When Langfuse returns an HTTP 429 Too Many Requests, Truto passes that error directly to the caller. However, Truto normalizes the upstream rate limit information into standardized headers (ratelimit-limit, ratelimit-remaining, ratelimit-reset) following the IETF specification. Claude (the caller) is responsible for reading these headers and executing its own backoff and retry logic.

Instead of building schema translation, cursor management, and secure token storage from scratch, you can use Truto to generate a Langfuse MCP server in seconds.

How to Generate a Langfuse MCP Server with Truto

Truto's MCP feature dynamically derives tool definitions directly from the integration's documented resources. Tools are never pre-built or cached; they are generated on the fly based on the specific capabilities of your connected Langfuse instance.

You can create a Langfuse MCP server using either the Truto UI or the API.

Method 1: Via the Truto UI

For teams who prefer a visual setup, generating an MCP server takes less than a minute:

1. Log into your Truto dashboard and navigate to the Integrated Accounts page.
2. Select your connected Langfuse integration.
3. Click on the MCP Servers tab.
4. Click Create MCP Server.
5. Select your desired configuration. You can filter tools by methods (e.g., read-only) or specific tags.
6. Copy the generated MCP server URL (e.g., https://api.truto.one/mcp/abc123def456...).

Method 2: Via the Truto API

For platform engineers building automated provisioning flows, you can generate an MCP server programmatically. This approach allows you to restrict the LLM to specific methods (like read operations only) or set an expiration time for ephemeral environments.

// POST /integrated-account/:id/mcp
 
const response = await fetch('https://api.truto.one/integrated-account/<LANGFUSE_ACCOUNT_ID>/mcp', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer YOUR_TRUTO_API_KEY',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    name: "Claude Langfuse Audit Server",
    config: {
      methods: ["read", "write"], 
      tags: ["datasets", "experiments"]
    },
    expires_at: "2026-12-31T23:59:59Z"
  })
});
 
const mcpServer = await response.json();
console.log(mcpServer.url); 
// Returns: https://api.truto.one/mcp/a1b2c3d4e5f6...

The returned URL contains a cryptographically hashed token that binds it strictly to your Langfuse workspace. Do not expose this URL publicly.

Connecting the MCP Server to Claude

Once you have your Truto MCP server URL, you must connect it to your Claude environment. You can do this through the Claude application UI or via a configuration file for Claude Desktop.

Method 1: Via the Claude UI

If you are using Claude's web or desktop interface with custom connector support:

1. Open Claude and navigate to Settings.
2. Click on Integrations (or Connectors depending on your tier).
3. Select Add MCP Server or Add custom connector.
4. Name your integration "Langfuse Truto".
5. Paste the Truto MCP server URL.
6. Click Add. Claude will instantly execute a protocol handshake, pull the available tools, and make them available in your workspace.

Method 2: Via Manual Configuration File

For developers using the Claude Desktop application, you can map the Truto server directly in your JSON config by utilizing the standard SSE (Server-Sent Events) transport command.

Locate your claude_desktop_config.json file (typically at ~/Library/Application Support/Claude/claude_desktop_config.json on macOS) and add the following:

{
  "mcpServers": {
    "langfuse-truto": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-sse",
        "https://api.truto.one/mcp/YOUR_SECURE_TOKEN"
      ]
    }
  }
}

Restart Claude Desktop. The application will detect the configuration, connect to Truto via SSE, and populate Claude with Langfuse tools.

Hero Tools for Langfuse

Truto exposes the entirety of the Langfuse REST API as discrete, callable tools. When Claude calls a tool, Truto maps the flat LLM arguments into the exact query parameters and JSON body payloads that Langfuse expects.

Here are 6 high-leverage hero tools your AI agents can use to audit and manage Langfuse data.

1. List Public Traces

Tool Name: list_all_langfuse_public_traces

This tool retrieves a paginated list of traces within a project. It is heavily utilized for auditing trace logs, inspecting latency issues, or analyzing overall token usage across a specific subset of user sessions. It supports complex filtering by userId, sessionId, tags, and time ranges.

"Fetch the latest 20 traces from Langfuse where the tag is 'production-bot'. Tell me the average latency and total cost for these calls."

2. List V2 Observations

Tool Name: list_all_langfuse_public_observations_v_2

This accesses the modern v2 Observations API, returning individual generations, spans, and events with deep metadata. Claude uses this tool when it needs to inspect the exact input prompts, outputs, and token counts for individual LLM calls, rather than just the top-level trace.

"List the recent observations in Langfuse using the v2 endpoint. Filter for observations where the model is 'gpt-4o' and summarize the prompt inputs being used."

3. List Experiment Items

Tool Name: list_all_langfuse_public_experiment_items

This tool fetches items tied to a specific experiment, retrieving the traceId, startTime, and associated scoring fields. It is essential for regression testing, allowing the model to look at how a specific prompt iteration performed against a benchmark dataset.

"Get the latest experiment items from the 'summarization-regression-test' experiment. Identify any trace IDs that received a score below 0.7."

4. Manage Dataset Items

Tool Name: list_all_langfuse_public_dataset_items

Datasets form the ground truth for LLM evaluations. This tool lists the input, expectedOutput, and metadata for items in a dataset. Claude can use this to audit dataset quality or prepare examples for a few-shot prompting pipeline.

"Pull the dataset items for the dataset named 'customer-service-faqs'. Check if any items have a blank expectedOutput and list their IDs."

5. Create a Score

Tool Name: create_a_langfuse_public_score

Agents can act as automated evaluators (LLM-as-a-judge). This tool allows Claude to attach a numeric or categorical score directly to a traceId or observationId based on its own analysis of the execution quality.

"Analyze trace ID 'tr-12345-abc'. Based on our safety guidelines, if the output contains restricted keywords, create a score named 'safety_eval' with a value of 0. Otherwise, score it 1."

6. Manage Evaluation Rules

Tool Name: create_a_langfuse_unstable_evaluation_rule

For advanced CI/CD setup, Claude can programmatically provision LLM-as-a-judge rules. This unstable endpoint creates evaluation rules mapped to specific targets (like observations) and connects them to predefined evaluators.

"Create a new evaluation rule in Langfuse named 'Automated Tone Check'. Target it at all observations and link it to the 'Tone Evaluator' we deployed yesterday."

For the full list of available tools, required schema structures, and field definitions, see the Langfuse Integration Page.

Workflows in Action

Exposing these tools to Claude transforms manual observability tasks into autonomous, repeatable workflows. Here are two real-world scenarios.

Scenario 1: The Automated Dataset Auditor

AI engineers frequently manage massive datasets for fine-tuning or evaluation. When data drifts, or when team members add incomplete items, the evaluation pipeline breaks. Claude can serve as an automated auditor to monitor dataset integrity.

"Check the dataset named 'support-routing-v2' in Langfuse. Review the latest items. If any item has an input longer than 1000 characters but is missing metadata tags, summarize the issue and list the item IDs."

Execution flow:

1. Claude calls list_all_langfuse_public_dataset_items passing datasetName: "support-routing-v2".
2. Truto translates this into a GET request to Langfuse.
3. Claude processes the returned JSON, iterating over the input lengths and metadata properties.
4. Claude generates a markdown report detailing the non-compliant items directly in the chat interface.

sequenceDiagram
  participant User as User
  participant Claude as Claude Desktop
  participant Truto as Truto MCP
  participant Langfuse as Langfuse API

  User ->> Claude: "Audit the support-routing-v2 dataset..."
  Claude ->> Truto: Call `list_all_langfuse_public_dataset_items`
  Truto ->> Langfuse: GET /api/public/datasets/.../items
  Langfuse -->> Truto: Return paginated items JSON
  Truto -->> Claude: Return tool result
  Claude ->> Claude: Analyze input lengths & metadata
  Claude -->> User: Render Markdown audit report

Scenario 2: LLM-as-a-Judge Regression Triage

When testing a new prompt variation against a baseline, engineers need to understand exactly why certain traces failed an automated evaluation. Claude can correlate experiment metrics with raw trace data and inject human-readable review scores.

"Fetch the latest items for the experiment 'sales-bot-v4'. Find any items that received a 'helpfulness' score below 0.5. For those items, retrieve the full trace to see what the bot actually said, explain why it likely failed, and create a new manual score named 'needs-human-review' with a value of 1 on those traces."

Execution flow:

1. Claude calls list_all_langfuse_public_experiment_items to get the recent batch of experiment items and their scores.
2. Claude identifies trace IDs where the 'helpfulness' score metric is < 0.5.
3. For each failing trace ID, Claude loops over get_single_langfuse_public_trace_by_id to read the actual input and output payload of the LLM call.
4. Claude synthesizes the failures.
5. Claude loops over create_a_langfuse_public_score to attach a needs-human-review flag to those specific traces in Langfuse.

flowchart TD
  A["Claude AI"] -->|"1. Fetch Experiment Items"| B["list_all_langfuse_public_experiment_items"]
  B -->|"Filter score < 0.5"| C["Identify Failed Trace IDs"]
  C -->|"2. Get Trace Details"| D["get_single_langfuse_public_trace_by_id"]
  D -->|"Analyze Input/Output"| E["Determine Failure Cause"]
  E -->|"3. Tag for Human Review"| F["create_a_langfuse_public_score"]

Security and Access Control

Handing an LLM unrestricted access to your observability stack presents security risks. Truto mitigates this by embedding strict access controls directly into the MCP token lifecycle.

• Method Filtering: You can enforce read-only access by configuring the MCP server with methods: ["read"]. This allows Claude to fetch traces and observations but prevents it from mutating datasets or creating unauthorized scores.
• Tag Filtering: Limit the server's scope to specific functional areas. By setting tags: ["datasets"], Claude will only see tools related to dataset management, hiding core project administration endpoints entirely.
• Require API Token Authentication: For maximum security, enable require_api_token_auth: true. This ensures that possessing the MCP URL alone is insufficient; the client must also pass a valid Truto user API token in the Authorization header to successfully execute tools.
• Ephemeral Servers: Set an expires_at timestamp when generating the server via the API. Truto will automatically destroy the MCP token and its associated Key-Value entries via a Durable Object alarm when the time expires, ensuring temporary debugging sessions do not leave lingering access points.

Connect Your Tech Stack to AI

Building custom MCP servers for complex observability platforms like Langfuse forces your engineering team into a perpetual cycle of reading vendor changelogs, updating schemas, and managing OAuth or API key state.

Truto removes the boilerplate. By abstracting the Langfuse API behind a dynamically generated, fully managed JSON-RPC 2.0 layer, you can give Claude immediate, secure access to your traces and datasets without writing a single line of integration code. Let your AI agents handle the evaluation regressions, so your engineers can focus on shipping better models.