---
title: "Connect OpenPipe to Claude: Evaluate Completions and Record Request Logs"
slug: connect-openpipe-to-claude-evaluate-completions-and-record-request-logs
date: 2026-07-08
author: Roopendra Talekar
categories: ["AI & Agents"]
excerpt: "A complete engineering guide to connecting OpenPipe to Claude via Truto's managed MCP server. Automate dataset curation, model fine-tuning, and LLM evaluations."
tldr: "Learn how to generate a secure, managed MCP server for OpenPipe using Truto, connect it natively to Claude, and execute model evaluations, logging, and dataset workflows using natural language."
canonical: https://truto.one/blog/connect-openpipe-to-claude-evaluate-completions-and-record-request-logs/
---

# Connect OpenPipe to Claude: Evaluate Completions and Record Request Logs


If you need to connect OpenPipe to Claude to automate model evaluation, manage datasets, and record completion logs, you need a [Model Context Protocol (MCP)](https://truto.one/what-is-mcp-model-context-protocol-the-2026-guide-for-saas-pms/) server. This server acts as the translation layer between Claude's tool calls and OpenPipe's REST API. You can either build and maintain this infrastructure yourself, or use a managed integration platform like Truto to dynamically generate a [secure, authenticated MCP server URL](https://truto.one/managed-mcp-for-claude-full-saas-api-access-without-security-headaches/). If your team uses ChatGPT, check out our guide on [connecting OpenPipe to ChatGPT](https://truto.one/connect-openpipe-to-chatgpt-fine-tune-models-and-manage-datasets/) or explore our broader architectural overview on [connecting OpenPipe to AI Agents](https://truto.one/connect-openpipe-to-ai-agents-automate-dataset-curation-and-fine-tuning/).

Giving a Large Language Model (LLM) read and write access to a model-training pipeline like OpenPipe is a [high-stakes engineering challenge](https://truto.one/how-to-safely-give-an-ai-agent-access-to-third-party-saas-data/). You have to handle API authentication, map nested dataset schemas to MCP tool definitions, and deal with strict ingestion rate limits. Every time OpenPipe deprecates an endpoint or introduces a new evaluation metric, you have to update your server code, redeploy, and test the integration. 

This guide breaks down exactly how to use Truto to generate a secure, managed MCP server for OpenPipe, connect it natively to Claude, and execute complex evaluation and training workflows using natural language.

## The Engineering Reality of the OpenPipe API

A custom MCP server is a self-hosted integration layer. While the open MCP standard provides a predictable way for models to discover tools, implementing it against OpenPipe's specific architecture requires navigating several unique challenges.

If you decide to build a custom MCP server for OpenPipe, you own the entire API lifecycle. Here are the specific engineering realities you will face:

**The Deprecation Minefield**
OpenPipe evolves rapidly. A look at their API surface reveals a high volume of deprecated endpoints. For example, older flat paths like `/unstable/dataset/list` have been replaced, and methods like `create_a_open_pipe_dataset_entry_create` and `create_a_open_pipe_finetune_create` are deprecated in favor of resource-centric paths (`/datasets/{dataset}/entries` and `/models`). If you hardcode an MCP server, you are responsible for monitoring OpenPipe's changelog, updating your JSON-RPC tool definitions, and migrating your LLM prompts to the new schemas. Truto handles this dynamically: tools are generated from live, curated documentation records. If an endpoint is updated or replaced, Truto's environment-level overrides automatically expose the correct schema to Claude without requiring a code deploy.

**Strict Conversational Schema Validation**
When appending entries to an OpenPipe dataset, the API expects a highly specific, nested JSON array structure mirroring the OpenAI conversational format (e.g., an array of `messages` containing `role` and `content` keys). If you expose this directly to Claude without clear schema definitions and descriptions, the model will frequently hallucinate formatting, resulting in 400 Bad Request errors. Truto parses OpenPipe's YAML documentation records into rigid JSON Schemas, automatically enforcing required properties and injecting context so Claude understands exactly how to format conversational arrays before attempting the execution.

**High-Volume Ingestion Rate Limits**
When you ask Claude to process a batch of 50 logs or run criteria judgments across a large dataset, it will attempt to fire rapid sequence tool calls. OpenPipe enforces rate limits to protect its ingestion endpoints. **It is critical to understand that Truto does not retry, throttle, or apply backoff on rate limit errors.** When OpenPipe returns an HTTP `429 Too Many Requests` error, Truto passes that error directly back to Claude. Truto normalizes the upstream rate limit information into standardized headers (`ratelimit-limit`, `ratelimit-remaining`, `ratelimit-reset`) per the IETF spec. Your AI agent is strictly responsible for reading this error and implementing its own retry and backoff logic. Truto will not absorb the error for you.

## How to Generate an OpenPipe MCP Server with Truto

Truto's MCP architecture derives tool definitions dynamically from two existing data sources: the OpenPipe integration's defined resources and its associated documentation records. A tool only appears in the MCP server if it has a corresponding documentation entry - this acts as a quality gate to ensure only well-documented endpoints are exposed to Claude.

You can generate an OpenPipe MCP server in two ways.

### Method 1: Via the Truto UI

For teams that prefer a visual setup, you can generate a server URL in seconds:

1. Navigate to the **Integrated Accounts** page for your connected OpenPipe instance.
2. Click the **MCP Servers** tab.
3. Click **Create MCP Server**.
4. Select your desired configuration. You can filter by methods (e.g., `read`, `write`) or by specific tool tags.
5. Click Save and copy the generated MCP server URL. It will look like this: `https://api.truto.one/mcp/a1b2c3d4e5f6...`

### Method 2: Via the Truto API

For platform engineers building automated environments, you can generate the MCP server programmatically. The API validates that the integration is AI-ready, generates a secure token, hashes it, and stores it in high-speed KV storage with an optional expiration.

Make a POST request to `/integrated-account/:id/mcp`:

```json
POST /integrated-account/<openpipe_account_id>/mcp
Content-Type: application/json
Authorization: Bearer <your_truto_api_token>

{
  "name": "OpenPipe Eval Agent",
  "config": {
    "methods": ["read", "write"],
    "require_api_token_auth": false
  },
  "expires_at": "2026-12-31T23:59:59Z"
}
```

The response will return the database record and the secure `url` your agent needs to connect.

## Connecting the MCP Server to Claude

Once you have the Truto MCP server URL, connecting it to Claude is purely a configuration step. The URL contains a cryptographic token that encodes the specific OpenPipe account and tool filters. 

### Method 1: Via the Claude UI

If you are using Claude Desktop or an enterprise workspace that supports visual connector management:

1. Open Claude and navigate to **Settings**.
2. Select **Integrations** (or Connectors).
3. Click **Add MCP Server**.
4. Paste the Truto MCP URL and click **Add**.

Claude will immediately handshake with the URL, request the tool schemas via JSON-RPC, and populate the available OpenPipe actions.

### Method 2: Via the Manual Config File

If you are configuring Claude Desktop locally for development, you can add the server directly to your `claude_desktop_config.json` file. Because Truto provides a remote HTTP endpoint rather than a local binary, you will use the official MCP SSE transport module.

```json
{
  "mcpServers": {
    "openpipe-truto": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-sse",
        "https://api.truto.one/mcp/a1b2c3d4e5f6..."
      ]
    }
  }
}
```

Restart Claude Desktop. The application will initialize the connection and verify the JSON-RPC capabilities.

## Core OpenPipe Tools for Claude

When Claude lists the available tools, it receives a flat input namespace. Truto automatically translates Claude's flat argument object into the specific path, query, and body schemas required by OpenPipe. 

Here are the most high-leverage OpenPipe tools you can expose to Claude.

### create_a_open_pipe_chat_completion

Generates a chat completion against a fine-tuned OpenPipe model. Claude can use this tool to generate responses using your proprietary models and immediately evaluate the output.

**Required fields:** `messages`, `model`.

> "Run a chat completion against our custom model 'openpipe:mistral-7b-v2' using the prompt 'Summarize the attached support ticket.' Show me the usage statistics returned in the payload."

### create_a_open_pipe_report_anthropic

Records a request log from an Anthropic model call directly into OpenPipe. This is essential for building a log history that can later be used to fine-tune a smaller, cheaper model. 

**Optional fields:** `statusCode`, `errorMessage`, `metadata`, `tags`.

> "I just generated a summary for user 891. Log this request into OpenPipe using the Anthropic report tool. Tag it with 'environment: production' and 'feature: ticket_summary'."

### create_a_open_pipe_criteria_judge

Judges an LLM completion against a specific OpenPipe criterion. This tool allows Claude to act as a routing and evaluation orchestrator, taking outputs from other tools and scoring them via OpenPipe's evaluation engine.

**Required fields:** `criterion_id`, `output`.

> "Take the summary you just generated and run it through the OpenPipe criteria judge using criterion ID 'crit_992a'. Tell me the score and the exact explanation provided by the judge."

### create_a_open_pipe_dataset

Creates a new dataset container in your OpenPipe project. Use this when organizing new pipelines for fine-tuning campaigns.

**Required fields:** `name`.

> "Create a new dataset in OpenPipe named 'Customer Support Escalations Q3'. Let me know the ID of the new dataset so we can start appending records."

### create_a_open_pipe_dataset_entry

Appends new data to an existing dataset. Claude can batch process raw text, format it into the required conversational array, and push it to OpenPipe. Note that OpenPipe accepts a maximum of 100 entries per request.

**Required fields:** `dataset_id`, `entries`.

> "Format these three email transcripts into OpenAI conversational format (system, user, assistant). Then add them as entries to dataset ID 'ds_8472'."

### list_all_open_pipe_models

Retrieves a list of all models currently active in the OpenPipe project. This provides Claude with context on what models are available for inference, their training status, and context windows.

> "List all the models in our OpenPipe project. Find the one with the largest context window and tell me its current OpenPipe status and pricing tier."

For the complete inventory of available OpenPipe endpoints, including pagination cursors and metadata update tools, view the [OpenPipe integration page](https://truto.one/integrations/detail/openpipe).

## Workflows in Action

Connecting OpenPipe to Claude transforms manual data science and MLOps tasks into natural language conversations. Here are two real-world workflows you can execute.

### Workflow 1: Automated LLM Evaluation and Reporting

In this scenario, a DevOps engineer wants Claude to test a newly deployed OpenPipe model, evaluate its response against a strict criterion, and log the execution data for dashboarding.

> "Run a chat completion on model 'openpipe:support-v1' using the prompt 'How do I reset my API key?' Evaluate the output using criterion ID 'crit_accuracy_01'. If the score is above 0.8, log the original request and response to OpenPipe as a successful Anthropic report with the tag 'eval_pass'."

**Execution Steps:**
1. Claude calls `create_a_open_pipe_chat_completion` with the specified model and messages array.
2. Claude takes the resulting string and calls `create_a_open_pipe_criteria_judge`, passing the `criterion_id` and the `output`.
3. Claude reads the numeric score. Because it is above 0.8, it calls `create_a_open_pipe_report_anthropic` with the payload and success tags.

```mermaid
sequenceDiagram
participant ClaudeUI as Claude Desktop
participant TrutoMCP as Truto MCP Server
participant OpenPipeAPI as OpenPipe API
ClaudeUI->>TrutoMCP: Call create_a_open_pipe_chat_completion
TrutoMCP->>OpenPipeAPI: POST /api/v1/chat/completions
OpenPipeAPI-->>TrutoMCP: HTTP 200 (Model Output)
TrutoMCP-->>ClaudeUI: JSON-RPC Response
ClaudeUI->>TrutoMCP: Call create_a_open_pipe_criteria_judge
TrutoMCP->>OpenPipeAPI: POST /api/v1/evaluations/judge
OpenPipeAPI-->>TrutoMCP: HTTP 200 (Score > 0.8)
TrutoMCP-->>ClaudeUI: JSON-RPC Response
ClaudeUI->>TrutoMCP: Call create_a_open_pipe_report_anthropic
TrutoMCP->>OpenPipeAPI: POST /api/v1/reports/anthropic
OpenPipeAPI-->>TrutoMCP: HTTP 200 OK
TrutoMCP-->>ClaudeUI: JSON-RPC Response
```

### Workflow 2: Creating a Dataset from Raw Examples

An AI product manager has a block of raw customer service text and wants to convert it into a training dataset for an upcoming fine-tuning job.

> "I am going to paste 5 customer support interactions. First, create a new OpenPipe dataset called 'Q4 Billing Disputes'. Then, format my text into valid system/user/assistant conversational JSON arrays. Finally, upload all 5 entries to the new dataset."

**Execution Steps:**
1. Claude calls `create_a_open_pipe_dataset` with the name 'Q4 Billing Disputes'.
2. OpenPipe returns the new `dataset_id`.
3. Claude parses the user's raw text, maps it to the strict JSON array schema required by OpenPipe.
4. Claude calls `create_a_open_pipe_dataset_entry` using the new `dataset_id` and the formatted entries payload.
5. Claude confirms the successful upload and reports back any entry indices that threw errors.

## Security and Access Control

Exposing an MLOps platform like OpenPipe to an LLM requires strict access boundaries. If an agent loops maliciously, it could overwrite crucial datasets or trigger massive inference costs. Truto secures your MCP server at the token level:

*   **Method Filtering:** When generating the server, configure `methods: ["read"]` to allow Claude to list models and datasets, but explicitly prevent it from creating entries or triggering new fine-tuning jobs.
*   **Tag Filtering:** Group specific OpenPipe resources under custom tags in Truto, ensuring Claude only has access to endpoints relevant to its specific workflow (e.g., tags: `["evaluations"]`).
*   **Require API Token Auth:** By setting `require_api_token_auth: true`, possession of the MCP URL is no longer enough. The client must also pass a valid Truto API token in the Authorization header, preventing anonymous endpoint discovery if the URL leaks.
*   **Expiring Access:** Set an `expires_at` ISO datetime when generating the server. Once the timestamp passes, the underlying KV storage drops the token and the server becomes permanently unreachable, perfect for temporary audit workflows.

## Moving Forward with Agentic MLOps

Connecting OpenPipe to Claude via Truto eliminates the grueling boilerplate of integrating nested JSON payloads, handling deprecation cycles, and managing authentication state. Instead of maintaining a brittle custom integration layer, your engineering team can focus on orchestrating high-value evaluations and dataset curation workflows.

By leveraging Truto's dynamic tool generation and strict schema enforcement, you guarantee that Claude always understands exactly how to communicate with OpenPipe. Just remember: when your agents start pushing thousands of logs, monitor those HTTP 429 rate limit headers and handle your backoffs.

> Ready to connect OpenPipe to Claude and automate your MLOps pipelines? Book a technical session with our engineering team to see Truto's managed MCP servers in action.
>
> [Talk to us](https://cal.com/truto/partner-with-truto)